XiphWiki - User contributions [en]

Videos/Digital Show and Tell

2013-02-26T08:25:41Z

Mindspillage: /* Epilogue */ +Credits section from video

<small>''Wiki edition''</small>
[[Image:dsat_001.jpg|400px|right]]

Continuing in the "firehose" tradition of [[Videos/A_Digital_Media_Primer_For_Geeks|Episode 01]], Xiph.Org's second video on digital media explores multiple facets of digital audio signals and how they ''really'' behave in the real world.

Demonstrations of sampling, quantization, bit-depth, and dither explore digital audio behavior on real audio equipment using both modern digital analysis and vintage analog bench equipment, just in case we can't trust those newfangled digital gizmos. You can download the source code for each demo and try it all for yourself!
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid2.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
Supported players: [http://www.videolan.org/vlc/ VLC 1.1+], [https://www.mozilla.com/en-US/firefox/ Firefox ], [http://www.chromium.org/Home Chrome ], [http://www.opera.com/ Opera]. Or see [http://www.webmproject.org/users/ other WebM] or [[TheoraSoftwarePlayers|other Theora]] players.

If you're having trouble with playback in a modern browser or player, please visit our [[Playback_Troubleshooting|playback troubleshooting and discussion]] page.
<br/>
<hr/>
<br/><br/><br/>
[[Image:Xiph_ep02_test.png|400px|right]]

Hi, I'm Monty Montgomery from [http://www.redhat.com/ Red Hat] and [http://xiph.org/ Xiph.Org].

A few months ago, I wrote
[http://people.xiph.org/~xiphmont/demo/neil-young.html an article on digital audio and why 24bit/192kHz music downloads don't make sense].
In the article, I
mentioned--almost in passing--that a digital waveform is
[http://people.xiph.org/~xiphmont/demo/neil-young.html#toc_sfam not a stairstep],
and you certainly don't get a stairstep when you
[[WikiPedia:Digital-to-analog_converter|convert from digital back to analog]].

Of everything in the entire article, '''that''' was the number one thing
people wrote about. In fact, more than half the mail I got was questions and
comments about basic digital signal behavior. Since there's interest, let's
take a little time to play with some ''simple'' digital signals.

==Veritas ex machina==
[[Image:Dsat_002.jpg|200px|right]]
[[Image:Dsat_003.jpg|200px|right]]
[[Image:Dsat_004.jpg|200px|right]]
[[Image:Dsat_005.jpg|200px|right]]

Pretend for a moment that we have no idea how digital signals really
behave. In that case it doesn't make sense for us to use digital test
equipment either. Fortunately for this exercise, there's still plenty
of working analog lab equipment out there.

First up, we need a [[WikiPedia:Function_generator|signal generator]] to provide us with analog input
signals--in this case, an
[http://www.home.agilent.com/en/pd-3325A%3Aepsg%3Apro-pn-3325A/synthesizer-function-generator?pm=PL&nid=-536900197.536896863&cc=SE&lc=swe HP3325]
from 1978. It's still a pretty good
generator, so if you don't mind the size, the weight, the power
consumption, and the noisy fan, you can find them on eBay... occasionally
for only slightly more than you'll pay for shipping.

Next, we'll observe our analog waveforms on [[WikiPedia:Oscilloscope_types#Cathode-ray_oscilloscope_.28CRO.29|analog oscilloscopes]],
like this Tektronix 2246 from the mid-90s, one of the last and very best analog scopes ever made. Every home lab should have one.

...and finally inspect the [[WikiPedia:Spectral_density#Electrical_engineering|frequency spectrum]] of our signals using an
[[WikiPedia:Spectrum_analyzer#Swept-tuned|analog spectrum analyzer]], this
[http://www.home.agilent.com/en/pd-3585A%3Aepsg%3Apro-pn-3585A/spectrum-analyzer-high-perf-20hz-40mhz?pm=PL&nid=-536900197.536897319&cc=SE&lc=swe HP3585]
from the same product line as
the signal generator. Like the other equipment here it has
[http://www.hp9845.net/9845/hardware/processors/ a rudimentary and hilariously large microcontroller],
but the signal path
from input to what you see on the screen is completely analog.

All of this equipment is vintage, but aside from its raw tonnage, the specs are still quite good.

At the moment, we have our signal generator set to output a nice 1 [[WikiPedia:Hertz#SI_multiples|kHz]]
sine wave at one [[WikiPedia:Volt|Volt]] [[WikiPedia:Amplitude#Root_mean_square_amplitude|RMS]].
We see the sine wave on the oscilloscope, can verify that it is indeed
1 kHz at 1 Volt RMS, which is 2.8 Volts
[[WikiPedia:Amplitude#Peak-to-peak_amplitude|peak-to-peak]],
and that matches the
measurement on the spectrum analyzer as well.

The analyzer also shows some low-level [[WikiPedia:White_noise|white noise]]
and just a bit of [[WikiPedia:Harmonic_distortion#Harmonic_distortion|harmonic distortion]],
with the highest peak about 70[[WikiPedia:Decibel|dB]] or so below
[[WikiPedia:Fundamental_frequency|the fundamental]].
Now, this doesn't matter at all in our demos, but I
wanted to point it out now just in case you didn't notice it until
later.

Now, we drop digital sampling in the middle.

For the conversion, we'll use a boring, consumer-grade, eMagic USB1
audio device. It's also more than ten years old at this point, and it's
getting obsolete.

A recent converter can easily have an order of magnitude better specs.
[[WikiPedia:Reconstruction_filter#Sampled_data_reconstruction_filters|Flatness]],
[[WikiPedia:Analog-to-digital_converter#Non-linearity|linearity]],
[[WikiPedia:Jitter#Sampling_jitter|jitter]],
[[WikiPedia:Noise_floor|noise behavior]],
[[WikiPedia:Digital-to-analog_converter#DAC_performance|everything]]...
you may not
have noticed. Just because we can measure an improvement doesn't
mean we can hear it, and even these old consumer boxes were already at
the edge of ideal transparency.

The eMagic connects to my ThinkPad, which displays a digital
waveform and spectrum for comparison, then the ThinkPad
sends the digital signal right back out to the eMagic for
re-conversion to analog and observation on the output scopes.

Input to output, left to right.
<br style="clear:both;"/>

==Stairsteps==
[[Image:Dsat 006.jpg|360px|right]]
[[Image:Dsat 007.png|360px|right]]
OK, it's go time. We begin by converting an analog signal to digital and
then right back to analog again with no other steps.

The signal generator is set to produce a 1kHz sine wave just like
before.

We can see our analog sine wave on our input-side oscilloscope.

We digitize our signal to
[[Videos/A_Digital_Media_Primer_For_Geeks#Raw_.28digital_audio.29_meat|16 bit PCM at 44.1kHz]],
same as on a CD.
The spectrum of the digitized signal matches what we saw earlier

and what we see now on the analog spectrum analyzer, aside from its
[[WikiPedia:High_impedance|high-impedance input]] being just a smidge noisier.

For now, the waveform display shows our digitized sine wave as a
stairstep pattern, one step for each sample.

And when we look at the output signal that's been converted
from digital back to analog, we see...

It's exactly like the original sine wave. No stairsteps.

OK, 1 kHz is still a fairly low frequency, maybe the stairsteps are just
hard to see or they're being smoothed away. Fair enough. Let's choose
a higher frequency, something close to [[WikiPedia:Nyquist_frequency|Nyquist]], say 15kHz.

Now the sine wave is represented by less than three samples per cycle, and...

the digital waveform looks pretty awful. Well, looks
can be deceiving. The analog output...

is still a perfect sine wave, exactly like the original.

Let's keep going up.

Let's see if I can do this without blocking any cameras.

16kHz.... 17kHz... 18kHz... 19kHz...

20kHz. Welcome to the upper limits of human hearing. The output
waveform is still perfect. No jagged edges, no dropoff, no stairsteps.

So where'd the stairsteps go? Don't answer, it's a trick question.
They were never there.

Drawing a digital waveform as a stairstep... was wrong to begin with.

Why? A stairstep is a continuous-time function. It's jagged, and it's
piecewise, but it has a defined value at every point in time.

A sampled signal is entirely different. It's discrete-time; it's only
got a value right at each instantaneous sample point and it's
undefined, there is no value at all, everywhere between. A
discrete-time signal is properly drawn as a lollipop graph.

The continuous, analog counterpart of a digital signal passes
smoothly through each sample point, and that's just as true for high
frequencies as it is for low.

Now, the interesting and not at all obvious bit is: [[WikiPedia:Nyquist%E2%80%93Shannon_sampling_theorem|there's only one
bandlimited signal that passes exactly through each sample point]]. It's
a unique solution. So if you sample a bandlimited signal and then
convert it back, the original input is also the only possible output.

[[Image:Dsat 008.png|360px|right]]

And before you say, "oh, I can draw a different signal that passes
through those points", well, yes you can, but if it differs even
minutely from the original, it includes frequency content at or beyond
Nyquist, breaks the bandlimiting requirement and isn't a valid
solution.

So how did everyone get confused and start thinking of digital signals
as stairsteps? I can think of two good reasons.

First: it's easy enough to convert a sampled signal to a true stairstep. Just
extend each sample value forward until the next sample period. This is
called a [[WikiPedia:Zero-order hold|zero-order hold]], and it's an important part of how some
digital-to-analog converters work, especially the simplest ones.

So, anyone who looks up [[WikiPedia:Digital-to-analog_converter#Practical_operation|digital-to-analog converter or
digital-to-analog conversion]] is probably going to see a diagram of a
stairstep waveform somewhere, but that's not a finished conversion,
and it's not the signal that comes out.

Second, and this is probably the more likely reason, engineers who
supposedly know better, like me, draw stairsteps even though they're
technically wrong. It's a sort of like a one-dimensional version of
[[WikiPedia:MacPaint#Development|fat bits in an image editor]].

Pixels aren't squares either, they're samples of a 2-dimensional
function space and so they're also, conceptually, infinitely small
points. Practically, it's a real pain in the ass to see or manipulate
infinitely small anything, so big squares it is. Digital stairstep
drawings are exactly the same thing.

It's just a convenient drawing. The stairsteps aren't really there.

==Bit-depth==
[[Image:Dsat_009.jpg|360px|right]]
[[Image:Dsat_010.jpg|260px|right]]

When we convert a digital signal back to analog, the result is
''also'' smooth regardless of the [[WikiPedia:Audio_bit_depth|bit depth]]. 24 bits or 16 bits...
or 8 bits... it doesn't matter.

So does that mean that the digital bit depth makes no difference at
all? Of course not.

Channel 2 here is the same sine wave input, but we quantize with
[[WikiPedia:Dither|dither]] down to 8 bits.

On the scope, we still see a nice
smooth sine wave on channel 2. Look very close, and you'll also see a
bit more noise. That's a clue.

If we look at the spectrum of the signal... aha! Our sine wave is
still there unaffected, but the noise level of the 8-bit signal on
the second channel is much higher!

And that's the difference the number of bits makes. That's it!

When we digitize a signal, first we sample it. The
sampling step is perfect; it loses nothing. But then we [[WikiPedia:Quantization_(sound_processing)|quantize]] it,
and [[WikiPedia:Quantization_error|quantization adds noise]].

The number of bits determines how much noise and so the level of the
noise floor.

What does this dithered quantization noise sound like? Let's listen
to our 8-bit sine wave.

That may have been hard to hear anything but the tone. Let's listen
to just the noise after we notch out the sine wave and then bring the
gain up a bit because the noise is quiet.

Those of you who have used analog recording equipment may have just
thought to yourselves, "My goodness! That sounds like tape hiss!"
Well, it doesn't just sound like tape hiss, it acts like it too, and
if we use a [[WikiPedia:Dither#Different_types|gaussian dither]] then it's
[[WikiPedia:Central_limit_theorem|mathematically equivalent]] in every way. It ''is'' tape hiss.

Intuitively, that means that we can measure tape hiss and thus the noise floor
of [[WikiPedia:Magnetic_tape_sound_recording|magnetic audio tape]]
in [[WikiPedia:Shannon–Hartley_theorem#Examples|bits instead of decibels]], in order to put things in a
digital perspective. [[WikiPedia:Compact cassettes|Compact cassettes]] (for those of you who are old enough to remember them) could reach as
deep as 9 bits in perfect conditions, though 5 to 6 bits was
more typical, especially if it was a recording made on a
[[WikiPedia:Cassette_deck|tape deck]]. That's right... your mix tapes were only about 6 bits
deep... if you were lucky!

The very best professional [[WikiPedia:Reel-to-reel_audio_tape_recording|open reel tape]] used in studios could barely
hit... any guesses? 13 bits ''with'' [[WikiPedia:Reel-to-reel_audio_tape_recording#Noise_reduction|advanced noise reduction]]. And
that's why seeing '[[WikiPedia:SPARS_code|D D D]]' on a [[WikiPedia:Compact_disk|Compact Disc]] used to be such a big,
high-end deal.

==Dither==
[[Image:Dsat_011.png|360px|right]]

I keep saying that I'm quantizing with [[Wikipedia:dither|dither]], so what is dither
exactly and, more importantly, what does it do?

The simple way to quantize a signal is to choose the digital
amplitude value closest to the original analog amplitude. [[WikiPedia:Rounding|Obvious]],
right? Unfortunately, the exact noise you get from this simple
quantization scheme depends somewhat on the input signal,

so we may get noise that's inconsistent, or causes distortion, or is
undesirable in some other way.
<br style="clear:both;"/>
<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Cameron Nicklaus Christou's thesis [http://uwspace.uwaterloo.ca/bitstream/10012/3867/1/thesis.pdf Optimal Dither and Noise Shaping in Image Processing] provides an ''excellent'' explanation of dither and noise shaping.
</div></center>

Dither is specially-constructed noise that substitutes for the noise
produced by simple quantization. Dither doesn't [[WikiPedia:Sound_masking|drown out or mask]]
quantization noise, it actually replaces it with noise characteristics
of our choosing that aren't influenced by the input.

Let's ''watch'' what dither does. The signal generator has too much noise for this test so we'll produce a mathematically perfect sine wave with the ThinkPad and quantize it to 8 bits with dithering.

We see a nice sine wave on the waveform display and output scope and, once the analog spectrum analyzer catches up...
a clean frequency peak with a uniform noise floor on both spectral displays
just like before. Again, this is with dither.

Now I turn dithering off.

The quantization noise, that dither had spread out into a nice, flat noise
floor, piles up into harmonic distortion peaks. The noise floor is
lower, but the level of distortion becomes nonzero, and the distortion
peaks sit higher than the dithering noise did.

At 8 bits this effect is exaggerated. At 16 bits,
even without dither, harmonic distortion is going to be so low as to
be completely inaudible.

Still, we can use dither to eliminate it completely if we so choose.

Turning the dither off again for a moment, you'll notice that the
absolute level of distortion from undithered quantization stays
approximately constant regardless of the input amplitude.
But when the signal level drops below a half a bit, everything
quantizes to zero.

In a sense, everything quantizing to zero is just 100% distortion!
Dither eliminates this distortion too. We reenable dither
and ... there's our signal back at 1/4 bit, with our nice flat noise floor.

The noise floor doesn't have to be flat. Dither is noise of our
choosing, so let's choose a noise as [http://www.acoustics.salford.ac.uk/res/cox/sound_quality/?content=subjective inoffensive] and
[[WikiPedia:Absolute_threshold_of_hearing|difficult to notice]]
as possible.

Our hearing is most sensitive in the midrange from 2kHz to 4kHz,
so that's where background noise is going to be the most obvious.
We can [[WikiPedia:Noise_shaping|shape dithering noise]] away from sensitive frequencies to where
hearing is less sensitive, usually the highest frequencies.

16-bit dithering noise is normally much too quiet to hear at all, but
let's listen to our noise shaping example, again with the gain
brought way up...

Lastly, dithered quantization noise ''is'' higher [[WikiPedia:power|Sound_power]] overall
than undithered quantization noise even when it sounds quieter, and
you can see that on a [[WikiPedia:VU_meter|VU meter]] during passages of near-silence. But
dither isn't only an on or off choice. We can reduce the dither's
power to balance less noise against a bit of distortion to minimize
the overall effect.

We'll also [[WikiPedia:Amplitude_modulation|modulate the input signal]] like this to show how a varying input affects the quantization noise. At
full dithering power, the noise is uniform, constant, and featureless
just like we expect:

As we reduce the dither's power, the input increasingly
affects the amplitude and the character of the quantization noise.
Shaped dither behaves similarly, but noise shaping lends one more nice
advantage. To make a long story short, it can use a somewhat lower
dither power before the input has as much effect on the output.

Despite all the time I just spent on dither, we're talking about
differences that start 100 decibels and more below [[WikiPedia:Full_scale|full scale]]. Maybe
if the CD had been
[http://www.research.philips.com/technologies/projects/cd/index.html 14 bits as originally designed],
dither ''might'' be
more important. Maybe. At 16 bits, really, it's mostly a wash. You
can think of dither as an insurance policy that gives several extra
decibels of dynamic range just in case. The simple fact is, though, no
one ever ruined a great recording by not dithering the final master.

==Bandlimitation and timing==
[[image:Dsat_013.jpg|360px|right]]
[[image:Dsat_014.gif|360px|right]]

We've been using [[WikiPedia:Sine_wave|sine waves]]. They're the obvious choice when what we
want to see is a system's behavior at a given isolated frequency. Now
let's look at something a bit more complex. What should we expect to
happen when I change the input to a [[WikiPedia:Square_wave|square wave]]...

The input scope confirms our 1kHz square wave. The output scope shows..

Exactly what it should.
...
What is a square wave really?

Well, we can say it's a waveform that's
some positive value for half a cycle and then transitions
instantaneously to a negative value for the other half. But that doesn't
really tell us anything useful about how this input
becomes this output .

Then we remember that [any waveform is also [[WikiPedia:Fourier_series|the sum of discrete frequencies]],
and a square wave is particularly simple sum: a fundamental and an
infinite series of [[WikiPedia:Even_and_odd_functions#Harmonics|odd harmonics]]. Sum them all up, you get a
square wave.

At first glance, that doesn't seem very useful either. You have to sum
up an infinite number of harmonics to get the answer. Ah, but we don't
have an infinite number of harmonics.

We're using a quite sharp [[WikiPedia:Low-pass_filter|anti-aliasing filter]] that cuts off right
above 20kHz, so our signal is [[WikiPedia:Bandlimiting|bandlimited]], which means we get this:

..and that's exactly what we see on the output scope.

The rippling you see around sharp edges in a bandlimited signal is
called the [[WikiPedia:/Gibbs_phenomenon|Gibbs effect]]. It happens whenever you slice off part of the
frequency domain in the middle of nonzero energy.

The usual rule of thumb you'll hear is "the sharper the cutoff, the
stronger the rippling", which is approximately true, but we have to be
careful how we think about it.
For example... what would you expect our quite sharp anti-aliasing filter
to do if I run our signal through it a second time?

Aside from adding a few fractional cycles of delay, the answer is...
nothing at all. The signal is already bandlimited. Bandlimiting it
again doesn't do anything. A second pass can't remove frequencies
that we already removed.

And that's important. People tend to think of the ripples as
a kind of [[WikiPedia:Sonic_artifact|artifact]] that's added by anti-aliasing and [[WikiPedia:Reconstruction_filter|anti-imaging]]
filters, implying that the ripples get worse each time the signal
passes through. We can see that in this case that didn't happen. So
was it really the filter that added the ripples the first time
through? No, not really. It's a subtle distinction, but Gibbs effect
ripples aren't added by filters, they're just part of what a
bandlimited signal ''is''.

Even if we synthetically construct what looks like a perfect digital
square wave,

it's still limited to the channel bandwidth. Remember,
the stairstep representation is misleading.

What we really have here are instantaneous sample points,

and only one bandlimited signal fits those points. All we did when we
drew our apparently perfect square wave was line up the sample points
just right so it appeared that there were no ripples if we played
[[WikiPedia:Interpolation|connect-the-dots]].

But the original bandlimited signal, complete with ripples, was
still there.

And that leads us to one more important point. You've probably heard
that the timing precision of a digital signal is limited by its sample
rate; put another way,

that digital signals can't represent anything that falls between the
samples.. implying that [[WikiPedia:Dirac_delta_function|impulses]] or
[[WikiPedia:Synthesizer#ADSR_envelope|fast attacks]] have to align exactly
with a sample, or the timing gets mangled... or they just disappear.

At this point, we can easily see why that's wrong.

Again, our input signals are bandlimited. And digital signals are
samples, not stairsteps, not 'connect-the-dots'. We most certainly
can, for example, put the rising edge of our bandlimited square wave
anywhere we want between samples.

It's represented perfectly and it's reconstructed perfectly.

==Epilogue==

[[Image:Moffey.jpg|360px|right]]

Just like in [[Videos/A_Digital_Media_Primer_For_Geeks|the previous episode]], we've covered a broad range of
topics, and yet barely scratched the surface of each one. If anything, my
sins of omission are greater this time around... but this is a good
stopping point.

Or maybe, a good starting point. Dig deeper. Experiment. I chose my
demos very carefully to be simple and give clear results. You can
reproduce every one of them on your own if you like. But let's face
it, sometimes we learn the most about a spiffy toy by breaking it open
and studying all the pieces that fall out. And that's OK, we're
engineers. Play with the demo parameters, hack up the code, set up
alternate experiments. The source code for everything, including the
little pushbutton demo application, is up at xiph.org.

In the course of experimentation, you're likely to run into something
that you didn't expect and can't explain. Don't worry! My earlier
snark aside, Wikipedia is fantastic for exactly this kind of casual
research. And, if you're really serious about understanding signals,
several universities have advanced materials online, such as the
[http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/index.htm 6.003]
and
[http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-007-electromagnetic-energy-from-motors-to-lasers-spring-2011 6.007]
Signals and Systems modules at MIT OpenCourseWare. And of
course, there's always the [http://webchat.freenode.net/?channels=xiph community here at Xiph.Org].

Digging deeper or not, I am out of coffee, so, until next time, happy
hacking!

==Credits==

Written by: Christopher (Monty) Montgomery and the Xiph.Org Community

Special thanks to:
*Heidi Baumgartner, for the second Tektronix oscilloscope
*Gregory Maxwell and Dr. Timothy Terriberry, for additional technical review

Intro, title and credits music:<br>
"Andy Warhol Is Gone", by Lousy Robot<br>
Used by permission of Lousy Robot.<br>
Original source track All Rights Reserved.<br>
[http://www.lousyrobot.com www.lousyrobot.com]

This Video Was Produced Entirely With Free and Open Source Software:<br>

*[http://www.gnu.org/ GNU]<br>
*[http://www.linux.org/ Linux]<br>
*[http://fedoraproject.org/ Fedora]<br>
*[http://cinelerra.org/ Cinelerra]<br>
*[http://www.gimp.org/ The Gimp]<br>
*[http://audacity.sourceforge.net/ Audacity]<br>
*[http://svn.xiph.org/trunk/postfish/README Postfish]<br>
*[http://gstreamer.freedesktop.org/ Gstreamer]<br>

All trademarks are the property of their respective owners.

*''Complete video'' [http://creativecommons.org/licenses/by-sa/3.0/legalcode CC-BY-SA]<br>
*''Text transcript and Wiki edition'' [http://creativecommons.org/licenses/by-sa/3.0/legalcode CC-BY-SA]<br>

A Co-Production of Xiph.Org and Red Hat, Inc.<br>
(C) 2012-2013, Some Rights Reserved<br>

== Use The Source Luke ==

As stated in the Epilogue, everything that appears in the video demos is driven by open source software, which means the source is both available for inspection and freely usable by the community. The Thinkpad that appears in the video was running Fedora 17 and Gnome Shell (Gnome 3). The demonstration software does not require Fedora specifically, but it does require Gnu/Linux to run in its current form.

=== The Spectrum and Waveform Viewer ===

The realtime software spectrum analyzer application that appears in the video was a preexisting application that was dusted off and updated for use in the video. The waveform viewer (effectively a simple software oscilloscope) was written from scratch making use of some of the internals from the spectrum analyzer application. Both are available from Xiph.Org svn:

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
*Source for the Spectrum and Waveform applications is found at:
https://svn.xiph.org/trunk/spectrum/
*The source can be checked out of svn using the following command line:
svn co https://svn.xiph.org/trunk/spectrum
*Trac is a convenient way to browse the source without checking out a copy:
https://trac.xiph.org/browser/trunk/spectrum
</div></center>

Spectrum and Waveform both expect an input stream on the command line, either as raw data or as a WAV file.

=== GTK-Bounce ===

The touch-controlled application used in the video is named 'gtk-bounce' and was custom-written for the sole purpose of the in-video demonstrations. It is so named because, for the most part, all it does is read the input from an audio device, and then immediately write the same data back out for playback. It also forwards a copy of this data to up to two external monitoring applications, and in several demos, applies simple filters or generates simple waveforms. It includes several demos not included in the video.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Source for gtk-bounce is found at:'''
https://svn.xiph.org/trunk/Xiph-episode-II/bounce/
The source can be checked out of svn using the following command line:
svn co https://svn.xiph.org/trunk/Xiph-episode-II/bounce/
Trac is a convenient way to browse the source without checking out a copy:
https://trac.xiph.org/browser/Xiph-episode-II/bounce/
</div></center>

The application is somewhat hardwired for specific demo purposes, but most of the hardwired settings can be found at the top of each source file. As found in SVN, the application expects an ALSA hardware audio device at hw:1, and if none if found, it will wait for one to appear. Once a sound device is successfully initialized, it expects to find and open two pipes named pipe0 and pipe1 for output in the current directory. In the video, the waveform and spectrum applications are started to take input from pipe0 and pipe1 respectively. In most of the demos, the output sent to the two pipes is identical, and matches the output data sent to the hardware device for conversion to analog. The only exception is the tenth demo panel (which does not appear in the video) where gtk-bounce can be set to monitor the hardware inputs to the pipes rather than the outputs.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Assuming gtk-bounce, spectrum and waveform have been checked out and built, the configuration seen in the video can be started using the following commands:'''
# make the pipe fifos for the applications to communicate (only needs to be done once)
mkfifo pipe0; mkfifo pipe1
# start all three applications
waveform pipe0 & spectrum pipe1 & gtk-bounce &
</div></center>

Videos/Digital Show and Tell

2013-02-25T03:23:14Z

Mindspillage: minor proofreading fixes

<<< Intro >>>>

Hi, I'm Monty Montgomery from Red Hat and Xiph.Org.

A few months ago, I wrote an article on digital audio and why
24bit/192kHz music downloads don't make sense. In the article, I
mentioned--almost in passing--that a digital waveform is not a
stairstep, and you certainly don't get a stairstep when you convert
from digital back to analog.

Of everything in the entire article, *that* was the number one thing
people wrote about. In fact, more than half the mail I got was questions and
comments about basic digital signal behavior. Since there's interest,
let's take a little time to play with some _simple_ digital signals.

<<<< veritas ex machina >>>

Pretend for a moment that we have no idea how digital signals really
behave. In that case it doesn't make sense for us to use digital test
equipment either. Fortunately for this exercise, there's still plenty
of working analog lab equipment out there.

[[ close on 3325 ]

First up, we need a signal generator to provide us with analog input
signals--in this case, an HP3325 from 1978. It's still a pretty good
generator, so if you don't mind the size, the weight, the power
consumption, and the noisy fan, you can find them on eBay... occasionally
for only slightly more than you'll pay for shipping.

[[ close on 2246 ]]

Next, we'll observe our analog waveforms on analog oscilloscopes, like this
Tektronix 2246 from the mid-90s, one of the last and very best analog
scopes ever made. Every home lab should have one.

[[ close on 3585]]

...and finally inspect the frequency spectrum of our signals using an
analog spectrum analyzer, this HP3585 from the same product line as
the signal generator. Like the other equipment here it has a
rudimentary and hilariously large microcontroller, but the signal path
from input to what you see on the screen is completely analog.

All of this equipment is vintage, but aside from its raw tonnage, the
specs are still quite good.

[[ out ]]

At the moment, we have our signal generator set to output a nice 1kHz
sine wave at one volt RMS,

we see the sine wave on the oscilloscope, can verify that it is indeed
1kHz at one volt RMS, which is 2.8V peak-to-peak, and that matches the
measurement on the spectrum analyzer as well.

The analyzer also shows some low-level white noise and just a bit of
harmonic distortion, with the highest peak about 70dB or so below the
fundamental. Now, this doesn't matter at all in our demos, but I
wanted to point it out now just in case you didn't notice it until
later.

[[ cut to complete setup ]]

Now, we drop digital sampling in the middle.

For the conversion, we'll use a boring, consumer-grade, eMagic USB1
audio device. It's also more than ten years old at this point, and it's
getting obsolete.

A recent converter can easily have an order of magnitude better specs.
Flatness, linearity, jitter, noise behavior, everything... you may not
have noticed. Just because we can measure an improvement doesn't
mean we can hear it, and even these old consumer boxes were already at
the edge of ideal transparency.

[[out to see emagic initialize and digital waveform appear on TP ]]

The eMagic connects to my ThinkPad, which displays a digital
waveform and spectrum for comparison, then the ThinkPad
sends the digital signal right back out to the eMagic for
re-conversion to analog and observation on the output scopes.

Input to output, left to right.

<<<< stairsteps >>>>

OK, it's go time. We begin by converting an analog signal to digital and
then right back to analog again with no other steps.

[[close to 3325]]
The signal generator is set to produce a 1kHz sine wave just like
before.

[[close to input scope]]
We can see our analog sine wave on our input-side oscilloscope.

[[close to TP: spectrum]]
We digitize our signal to 16 bit PCM at 44.1kHz, same as on a CD.
The spectrum of the digitized signal matches what we saw earlier

[[close to SA]]
and what we see now on the analog spectrum analyzer, aside from its
high-impedance input being just a smidge noisier.

[[close to TP ; overview/waveform ]]
For now, the waveform display shows our digitized sine wave as a
stairstep pattern, one step for each sample.

[[ out ]]
And when we look at the output signal that's been converted
from digital back to analog, we see...

[[close to output scope: press CH1 button to show waveform]]
It's exactly like the original sine wave. No stairsteps.

[[ out ]]
OK, 1kHz is still a fairly low frequency, maybe the stairsteps are just
hard to see or they're being smoothed away. Fair enough. Let's choose
a higher frequency, something close to Nyquist, say 15kHz.

[[set 3325 to 15kHz ]]
Now the sine wave is represented by less than three samples per cycle, and...

[[close to TP]]
the digital waveform looks pretty awful. Well, looks
can be deceiving. The analog output...

[[close to output scope]]
is still a perfect sine wave, exactly like the original.

Let's keep going up.

Let's see if I can do this without blocking any cameras.

16kHz.... 17kHz... 18kHz... 19kHz...

20kHz. Welcome to the upper limits of human hearing. The output
waveform is still perfect. No jagged edges, no dropoff, no stairsteps.

So where'd the stairsteps go? Don't answer, it's a trick question.
They were never there.

Drawing a digital waveform as a stairstep... was wrong to begin with.

Why? A stairstep is a continuous-time function. It's jagged, and it's
piecewise, but it has a defined value at every point in time.

A sampled signal is entirely different. It's discrete-time; it's only
got a value right at each instantaneous sample point and it's
undefined, there is no value at all, everywhere between. A
discrete-time signal is properly drawn as a lollipop graph.

The continuous, analog counterpart of a digital signal passes
smoothly through each sample point, and that's just as true for high
frequencies as it is for low.

Now, the interesting and not at all obvious bit is: there's only one
bandlimited signal that passes exactly through each sample point. It's
a unique solution. So if you sample a bandlimited signal and then
convert it back, the original input is also the only possible output.

And before you say, "oh, I can draw a different signal that passes
through those points", well, yes you can, but if it differs even
minutely from the original, it includes frequency content at or beyond
Nyquist, breaks the bandlimiting requirement and isn't a valid
solution.

[[ out ]]
So how did everyone get confused and start thinking of digital signals
as stairsteps? I can think of two good reasons.

[[close to TP; freeze display; draw in zero-order]]
First: it's easy enough to convert a sampled signal to a true stairstep. Just
extend each sample value forward until the next sample period. This is
called a zero-order hold, and it's an important part of how some
digital-to-analog converters work, especially the simplest ones.

[[ Wikipedia DAC lookup + scroll down to hold image]]
So, anyone who looks up digital-to-analog converter or
digital-to-analog conversion is probably going to see a diagram of a
stairstep waveform somewere, but that's not a finished conversion,
and it's not the signal that comes out.

[[ out ]]
Second, and this is probably the more likely reason, engineers who
supposedly know better, like me, draw stairsteps even though they're
technically wrong. It's a sort of like a one-dimensional version of
fat bits in an image editor.

[[gimp RMD animation]]]
Pixels aren't squares either, they're samples of a 2-dimensional
function space and so they're also, conceptually, infinitely small
points. Practically, it's a real pain in the ass to see or manipulate
infinitely small anything, so big squares it is. Digital stairstep
drawings are exactly the same thing.

[[ out ]]
It's just a convenient drawing. The stairsteps aren't really there.

<<< bit-depth >>>

When we convert a digital signal back to analog, the result is
_also_ smooth regardless of the bit depth. 24 bits or 16 bits...
or 8 bits... it doesn't matter.

So does that mean that the digital bit depth makes no difference at
all? Of course not.

Channel 2 here is the same sine wave input, but we quantize with
dither down to eight bits.

On the scope, we still see a nice
smooth sine wave on channel 2. Look very close, and you'll also see a
bit more noise. That's a clue.

If we look at the spectrum of the signal... aha! Our sine wave is
still there unaffected, but the noise level of the eight-bit signal on
the second channel is much higher!

And that's the difference the number of bits makes. That's it!

When we digitize a signal, first we sample it. The
sampling step is perfect; it loses nothing. But then we quantize it,
and quantization adds noise.

[[panel 2; demonstrate changing bit depth on tablet ]]

The number of bits determines how much noise and so the level of the
noise floor. [[demonstrate changing bit depth on tablet]].

[[ out ]]

What does this dithered quantization noise sound like? Let's listen
to our eight-bit sine wave.

[[audio: eight bit sine]]

That may have been hard to hear anything but the tone. Let's listen
to just the noise after we notch out the sine wave and then bring the
gain up a bit because the noise is quiet.

[[audio: hit notch + gain button]]

Those of you who have used analog recording equipment may have just
thought to yourselves, "My goodness! That sounds like tape hiss!"
Well, it doesn't just sound like tape hiss, it acts like it too, and
if we use a gaussian dither then it's mathematically
equivalent in every way. It _is_ tape hiss.

Intuitively, that means that we can measure tape hiss and thus the noise floor
of magnetic audio tape in bits instead of decibels, in order to put things in a
digital perspective. Compact cassettes...

[[ reveal cassette ]]

for those of you who are old enough to remember them, could reach as
deep as nine bits in perfect conditions, though five to six bits was
more typical, especially if it was a recording made on a tape
deck. That's right... your mix tapes were only about six bits
deep... if you were lucky!

The very best professional open reel tape used in studios could barely
hit... any guesses? 13 bits _with_ advanced noise reduction. And
that's why seeing 'D D D' on a Compact Disc used to be such a big,
high-end deal.

<<< dither >>>

I keep saying that I'm quantizing with dither, so what is dither
exactly and, more importantly, what does it do?

[[ Illustration: quantization ]]

The simple way to quantize a signal is to choose the digital
amplitude value closest to the original analog amplitude. Obvious,
right? Unfortunately, the exact noise you get from this simple
quantization scheme depends somewhat on the input signal,

[[Illustration: correlated quantization noise ]]

so we may get noise that's inconsistent, or causes distortion, or is
undesirable in some other way.

[show/attribute the dither paper]
Dither is specially-constructed noise that substitutes for the noise
produced by simple quantization. Dither doesn't drown out or mask
quantization noise, it actually replaces it with noise characteristics
of our choosing that aren't influenced by the input.

[[out]]

Let's _watch_ what dither does. The signal generator has too much
noise for this test so

[[close: panel 3]]

we'll produce a mathematically perfect sine wave with the ThinkPad
[[press]]

and quantize it to eight bits [[press]]

with dithering. [[press]]

We see a nice sine wave on the waveform display

[[ show outscope ]] and output scope

[[ show analyzer]] and, once the analog spectrum analyzer catches up...

[[time accel sweep]] a clean frequency peak with a uniform noise floor
on both spectral displays

[[ overview: spectrum ]] just like before. Again, this is with dither.

Now I turn dithering off. [[ deactivate dither ]]

The quantization noise, that dither had spread out into a nice, flat noise
floor, piles up into harmonic distortion peaks. The noise floor is
lower, but the level of distortion becomes nonzero, and the distortion
peaks sit higher than the dithering noise did.

At eight bits this effect is exaggerated. At sixteen bits, [[click 16]]

even without dither, harmonic distortion is going to be so low as to
be completely inaudible.

[[draw line across -100]]

Still, we can use dither to eliminate it completely if we so choose.

Turning the dither off again for a moment, you'll notice that the
absolute level of distortion from undithered quantization stays
approximately constant regardless of the input amplitude.

[[ overview: waveform ]]

But when the signal level drops below a half a bit, everything
quantizes to zero.

[[ overview: spectrum ]]

In a sense, everything quantizing to zero is just 100% distortion!
Dither eliminates this distortion too. We reenable dither
and...

[[dither on]]

there's our signal back at 1/4 bit, with our nice flat noise floor.

[[ out ]]

The noise floor doesn't have to be flat. Dither is noise of our
choosing, so let's choose a noise as inoffensive and difficult to
notice as possible.

[[panel 5]]

Our hearing is most sensitive in the midrange from 2kHz to 4kHz,
so that's where background noise is going to be the most obvious.

[[annotate: underline 2-4kHz]]

[[click shaped]

We can shape dithering noise away from sensitive frequencies to where
hearing is less sensitive, usually the highest frequencies.

[[annotate: arrow to HF]]

[[out]]
16-bit dithering noise is normally much too quiet to hear at all, but
let's listen to our noise shaping example, again with the gain
brought way up...

[[close]]
[[unshaped white to shaped]]

[[out]] Lastly, dithered quantization noise _is_ higher power overall
than undithered quantization noise even when it sounds quieter, and
you can see that on a VU meter during passages of near-silence. But
dither isn't only an on or off choice. We can reduce the dither's
power to balance less noise against a bit of distortion to minimize
the overall effect.

[[ panel 6 audio :: flat, unmodulated ]]
We'll also modulate the input signal like this:
[[ panel 6 audio :: flat, modulated ]]
...to show how a varying input affects the quantization noise. At
full dithering power, the noise is uniform, constant, and featureless
just like we expect:
[[ panel 6 audio :: flat, modulated, notch ]]
As we reduce the dither's power, the input increasingly
affects the amplitude and the character of the quantization noise:
[[ panel 6 audio :: flat, modulated, notch ]]
Shaped dither behaves similarly, but noise shaping lends one more nice
advantage. To make a long story short, it can use a somewhat lower
dither power before the input has as much effect on the output.
[[ panel 6 audio :: shaped, modulated, notch ]]
[[ reset panel :: shaped, unmodulated, no notch ]]

[[out]]

Despite all the time I just spent on dither, we're talking about
differences that start 100 decibels and more below full scale. Maybe
if the CD had been 14 bits as originally designed, dither _might_ be
more important. Maybe. At 16 bits, really, it's mostly a wash. You
can think of dither as an insurance policy that gives several extra
decibels of dynamic range just in case. The simple fact is, though, no
one ever ruined a great recording by not dithering the final master.

<<<< bandlimitation and timing >>>>

We've been using sine waves. They're the obvious choice when what we
want to see is a system's behavior at a given isolated frequency. Now
let's look at something a bit more complex. What should we expect to
happen when I change the input to a square wave...

[[close to sig analyzer-- press the button]]

[[close to input scope]]
The input scope confirms our 1kHz square wave. The output scope shows..

[[close to output scope]]
Exactly what it should.
...
What is a square wave really?
[[illustrate]]

Well, we can say it's a waveform that's
some positive value for half a cycle and then transitions
instantaneously to a negative value for the other half. But that doesn't
really tell us anything useful about how this input [[close/point]]
becomes this output [[close/point]].

[[animated diagram]]
Then we remember that any waveform is also the sum of discrete frequencies,
and a square wave is particularly simple sum: a fundamental and an
infinite series of odd harmonics. Sum them all up, you get a
squarewave.

[[out]]
At first glance, that doesn't seem very useful either. You have to sum
up an infinite number of harmonics to get the answer. Ah, but we don't
have an infinite number of harmonics.

[[close to panel, annotate circling cutoff, and line at 20kHz on spectrum]]

We're using a quite sharp anti-aliasing filter that cuts off right
above 20kHz, so our signal is bandlimited, which means we get this:

[[diagram]]

..and that's exactly what we see on the output scope.
[[pan/fade to scope display showing they line up perfectly]]

The rippling you see around sharp edges in a bandlimited signal is
called the Gibbs effect. It happens whenever you slice off part of the
frequency domain in the middle of nonzero energy.

[[out]]
The usual rule of thumb you'll hear is "the sharper the cutoff, the
stronger the rippling", which is approximately true, but we have to be
careful how we think about it.

For example... what would you expect our quite sharp anti-aliasing filter
to do if I run our signal through it a second time?

[[ plug plug go]]
[[outscope]]

Aside from adding a few fractional cycles of delay, the answer is...
nothing at all. The signal is already bandlimited. Bandlimiting it
again doesn't do anything. A second pass can't remove frequencies
that we already removed.

[[out]] And that's important. People tend to think of the ripples as
a kind of artifact that's added by anti-aliasing and anti-imaging
filters, implying that the ripples get worse each time the signal
passes through. We can see that in this case that didn't happen. So
was it really the filter that added the ripples the first time
through? No, not really. It's a subtle distinction, but Gibbs effect
ripples aren't added by filters, they're just part of what a
bandlimited signal _is_.

[[close: panel 8]]

Even if we synthetically construct what looks like a perfect digital
square wave,

[[ turn on digital 'squarewave' ]]

it's still limited to the channel bandwidth. Remember,
the stairstep representation is misleading.

[[go to lollipop]]

What we really have here are instantaneous sample points,

[[to diagram, trace original ]]

and only one bandlimited signal fits those points. All we did when we
drew our apparently perfect square wave was line up the sample points
just right so it appeared that there were no ripples if we played
connect-the-dots.

[[ diagram: shift samples forward and back; fade to waveform display
showing same ]]

But the original bandlimited signal, complete with ripples, was
still there.

[[ show output scope ]]
[[ out ]]

And that leads us to one more important point. You've probably heard
that the timing precision of a digital signal is limited by its sample
rate; put another way,

[[diagram]]

that digital signals can't represent anything that falls between the
samples.. implying that impulses or fast attacks have to align exactly
with a sample, or the timing gets mangled... or they just disapper.

[[ scribble it out ]]

At this point, we can easily see why that's wrong.

[[ diagram: both an edge and an impulse ]]

Again, our input signals are bandlimited. And digital signals are
samples, not stairsteps, not 'connect-the-dots'. We most certainly
can, for example, put the rising edge of our bandlimited square wave
anywhere we want between samples.

It's represented perfectly [[show on the waveform display, move slider]]
and it's reconstructed perfectly [[show on output scope with moving slider]].

<<< epilogue >>>

[[ back in :20 sign ]]

Just like in the previous episode, we've covered a broad range of
topics, and yet barely scratched the surface of each one. If anything, my
sins of omission are greater this time around... but this is a good
stopping point.

Or maybe, a good starting point. Dig deeper. Experiment. I chose my
demos very carefully to be simple and give clear results. You can
reproduce every one of them on your own if you like. But let's face
it, sometimes we learn the most about a spiffy toy by breaking it open
and studying all the pieces that fall out. And that's OK, we're
engineers. Play with the demo parameters, hack up the code, set up
alternate experiments. The source code for everything, including the
little pushbutton demo application, is up at xiph.org.

In the course of experimentation, you're likely to run into something
that you didn't expect and can't explain. Don't worry! My earlier
snark aside, Wikipedia is fantastic for exactly this kind of casual
research. And, if you're really serious about understanding signals,
several universities have advanced materials online, such as the 6.003
and 6.007 Signals and Systems modules at MIT OpenCourseWare. And of
course, there's always the community here at Xiph.Org.

Digging deeper or not, I am out of coffee, so, until next time, happy
hacking!

Videos/A Digital Media Primer For Geeks

2010-09-22T06:47:31Z

Mindspillage: /* Introduction */ trailing quote

<small>''Wiki edition''</small>
[[Image:Dmpfg_001.jpg|360px|right]]

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
Players supporting WEBM: [http://www.videolan.org/vlc/ VLC 1.1+], [https://www.mozilla.com/en-US/firefox/all-beta.html Firefox 4 (beta)], [http://www.chromium.org/getting-involved/dev-channel Chrome (development versions)], [http://www.opera.com/ Opera], [http://www.webmproject.org/users/ more…]

Players supporting Ogg/Theora: [http://www.videolan.org/vlc/ VLC], [http://www.firefox.com/ Firefox], [http://www.opera.com/ Opera], [[TheoraSoftwarePlayers|more…]]
<br/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals." This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*[http://www.xiph.org/about/ About Xiph.Org]: Why you should care about open media
*[http://www.0xdeadbeef.com/weblog/2010/01/html5-video-and-h-264-what-history-tells-us-and-why-were-standing-with-the-web/ HTML5 Video and H.264: what history tells us and why we're standing with the web]: Chris Blizzard of Mozilla on free formats and the open web
*[http://diveintohtml5.org/video.html Dive into HTML5]: tutorial on HTML5 web video
*[http://webchat.freenode.net/?channels=vp8 Chat with the creators of the video] via freenode IRC in #vp8.
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[WikiPedia:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
*Wikipedia: [[WikiPedia:Passive_analogue_filter_development|The history of analog filters]] such as the [[WikiPedia:RC circuit|RC low-pass]] show connected to the spectrum analyizer in the video.
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]]. Filters that achieve such hard edges often do so at the expense of increased [[wikipedia:Ripple_(filters)#Frequency-domain_ripple|ripple]] and [http://www.ocf.berkeley.edu/~ashon/audio/phase/phaseaud2.htm phase distortion].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs loosely inspired by PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wikipedia:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>

===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===
[[Image:Dmpfg_015.jpg|360px|right]]

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===
[[Image:Dmpfg_016.jpg|360px|right]]

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** If we were all [[wikipedia:Dichromacy|dichromats]] then video would only need two color channels. Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
** [http://www.xritephoto.com/ph_toolframe.aspx?action=coloriq Test your color vision] (or at least your monitor).
* YCbCr is defined in terms of RGB by the ITU in two incompatible standards: [[wikipedia:Rec. 601|Rec. 601]] and [[wikipedia:Rec. 709|Rec. 709]]. Both conversion standards are lossy, which has prompted some to adopt a lossless alternative called [http://wiki.multimedia.cx/index.php?title=YCoCg YCoCg].
</div></center>

<br style="clear:both;"/>

==Containers==
[[Image:Dmpfg_017.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>

==Credits==
[[Image:Dmpfg_018.jpg|360px|right]]
[[Image:Dmpfg_019.png|360px|right]]

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the [http://microscopicseptet.com/ Microscopic Septet]<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
[http://www.cuneiformrecords.com www.cuneiformrecords.com]

This Video Was Produced Entirely With Free and Open Source Software:<br>

[http://www.gnu.org/ GNU]<br>
[http://www.linux.org/ Linux]<br>
[http://fedoraproject.org/ Fedora]<br>
[http://cinelerra.org/ Cinelerra]<br>
[http://www.gimp.org/ The Gimp]<br>
[http://audacity.sourceforge.net/ Audacity]<br>
[http://svn.xiph.org/trunk/postfish/README Postfish]<br>
[http://gstreamer.freedesktop.org/ Gstreamer]<br>

All trademarks are the property of their respective owners.

''Complete video'' [http://creativecommons.org/licenses/by-nc-sa/3.0/legalcode CC-BY-NC-SA]<br>
''Text transcript and Wiki edition'' [http://creativecommons.org/licenses/by-sa/3.0/legalcode CC-BY-SA]<br>

A Co-Production of Xiph.Org and Red Hat Inc.<br>
(C) 2010, Some Rights Reserved<br>

<br style="clear:both;"/><hr/>
<center><font size="+1">''[[A Digital Media Primer For Geeks (episode 1)/making|Learn more about the making of this video…]]''</font></center>

Videos/A Digital Media Primer For Geeks

2010-09-22T04:38:36Z

Mindspillage: /* Credits */ microscopic septet link, license link

[[Image:Dmpfg_001.jpg|360px|right]]
<small>''Wiki edition''</small>

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*[http://www.0xdeadbeef.com/weblog/2010/01/html5-video-and-h-264-what-history-tells-us-and-why-were-standing-with-the-web/ HTML5 Video and H.264: what history tells us and why we're standing with the web]: Chris Blizzard of Mozilla on free formats and the open web
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs loosely inspired by PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wikipedia:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>

===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===
[[Image:Dmpfg_015.jpg|360px|right]]

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===
[[Image:Dmpfg_016.jpg|360px|right]]

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** If we were all [[wikipedia:Dichromacy|dichromats]] then video would only need two color channels. Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
** [http://www.xritephoto.com/ph_toolframe.aspx?action=coloriq Test your color vision] (or at least your monitor).
* YCbCr is defined in terms of RGB by the ITU in two incompatible standards: [[wikipedia:Rec. 601|Rec. 601]] and [[wikipedia:Rec. 709|Rec. 709]]. Both conversion standards are lossy, which has prompted some to adopt a lossless alternative called [http://wiki.multimedia.cx/index.php?title=YCoCg YCoCg].
</div></center>

<br style="clear:both;"/>

==Containers==
[[Image:Dmpfg_017.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>

==Credits==
[[Image:Dmpfg_018.jpg|360px|right]]

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the [http://microscopicseptet.com/ Microscopic Septet]<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
[http://www.cuneiformrecords.com www.cuneiformrecords.com]

This Video Was Produced Entirely With Free and Open Source Software:<br>

[http://www.gnu.org/ GNU]<br>
[http://www.linux.org/ Linux]<br>
[http://fedoraproject.org/ Fedora]<br>
[http://cinelerra.org/ Cinelerra]<br>
[http://www.gimp.org/ The Gimp]<br>
[http://audacity.sourceforge.net/ Audacity]<br>
[http://svn.xiph.org/trunk/postfish/README Postfish]<br>
[http://gstreamer.freedesktop.org/ Gstreamer]<br>

[http://creativecommons.org/licenses/by-nc-sa/3.0/legalcode CC-BY-NC-SA]<br>
A Co-Production of Xiph.Org and Red Hat Inc.<br>
(C) 2010, Some Rights Reserved<br>

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-22T04:36:52Z

Mindspillage: /* Credits */ project links

[[Image:Dmpfg_001.jpg|360px|right]]
<small>''Wiki edition''</small>

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*[http://www.0xdeadbeef.com/weblog/2010/01/html5-video-and-h-264-what-history-tells-us-and-why-were-standing-with-the-web/ HTML5 Video and H.264: what history tells us and why we're standing with the web]: Chris Blizzard of Mozilla on free formats and the open web
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs loosely inspired by PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wikipedia:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>

===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===
[[Image:Dmpfg_015.jpg|360px|right]]

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===
[[Image:Dmpfg_016.jpg|360px|right]]

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** If we were all [[wikipedia:Dichromacy|dichromats]] then video would only need two color channels. Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
** [http://www.xritephoto.com/ph_toolframe.aspx?action=coloriq Test your color vision] (or at least your monitor).
* YCbCr is defined in terms of RGB by the ITU in two incompatible standards: [[wikipedia:Rec. 601|Rec. 601]] and [[wikipedia:Rec. 709|Rec. 709]]. Both conversion standards are lossy, which has prompted some to adopt a lossless alternative called [http://wiki.multimedia.cx/index.php?title=YCoCg YCoCg].
</div></center>

<br style="clear:both;"/>

==Containers==
[[Image:Dmpfg_017.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>

==Credits==
[[Image:Dmpfg_018.jpg|360px|right]]

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
[http://www.cuneiformrecords.com www.cuneiformrecords.com]

This Video Was Produced Entirely With Free and Open Source Software:<br>

[http://www.gnu.org/ GNU]<br>
[http://www.linux.org/ Linux]<br>
[http://fedoraproject.org/ Fedora]<br>
[http://cinelerra.org/ Cinelerra]<br>
[http://www.gimp.org/ The Gimp]<br>
[http://audacity.sourceforge.net/ Audacity]<br>
[http://svn.xiph.org/trunk/postfish/README Postfish]<br>
[http://gstreamer.freedesktop.org/ Gstreamer]<br>

CC BY-NC-SA<br>
A Co-Production of Xiph.Org and Red Hat Inc.<br>
(C) 2010, Some Rights Reserved<br>

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-22T04:35:24Z

Mindspillage: /* Credits */ project links

[[Image:Dmpfg_001.jpg|360px|right]]
<small>''Wiki edition''</small>

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*[http://www.0xdeadbeef.com/weblog/2010/01/html5-video-and-h-264-what-history-tells-us-and-why-were-standing-with-the-web/ HTML5 Video and H.264: what history tells us and why we're standing with the web]: Chris Blizzard of Mozilla on free formats and the open web
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs loosely inspired by PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wikipedia:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>

===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===
[[Image:Dmpfg_015.jpg|360px|right]]

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===
[[Image:Dmpfg_016.jpg|360px|right]]

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** If we were all [[wikipedia:Dichromacy|dichromats]] then video would only need two color channels. Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
** [http://www.xritephoto.com/ph_toolframe.aspx?action=coloriq Test your color vision] (or at least your monitor).
* YCbCr is defined in terms of RGB by the ITU in two incompatible standards: [[wikipedia:Rec. 601|Rec. 601]] and [[wikipedia:Rec. 709|Rec. 709]]. Both conversion standards are lossy, which has prompted some to adopt a lossless alternative called [http://wiki.multimedia.cx/index.php?title=YCoCg YCoCg].
</div></center>

<br style="clear:both;"/>

==Containers==
[[Image:Dmpfg_017.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>

==Credits==
[[Image:Dmpfg_018.jpg|360px|right]]

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
[http://www.cuneiformrecords.com www.cuneiformrecords.com]

This Video Was Produced Entirely With Free and Open Source Software:<br>

[http://www.gnu.org/ GNU]<br>
[http://www.linux.org/ Linux]<br>
[http://fedoraproject.org/ Fedora]<br>
[http://cinelerra.org/ Cinelerra]<br>
[http://www.gimp.org/ The Gimp]<br>
[http://audacity.sourceforge.net/ Audacity]<br>
Postfish<br>
[http://gstreamer.freedesktop.org/ Gstreamer]<br>

CC BY-NC-SA<br>
A Co-Production of Xiph.Org and Red Hat Inc.<br>
(C) 2010, Some Rights Reserved<br>

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-22T04:29:07Z

Mindspillage: /* Credits */ line breaks

[[Image:Dmpfg_001.jpg|360px|right]]
<small>''Wiki edition''</small>

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*[http://www.0xdeadbeef.com/weblog/2010/01/html5-video-and-h-264-what-history-tells-us-and-why-were-standing-with-the-web/ HTML5 Video and H.264: what history tells us and why we're standing with the web]: Chris Blizzard of Mozilla on free formats and the open web
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs loosely inspired by PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wikipedia:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>

===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===
[[Image:Dmpfg_015.jpg|360px|right]]

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===
[[Image:Dmpfg_016.jpg|360px|right]]

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** If we were all [[wikipedia:Dichromacy|dichromats]] then video would only need two color channels. Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
** [http://www.xritephoto.com/ph_toolframe.aspx?action=coloriq Test your color vision] (or at least your monitor).
* YCbCr is defined in terms of RGB by the ITU in two incompatible standards: [[wikipedia:Rec. 601|Rec. 601]] and [[wikipedia:Rec. 709|Rec. 709]]. Both conversion standards are lossy, which has prompted some to adopt a lossless alternative called [http://wiki.multimedia.cx/index.php?title=YCoCg YCoCg].
</div></center>

<br style="clear:both;"/>

==Containers==
[[Image:Dmpfg_017.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
[http://www.cuneiformrecords.com www.cuneiformrecords.com]

This Video Was Produced Entirely With Free and Open Source Software:<br>

GNU<br>
Linux<br>
Fedora<br>
Cinelerra<br>
The Gimp<br>
Audacity<br>
Postfish<br>
Gstreamer<br>

CC BY-NC-SA<br>
A Co-Production of Xiph.Org and Red Hat Inc.<br>
(C) 2010, Some Rights Reserved<br>

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-22T04:28:18Z

Mindspillage: /* Credits */ link cuneiform

[[Image:Dmpfg_001.jpg|360px|right]]
<small>''Wiki edition''</small>

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*[http://www.0xdeadbeef.com/weblog/2010/01/html5-video-and-h-264-what-history-tells-us-and-why-were-standing-with-the-web/ HTML5 Video and H.264: what history tells us and why we're standing with the web]: Chris Blizzard of Mozilla on free formats and the open web
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs loosely inspired by PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wikipedia:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>

===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===
[[Image:Dmpfg_015.jpg|360px|right]]

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===
[[Image:Dmpfg_016.jpg|360px|right]]

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** If we were all [[wikipedia:Dichromacy|dichromats]] then video would only need two color channels. Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
** [http://www.xritephoto.com/ph_toolframe.aspx?action=coloriq Test your color vision] (or at least your monitor).
* YCbCr is defined in terms of RGB by the ITU in two incompatible standards: [[wikipedia:Rec. 601|Rec. 601]] and [[wikipedia:Rec. 709|Rec. 709]]. Both conversion standards are lossy, which has prompted some to adopt a lossless alternative called [http://wiki.multimedia.cx/index.php?title=YCoCg YCoCg].
</div></center>

<br style="clear:both;"/>

==Containers==
[[Image:Dmpfg_017.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
[http://www.cuneiformrecords.com www.cuneiformrecords.com]

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-22T04:24:12Z

Mindspillage: /* Introduction */ typo

[[Image:Dmpfg_001.jpg|360px|right]]
<small>''Wiki edition''</small>

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*[http://www.0xdeadbeef.com/weblog/2010/01/html5-video-and-h-264-what-history-tells-us-and-why-were-standing-with-the-web/ HTML5 Video and H.264: what history tells us and why we're standing with the web]: Chris Blizzard of Mozilla on free formats and the open web
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs loosely inspired by PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wikipedia:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>

===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===
[[Image:Dmpfg_015.jpg|360px|right]]

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===
[[Image:Dmpfg_016.jpg|360px|right]]

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** If we were all [[wikipedia:Dichromacy|dichromats]] then video would only need two color channels. Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
** [http://www.xritephoto.com/ph_toolframe.aspx?action=coloriq Test your color vision] (or at least your monitor).
* YCbCr is defined in terms of RGB by the ITU in two incompatible standards: [[wikipedia:Rec. 601|Rec. 601]] and [[wikipedia:Rec. 709|Rec. 709]]. Both conversion standards are lossy, which has prompted some to adopt a lossless alternative called [http://wiki.multimedia.cx/index.php?title=YCoCg YCoCg].
</div></center>

<br style="clear:both;"/>

==Containers==
[[Image:Dmpfg_017.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-22T04:23:47Z

Mindspillage: /* Introduction */ link

[[Image:Dmpfg_001.jpg|360px|right]]
<small>''Wiki edition''</small>

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*[http://www.0xdeadbeef.com/weblog/2010/01/html5-video-and-h-264-what-history-tells-us-and-why-were-standing-with-the-web/ HTML5 Video and H.264: what ihstory tells us and why we're standing with the web]: Chris Blizzard of Mozilla on free formats and the open web
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs loosely inspired by PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wikipedia:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>

===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===
[[Image:Dmpfg_015.jpg|360px|right]]

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===
[[Image:Dmpfg_016.jpg|360px|right]]

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** If we were all [[wikipedia:Dichromacy|dichromats]] then video would only need two color channels. Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
** [http://www.xritephoto.com/ph_toolframe.aspx?action=coloriq Test your color vision] (or at least your monitor).
* YCbCr is defined in terms of RGB by the ITU in two incompatible standards: [[wikipedia:Rec. 601|Rec. 601]] and [[wikipedia:Rec. 709|Rec. 709]]. Both conversion standards are lossy, which has prompted some to adopt a lossless alternative called [http://wiki.multimedia.cx/index.php?title=YCoCg YCoCg].
</div></center>

<br style="clear:both;"/>

==Containers==
[[Image:Dmpfg_017.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-22T03:16:28Z

Mindspillage: /* gamma */ fix link

[[Image:Dmpfg_001.jpg|360px|right]]
<small>''Wiki edition''</small>

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs derived from PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wikipedia:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>

===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
</div></center>

<br style="clear:both;"/>

==Containers==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>
==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-22T03:12:58Z

Mindspillage: /* frame rate and interlacing */ link

[[Image:Dmpfg_001.jpg|360px|right]]
<small>''Wiki edition''</small>

This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
[[Image:Dmpfg_000.jpg|360px|right]]
[[Image:Dmpfg_002.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|360px|right]]
[[Image:Dmpfg_006.jpg|360px|right]]
[[Image:Dmpfg_007.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>

[[WikiPedia:Sound|Sound]] is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the [[WikiPedia:Oscilloscope|'scope]] down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's [[WikiPedia:Continuous_function|continuous]] in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is [[WikiPedia:Discrete_math|discrete]] in both value and time.
In the simplest and most common system, called [[WikiPedia:Pulse code modulation|Pulse Code Modulation]],
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the [[WikiPedia:Nyquist-Shannon sampling theorem|Sampling Theorem]] says we can and tells us
how. Published in its most recognizable form by [[WikiPedia:Claude Shannon|Claude Shannon]] in 1949
and built on the work of [[WikiPedia:Harry Nyquist|Nyquist]], and [[WikiPedia:Ralph Hartley|Hartley]], and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The [[WikiPedia:Telegraph|telegraph]] predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... [[WikiPedia:Tickertape|tickertape]]. Harry Nyquist of [[WikiPedia:Bell_labs|Bell Labs]] was
researching telegraph pulse transmission when he published his
description of what later became known as the [[WikiPedia:Nyquist_frequency|Nyquist frequency]], the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A [[WikiPedia:Low-pass_filter#Continuous-time_low-pass_filters|second-order low-pass filter]], for example,
requires two passive components. An all-analog [[WikiPedia:Short-time_Fourier_transform|short-time Fourier
transform]], a few hundred. Well, maybe a thousand if you want to build
something really fancy (bang on the [http://www.testequipmentdepot.com/usedequipment/hewlettpackard/spectrumanalyzers/3585a.htm 3585]). Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And [[WikiPedia:File:Transistor_Count_and_Moore's_Law_-_2008.svg|since we
do]], digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds [[WikiPedia:Johnson–Nyquist_noise|noise]] and [[WikiPedia:Distortion#Electronic_signals|distortion]], usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an [[WikiPedia:Inductor|inductor]] the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use [[WikiPedia:Lossy_compression|lossy]] algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern [[WikiPedia:Digital-to-analog_converter|conversion stages]] are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
[[WikiPedia:Delta-sigma_modulation|Sigma-Delta coding]] used by the [[WikiPedia:Super_Audio_CD|SACD]], which is a form of [[wikipedia:Pulse-density_modulation|Pulse Density
Modulation]]. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.
<br style="clear:both;"/>
===sample rate===
[[Image:Dmpfg_009.jpg|360px|right]]
[[Image:Dmpfg_008.jpg|360px|right]]
The first parameter is the [[wikipedia:Sampling_rate|sampling rate]]. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally [[wikipedia:Bandlimiting|band-limited]] voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathematician [[wikipedia:Joseph_Fourier|Jean
Baptiste Joseph Fourier]] showed that we can also think of signals like
audio as a set of component frequencies. This [[wikipedia:Frequency_domain|frequency-domain]]
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it [[wikipedia:Basis_(linear_algebra)|a different way]]. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as [[wikipedia:Aliasing|aliasing distortion]].

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
[[wikipedia:Hearing_range|20kHz]] but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
[[wikipedia:Octave_(electronics)|octave]] or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.
<br style="clear:both;"/>
===sample format===
[[Image:Dmpfg_anim.gif|right]]

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was [[wikipedia:Quantization_(sound_processing)#Audio_quantization|eight-bit]] [[wikipedia:Linear_pulse_code_modulation|linear]], encoded as an [[wikipedia:Signedness|unsigned]] [[wikipedia:Integer_(computer_science)#Bytes_and_octets|byte]]. The
[[wikipedia:Dynamic_range#Audio|dynamic range]] is limited to about [[wikipedia:Decibel|50dB]] and the [[wikipedia:Quantization_error|quantization noise]], as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called [[wikipedia:A-law_algorithm|A-law]] and [[wikipedia:Μ-law_algorithm|μ-law]]. These formats encode a roughly
[[wikipedia:Audio_bit_depth#Dynamic_range|14 bit dynamic range]] into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit [[wikipedia:Two's_complement|two's-complement]] signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are [[wikipedia:Clipping_(audio)|clipped]].

In mixing and mastering, it's not unusual to use [[wikipedia:Floating_point|floating-point]]
numbers for PCM instead of [[wikipedia:Integer_(computer_science)|integers]]. A 32 bit [[wikipedia:IEEE_754-2008|IEEE754]] float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in [[wikipedia:Endianness|big- or little-endian order]], and both
endiannesses are common. For example, Microsoft [[wikipedia:WAV|WAV]] files are little-endian,
and Apple [[wikipedia:AIFC|AIFC]] files tend to be big-endian. Be aware of it.
<br style="clear:both;"/>
===channels===

The third PCM parameter is the number of [[wikipedia:Multichannel_audio|channels]]. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.
<br style="clear:both;"/>
===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [[wikipedia:Roll-off|Wikipedia's article on filter roll-off]], to learn why it's hard to build analog filters with a very narrow [[wikipedia:Transition_band|transition band]] between the [[wikipedia:Passband|passband]] and the [[wikipedia:Stopband|stopband]].
* [http://wiki.multimedia.cx/index.php?title=PCM Some more minutiae] about PCM in practice.
* [[wikipedia:DPCM|DPCM]] and [[wikipedia:ADPCM|ADPCM]], simple audio codecs derived from PCM.
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
[[Image:Dmpfg_010.jpg|360px|right]]
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. [[wikipedia:Red_Book_(audio_Compact_Disc_standard)#Technical_details|Raw CD audio]] is about 1.4 megabits
per second. Raw [[wikipedia:1080i|1080i]] HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By [[wikipedia:Moore's_law|Moore's law]]... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal [[wikipedia:NTSC|analog television broadcast]]. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.
<br style="clear:both;"/>
===resolution and aspect===
[[Image:Dmpfg_011.jpg|360px|right]]

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of [[wikipedia:Scan_line|scanlines]] in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
[[wikipedia:Bandwidth_(signal_processing)|bandwidth]]. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of [[wikipedia:DVD-Video#Frame_size_and_frame_rate|704 by 480]], a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of [[wikipedia:Standard-definition_television#Resolution|10:11]], making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.
<br style="clear:both;"/>
===frame rate and interlacing===
[[Image:Dmpfg_012.jpg|360px|right]]

The second obvious video parameter is the [[wikipedia:Frame_rate|frame rate]], the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
[[wikipedia:Interlace|interlacing]].

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize [[wikipedia:Flicker_(screen)|flicker]]
on phosphor-based [[wikipedia:Cathode_ray_tube|CRTs]]. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
[[wikipedia:Deinterlacing|deinterlace]] a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.
<br style="clear:both;"/>

===gamma===
[[Image:Dmpfg_013.jpg|360px|right]]

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary [[wiki:Gamma_correction|gamma correction]] circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard [[wikipedia:sRGB|sRGB]] computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.
<br style="clear:both;"/>
===color and colorspace===
[[Image:Dmpfg_014.jpg|360px|right]]

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as [[wikipedia:Additive_color|additive primaries]] to
produce a full range of color output. The primary pigments in
printing are [[wikipedia:CMYK|Cyan, Magenta, and Yellow]] for the same reason; pigments
are [[wikipedia:Subtractive_color|subtractive]], and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to [[wikipedia:Luminance_(relative)|luminosity]] than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution [[wikipedia:Luma_(video)|luma channel]]—the black & white—along with
additional, often lower resolution [[wikipedia:Chrominance|chroma channels]], the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called [[wikipedia:Y'CbCr|Y'CbCr]], but the more generic term YUV is
widely used to describe all the analog and digital variants of this
color model.
<br style="clear:both;"/>

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually [[wikipedia:Chroma_subsampling|halved or even quartered]] in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are [[wikipedia:Chroma_subsampling#4:4:4_Y.27CbCr|4:4:4]] video, which isn't actually subsampled at all, [[wikipedia:Chroma_subsampling#4:2:2|4:2:2]] video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, [[wikipedia:Chroma_subsampling#4:2:0|4:2:0]] video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, [[wikipedia:Chroma_subsampling#4:1:1|4:1:1]], and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, [[wikipedia:Motion_Jpeg|motion
JPEG]], [[wikipedia:MPEG-1#Part_2:_Video|MPEG-1 video]], [[wikipedia:MPEG-2#Video_coding_.28simplified.29|MPEG-2 video]], [[wikipedia:DV#DV_Compression|DV]], [[wikipedia:Theora|Theora]], and [[wikipedia:WebM|WebM]] all use or can
use 4:2:0 subsampling, but they site the chroma pixels [http://www.mir.com/DMG/chroma.html three different ways].

Motion JPEG, MPEG-1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG-2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.
<br style="clear:both;"/>

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least [http://www.fourcc.org/yuv.php 50 different formats] in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or [[wikipedia:FourCC|fourcc]] code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. [http://www.fourcc.org/yuv.php#UYVY YV12]
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of [[wikipedia:YUV#BT.709_and_BT.601|several YUV colorspace definitions]].
<br style="clear:both;"/>

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* [http://wiki.multimedia.cx/index.php?title=YUV4MPEG2 The y4m format] is the most common simple container for raw YUV video. People occasionally use [[OggYUV]] to store it in Ogg instead.
* Learn about [[wikipedia:High_dynamic_range_imaging|high dynamic range imaging]], which achieves better representation of the full range of brightnesses in the real world by using more than 8 bits per channel.
* Learn about how [[wikipedia:Trichromatic_vision|trichromatic color vision]] works in humans, and how human color perception is encoded in the [[wikipedia:CIE 1931 color space|CIE 1931 XYZ color space]].
** Compare with the [[wikipedia:Lab_color_space|Lab color space]], mathematically equivalent but structured to account for "perceptual uniformity".
** Some humans might be [[wikipedia:Tetrachromacy#Possibility_of_human_tetrachromats|tetrachromats]], in which case they would need an additional color channel for video to fully represent their vision.
</div></center>

<br style="clear:both;"/>

==Containers==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other [[wikipedia:Metadata#Video|metadata]] we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a [[wikipedia:Container_format_(digital)|container]]. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.
<br style="clear:both;"/>
==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-21T22:25:25Z

Mindspillage: /* Postprocessing */ sp

<small>''Wiki edition''</small>
[[Image:Dmpfg_001.jpg|thumb|360px|right]]
This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

[[Image:Dmpfg_002.jpg|thumb|360px|right]]
It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>
[[Image:Dmpfg_004.jpg|thumb|360px|right]]

Sound is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the 'scope down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's continuous in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is discrete in both value and time.
In the simplest and most common system, called Pulse Code Modulation,
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

[[Image:Dmpfg_006.jpg|thumb|360px|right]]
Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the Sampling Theorem says we can and tells us
how. Published in its most recognizable form by Claude Shannon in 1949
and built on the work of Nyquist, and Hartley, and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The telegraph predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... tickertape. Harry Nyquist of Bell Labs was
researching telegraph pulse transmission when he published his
description of what later became known as the Nyquist frequency, the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

[[Image:Dmpfg_007.jpg|thumb|360px|right]]
Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A second-order low-pass filter, for example,
requires two passive components. An all-analog short-time Fourier
transform, a few hundred. Well, maybe a thousand if you want to build
something really fancy [bang on the 3585]. Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And since we
do, digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds noise and distortion, usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an inductor the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use lossy algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern conversion stages are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
Sigma-Delta coding used by the SACD, which is a form of Pulse Density
Modulation. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.

===sample rate===

The first parameter is the sampling rate. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally band-limited voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

[[Image:Dmpfg_008.jpg|thumb|360px|right]]
Stepping back for just a second, the French mathematician Jean
Baptiste Joseph Fourier showed that we can also think of signals like
audio as a set of component frequencies. This frequency-domain
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it a different way. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as aliasing distortion.

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
20kHz but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
octave or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.

===sample format===

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was eight-bit linear, encoded as an unsigned byte. The
dynamic range is limited to about 50dB and the quantization noise, as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called A-law and mu-law. These formats encode a roughly
14 bit dynamic range into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit two's-complement signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are clipped.

In mixing and mastering, it's not unusual to use floating-point
numbers for PCM instead of integers. A 32 bit IEEE754 float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in big- or little-endian order, and both
endiannesses are common. For example, Microsoft WAV files are little-endian,
and Apple AIFC files tend to be big-endian. Be aware of it.

===channels===

The third PCM parameter is the number of channels. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.

===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. Raw CD audio is about 1.4 megabits
per second. Raw 1080i HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By Moore's law... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal analog television broadcast. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.

===resolution and aspect===

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of scanlines in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
bandwidth. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of 704 by 480, a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of 10:11, making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.

===frame rate and interlacing===

The second obvious video parameter is the frame rate, the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
interlacing.

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize flicker
on phosphor-based CRTs. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
deinterlace a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.

===gamma===

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary gamma correction circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard sRGB computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.

===color and colorspace===

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as additive primaries to
produce a full range of color output. The primary pigments in
printing are Cyan, Magenta, and Yellow for the same reason; pigments
are subtractive, and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to luminosity than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution luma channel—the black & white—along with
additional, often lower resolution chroma channels, the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called Y'CbCr, but the more generic term YUV is
widely used to decribe all the analog and digital variants of this
color model.

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually halved or even quartered in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are 4:4:4 video, which isn't actually subsampled at all, 4:2:2 video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, 4:2:0 video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, 4:1:1, and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, motion
JPEG, MPEG-1 video, MPEG-2 video, DV, Theora, and WebM all use or can
use 4:2:0 subsampling, but they site the chroma pixels three different
ways.

Motion JPEG, MPEG1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least 50 different formats in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or fourcc code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. YV12
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of several YUV colorspace definitions.

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Containers==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other metadata we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a container. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast, and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orange audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-21T22:24:00Z

Mindspillage: /* Video shooting sequence */ sp

<small>''Wiki edition''</small>
[[Image:Dmpfg_001.jpg|thumb|360px|right]]
This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

[[Image:Dmpfg_002.jpg|thumb|360px|right]]
It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>
[[Image:Dmpfg_004.jpg|thumb|360px|right]]

Sound is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the 'scope down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's continuous in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is discrete in both value and time.
In the simplest and most common system, called Pulse Code Modulation,
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

[[Image:Dmpfg_006.jpg|thumb|360px|right]]
Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the Sampling Theorem says we can and tells us
how. Published in its most recognizable form by Claude Shannon in 1949
and built on the work of Nyquist, and Hartley, and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The telegraph predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... tickertape. Harry Nyquist of Bell Labs was
researching telegraph pulse transmission when he published his
description of what later became known as the Nyquist frequency, the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

[[Image:Dmpfg_007.jpg|thumb|360px|right]]
Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A second-order low-pass filter, for example,
requires two passive components. An all-analog short-time Fourier
transform, a few hundred. Well, maybe a thousand if you want to build
something really fancy [bang on the 3585]. Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And since we
do, digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds noise and distortion, usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an inductor the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use lossy algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern conversion stages are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
Sigma-Delta coding used by the SACD, which is a form of Pulse Density
Modulation. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.

===sample rate===

The first parameter is the sampling rate. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally band-limited voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

[[Image:Dmpfg_008.jpg|thumb|360px|right]]
Stepping back for just a second, the French mathematician Jean
Baptiste Joseph Fourier showed that we can also think of signals like
audio as a set of component frequencies. This frequency-domain
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it a different way. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as aliasing distortion.

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
20kHz but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
octave or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.

===sample format===

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was eight-bit linear, encoded as an unsigned byte. The
dynamic range is limited to about 50dB and the quantization noise, as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called A-law and mu-law. These formats encode a roughly
14 bit dynamic range into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit two's-complement signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are clipped.

In mixing and mastering, it's not unusual to use floating-point
numbers for PCM instead of integers. A 32 bit IEEE754 float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in big- or little-endian order, and both
endiannesses are common. For example, Microsoft WAV files are little-endian,
and Apple AIFC files tend to be big-endian. Be aware of it.

===channels===

The third PCM parameter is the number of channels. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.

===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. Raw CD audio is about 1.4 megabits
per second. Raw 1080i HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By Moore's law... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal analog television broadcast. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.

===resolution and aspect===

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of scanlines in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
bandwidth. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of 704 by 480, a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of 10:11, making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.

===frame rate and interlacing===

The second obvious video parameter is the frame rate, the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
interlacing.

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize flicker
on phosphor-based CRTs. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
deinterlace a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.

===gamma===

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary gamma correction circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard sRGB computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.

===color and colorspace===

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as additive primaries to
produce a full range of color output. The primary pigments in
printing are Cyan, Magenta, and Yellow for the same reason; pigments
are subtractive, and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to luminosity than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution luma channel—the black & white—along with
additional, often lower resolution chroma channels, the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called Y'CbCr, but the more generic term YUV is
widely used to decribe all the analog and digital variants of this
color model.

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually halved or even quartered in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are 4:4:4 video, which isn't actually subsampled at all, 4:2:2 video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, 4:2:0 video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, 4:1:1, and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, motion
JPEG, MPEG-1 video, MPEG-2 video, DV, Theora, and WebM all use or can
use 4:2:0 subsampling, but they site the chroma pixels three different
ways.

Motion JPEG, MPEG1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least 50 different formats in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or fourcc code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. YV12
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of several YUV colorspace definitions.

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Containers==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other metadata we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a container. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking a lot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orage audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-21T22:22:41Z

Mindspillage: /* Credits */ sp, punct

<small>''Wiki edition''</small>
[[Image:Dmpfg_001.jpg|thumb|360px|right]]
This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

[[Image:Dmpfg_002.jpg|thumb|360px|right]]
It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>
[[Image:Dmpfg_004.jpg|thumb|360px|right]]

Sound is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the 'scope down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's continuous in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is discrete in both value and time.
In the simplest and most common system, called Pulse Code Modulation,
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

[[Image:Dmpfg_006.jpg|thumb|360px|right]]
Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the Sampling Theorem says we can and tells us
how. Published in its most recognizable form by Claude Shannon in 1949
and built on the work of Nyquist, and Hartley, and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The telegraph predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... tickertape. Harry Nyquist of Bell Labs was
researching telegraph pulse transmission when he published his
description of what later became known as the Nyquist frequency, the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

[[Image:Dmpfg_007.jpg|thumb|360px|right]]
Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A second-order low-pass filter, for example,
requires two passive components. An all-analog short-time Fourier
transform, a few hundred. Well, maybe a thousand if you want to build
something really fancy [bang on the 3585]. Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And since we
do, digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds noise and distortion, usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an inductor the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use lossy algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern conversion stages are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
Sigma-Delta coding used by the SACD, which is a form of Pulse Density
Modulation. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.

===sample rate===

The first parameter is the sampling rate. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally band-limited voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

[[Image:Dmpfg_008.jpg|thumb|360px|right]]
Stepping back for just a second, the French mathematician Jean
Baptiste Joseph Fourier showed that we can also think of signals like
audio as a set of component frequencies. This frequency-domain
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it a different way. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as aliasing distortion.

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
20kHz but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
octave or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.

===sample format===

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was eight-bit linear, encoded as an unsigned byte. The
dynamic range is limited to about 50dB and the quantization noise, as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called A-law and mu-law. These formats encode a roughly
14 bit dynamic range into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit two's-complement signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are clipped.

In mixing and mastering, it's not unusual to use floating-point
numbers for PCM instead of integers. A 32 bit IEEE754 float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in big- or little-endian order, and both
endiannesses are common. For example, Microsoft WAV files are little-endian,
and Apple AIFC files tend to be big-endian. Be aware of it.

===channels===

The third PCM parameter is the number of channels. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.

===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. Raw CD audio is about 1.4 megabits
per second. Raw 1080i HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By Moore's law... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal analog television broadcast. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.

===resolution and aspect===

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of scanlines in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
bandwidth. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of 704 by 480, a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of 10:11, making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.

===frame rate and interlacing===

The second obvious video parameter is the frame rate, the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
interlacing.

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize flicker
on phosphor-based CRTs. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
deinterlace a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.

===gamma===

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary gamma correction circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard sRGB computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.

===color and colorspace===

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as additive primaries to
produce a full range of color output. The primary pigments in
printing are Cyan, Magenta, and Yellow for the same reason; pigments
are subtractive, and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to luminosity than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution luma channel—the black & white—along with
additional, often lower resolution chroma channels, the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called Y'CbCr, but the more generic term YUV is
widely used to decribe all the analog and digital variants of this
color model.

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually halved or even quartered in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are 4:4:4 video, which isn't actually subsampled at all, 4:2:2 video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, 4:2:0 video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, 4:1:1, and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, motion
JPEG, MPEG-1 video, MPEG-2 video, DV, Theora, and WebM all use or can
use 4:2:0 subsampling, but they site the chroma pixels three different
ways.

Motion JPEG, MPEG1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least 50 different formats in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or fourcc code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. YV12
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of several YUV colorspace definitions.

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Containers==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other metadata we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a container. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well-earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then—Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:<br>
"Boo Boo Coming", by Joel Forrester<br>
Performed by the Microscopic Septet<br>
Used by permission of Cuneiform Records.<br>
Original source track All Rights Reserved.<br>
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>

==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking alot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orage audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-21T22:20:42Z

Mindspillage: /* Containers */ sp, punct

<small>''Wiki edition''</small>
[[Image:Dmpfg_001.jpg|thumb|360px|right]]
This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

[[Image:Dmpfg_002.jpg|thumb|360px|right]]
It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>
[[Image:Dmpfg_004.jpg|thumb|360px|right]]

Sound is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the 'scope down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's continuous in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is discrete in both value and time.
In the simplest and most common system, called Pulse Code Modulation,
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

[[Image:Dmpfg_006.jpg|thumb|360px|right]]
Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the Sampling Theorem says we can and tells us
how. Published in its most recognizable form by Claude Shannon in 1949
and built on the work of Nyquist, and Hartley, and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The telegraph predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... tickertape. Harry Nyquist of Bell Labs was
researching telegraph pulse transmission when he published his
description of what later became known as the Nyquist frequency, the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

[[Image:Dmpfg_007.jpg|thumb|360px|right]]
Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A second-order low-pass filter, for example,
requires two passive components. An all-analog short-time Fourier
transform, a few hundred. Well, maybe a thousand if you want to build
something really fancy [bang on the 3585]. Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And since we
do, digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds noise and distortion, usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an inductor the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use lossy algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern conversion stages are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
Sigma-Delta coding used by the SACD, which is a form of Pulse Density
Modulation. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.

===sample rate===

The first parameter is the sampling rate. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally band-limited voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

[[Image:Dmpfg_008.jpg|thumb|360px|right]]
Stepping back for just a second, the French mathematician Jean
Baptiste Joseph Fourier showed that we can also think of signals like
audio as a set of component frequencies. This frequency-domain
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it a different way. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as aliasing distortion.

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
20kHz but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
octave or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.

===sample format===

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was eight-bit linear, encoded as an unsigned byte. The
dynamic range is limited to about 50dB and the quantization noise, as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called A-law and mu-law. These formats encode a roughly
14 bit dynamic range into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit two's-complement signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are clipped.

In mixing and mastering, it's not unusual to use floating-point
numbers for PCM instead of integers. A 32 bit IEEE754 float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in big- or little-endian order, and both
endiannesses are common. For example, Microsoft WAV files are little-endian,
and Apple AIFC files tend to be big-endian. Be aware of it.

===channels===

The third PCM parameter is the number of channels. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.

===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. Raw CD audio is about 1.4 megabits
per second. Raw 1080i HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By Moore's law... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal analog television broadcast. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.

===resolution and aspect===

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of scanlines in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
bandwidth. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of 704 by 480, a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of 10:11, making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.

===frame rate and interlacing===

The second obvious video parameter is the frame rate, the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
interlacing.

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize flicker
on phosphor-based CRTs. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
deinterlace a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.

===gamma===

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary gamma correction circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard sRGB computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.

===color and colorspace===

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as additive primaries to
produce a full range of color output. The primary pigments in
printing are Cyan, Magenta, and Yellow for the same reason; pigments
are subtractive, and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to luminosity than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution luma channel—the black & white—along with
additional, often lower resolution chroma channels, the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called Y'CbCr, but the more generic term YUV is
widely used to decribe all the analog and digital variants of this
color model.

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually halved or even quartered in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are 4:4:4 video, which isn't actually subsampled at all, 4:2:2 video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, 4:2:0 video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, 4:1:1, and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, motion
JPEG, MPEG-1 video, MPEG-2 video, DV, Theora, and WebM all use or can
use 4:2:0 subsampling, but they site the chroma pixels three different
ways.

Motion JPEG, MPEG1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least 50 different formats in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or fourcc code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. YV12
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of several YUV colorspace definitions.

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Containers==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight-up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally-visible
structure, but they're often uniformly sized. We could just string
them together in a rigid predetermined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames, though, aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other metadata we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata—that is, data about the data—within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a container. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate, and present the media. In general, any container can hold
any kind of data. And data can be put into any container.

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then--- Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:
"Boo Boo Coming", by Joel Forrester
Performed by the Microscopic Septet
Used by permission of Cuneiform Records.
Original source track All Rights Reserved.
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>
==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking alot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orage audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-21T22:19:20Z

Mindspillage: /* Video vegetables (they're good for you!) */ sp, punct

<small>''Wiki edition''</small>
[[Image:Dmpfg_001.jpg|thumb|360px|right]]
This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

[[Image:Dmpfg_002.jpg|thumb|360px|right]]
It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>
[[Image:Dmpfg_004.jpg|thumb|360px|right]]

Sound is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the 'scope down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's continuous in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is discrete in both value and time.
In the simplest and most common system, called Pulse Code Modulation,
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

[[Image:Dmpfg_006.jpg|thumb|360px|right]]
Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the Sampling Theorem says we can and tells us
how. Published in its most recognizable form by Claude Shannon in 1949
and built on the work of Nyquist, and Hartley, and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The telegraph predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... tickertape. Harry Nyquist of Bell Labs was
researching telegraph pulse transmission when he published his
description of what later became known as the Nyquist frequency, the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

[[Image:Dmpfg_007.jpg|thumb|360px|right]]
Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A second-order low-pass filter, for example,
requires two passive components. An all-analog short-time Fourier
transform, a few hundred. Well, maybe a thousand if you want to build
something really fancy [bang on the 3585]. Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And since we
do, digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds noise and distortion, usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an inductor the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use lossy algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern conversion stages are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
Sigma-Delta coding used by the SACD, which is a form of Pulse Density
Modulation. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.

===sample rate===

The first parameter is the sampling rate. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally band-limited voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

[[Image:Dmpfg_008.jpg|thumb|360px|right]]
Stepping back for just a second, the French mathematician Jean
Baptiste Joseph Fourier showed that we can also think of signals like
audio as a set of component frequencies. This frequency-domain
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it a different way. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as aliasing distortion.

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
20kHz but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
octave or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.

===sample format===

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was eight-bit linear, encoded as an unsigned byte. The
dynamic range is limited to about 50dB and the quantization noise, as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called A-law and mu-law. These formats encode a roughly
14 bit dynamic range into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit two's-complement signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are clipped.

In mixing and mastering, it's not unusual to use floating-point
numbers for PCM instead of integers. A 32 bit IEEE754 float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in big- or little-endian order, and both
endiannesses are common. For example, Microsoft WAV files are little-endian,
and Apple AIFC files tend to be big-endian. Be aware of it.

===channels===

The third PCM parameter is the number of channels. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.

===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. Raw CD audio is about 1.4 megabits
per second. Raw 1080i HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process, and store per
second. By Moore's law... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty-year-old black and white television could still show a
normal analog television broadcast. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.

===resolution and aspect===

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of scanlines in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
bandwidth. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of 704 by 480, a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of 10:11, making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.

===frame rate and interlacing===

The second obvious video parameter is the frame rate, the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
interlacing.

In the very earliest days of broadcast video, engineers sought the
fastest practical frame rate to smooth motion and to minimize flicker
on phosphor-based CRTs. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
deinterlace a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.

===gamma===

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary gamma correction circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard sRGB computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.

===color and colorspace===

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as additive primaries to
produce a full range of color output. The primary pigments in
printing are Cyan, Magenta, and Yellow for the same reason; pigments
are subtractive, and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be, and sometimes is, represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to luminosity than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution luma channel—the black & white—along with
additional, often lower resolution chroma channels, the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset, and quantized for digital video, it's
usually more correctly called Y'CbCr, but the more generic term YUV is
widely used to decribe all the analog and digital variants of this
color model.

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually halved or even quartered in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are 4:4:4 video, which isn't actually subsampled at all, 4:2:2 video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, 4:2:0 video in which both the horizontal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, 4:1:1, and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, motion
JPEG, MPEG-1 video, MPEG-2 video, DV, Theora, and WebM all use or can
use 4:2:0 subsampling, but they site the chroma pixels three different
ways.

Motion JPEG, MPEG1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizarre.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizontal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least 50 different formats in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings, due either to
convenience once-upon-a-time on some particular piece of hardware, or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or fourcc code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. YV12
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of several YUV colorspace definitions.

===done!===

That wraps up our not-so-quick and yet very incomplete tour of raw
video. The good news is we can already get quite a lot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Containers==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally visible
structure, but they're often uniformly sized. We could just string
them together in a rigid pre-determined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames though aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other metadata we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata, that is, data about the data, within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a container. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate and present the media. In general, any container can hold
any kind of data. And data can be put into any container.

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then--- Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:
"Boo Boo Coming", by Joel Forrester
Performed by the Microscopic Septet
Used by permission of Cuneiform Records.
Original source track All Rights Reserved.
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>
==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking alot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orage audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-21T21:53:49Z

Mindspillage: /* Raw (digital audio) meat */ sp, punct

<small>''Wiki edition''</small>
[[Image:Dmpfg_001.jpg|thumb|360px|right]]
This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

[[Image:Dmpfg_002.jpg|thumb|360px|right]]
It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Analog_vs_Digital|Discuss this section]]</small>
[[Image:Dmpfg_004.jpg|thumb|360px|right]]

Sound is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the 'scope down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's continuous in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is discrete in both value and time.
In the simplest and most common system, called Pulse Code Modulation,
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

[[Image:Dmpfg_006.jpg|thumb|360px|right]]
Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the Sampling Theorem says we can and tells us
how. Published in its most recognizable form by Claude Shannon in 1949
and built on the work of Nyquist, and Hartley, and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The telegraph predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... tickertape. Harry Nyquist of Bell Labs was
researching telegraph pulse transmission when he published his
description of what later became known as the Nyquist frequency, the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

[[Image:Dmpfg_007.jpg|thumb|360px|right]]
Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A second-order low-pass filter, for example,
requires two passive components. An all-analog short-time Fourier
transform, a few hundred. Well, maybe a thousand if you want to build
something really fancy [bang on the 3585]. Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And since we
do, digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds noise and distortion, usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an inductor the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use lossy algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern conversion stages are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Raw_.28digital_audio.29_meat|Discuss this section]]</small>

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist: for example, the
Sigma-Delta coding used by the SACD, which is a form of Pulse Density
Modulation. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.

===sample rate===

The first parameter is the sampling rate. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore, the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally band-limited voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate: the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like—a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

[[Image:Dmpfg_008.jpg|thumb|360px|right]]
Stepping back for just a second, the French mathematician Jean
Baptiste Joseph Fourier showed that we can also think of signals like
audio as a set of component frequencies. This frequency-domain
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it a different way. Here we see the
frequency-domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a low-pass filter
before sampling, the sampling process will fold them down into the
representable frequency range as aliasing distortion.

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the low pass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
20kHz but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build, and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the low pass has an extra
octave or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.

===sample format===

The second fundamental PCM parameter is the sample format; that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was eight-bit linear, encoded as an unsigned byte. The
dynamic range is limited to about 50dB and the quantization noise, as
you can hear, is pretty severe. Eight-bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called A-law and mu-law. These formats encode a roughly
14 bit dynamic range into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight-bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16- or 24-bit two's-complement signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels, and thus
beyond the maximum representable range, are clipped.

In mixing and mastering, it's not unusual to use floating-point
numbers for PCM instead of integers. A 32 bit IEEE754 float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating-point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating-point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in big- or little-endian order, and both
endiannesses are common. For example, Microsoft WAV files are little-endian,
and Apple AIFC files tend to be big-endian. Be aware of it.

===channels===

The third PCM parameter is the number of channels. The convention in
raw PCM is to encode multiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.

===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is ''so easy''! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Video_vegetables_.28they.27re_good_for_you.21.29|Discuss this section]]</small>

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. Raw CD audio is about 1.4 megabits
per second. Raw 1080i HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process and store per
second. By Moore's law... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty year old black and white television could still show a
normal analog television broadcast. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.

===resolution and aspect===

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of scanlines in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
bandwidth. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of 704 by 480, a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of 10:11, making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.

===frame rate and interlacing===

The second obvious video parameter is the frame rate, the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
interlacing.

In the very earliest days of broadcast video, engineers sought the
fastest practical framerate to smooth motion and to minimize flicker
on phosphor-based CRTs. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
deinterlace a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.

===gamma===

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary gamma correction circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard sRGB computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.

===color and colorspace===

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as additive primaries to
produce a full range of color output. The primary pigments in
printing are Cyan, Magenta, and Yellow for the same reason; pigments
are subtractive, and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be and sometimes is represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to luminosity than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution luma channel, the black & white, along with
additional, often lower resolution chroma channels, the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset and quantized for digital video, it's
usually more correctly called Y'CbCr, but the more generic term YUV is
widely used to decribe all the analog and digital variants of this
color model.

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually halved or even quartered in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are 4:4:4 video, which isn't actually subsampled at all, 4:2:2 video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, 4:2:0 video in which both the horizonal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, 4:1:1 and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, motion
JPEG, MPEG-1 video, MPEG-2 video, DV, Theora and WebM all use or can
use 4:2:0 subsampling, but they site the chroma pixels three different
ways.

Motion JPEG, MPEG1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizaare.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizonatal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least 50 different formats in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings due either to
convenience once-upon-a-time on some particular piece of hardware or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or fourcc code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. YV12
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of several YUV colorspace definitions.

===done!===

That wraps up our not so quick and yet very incomplete tour of raw
video. The good news is we can already get quite alot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Containers==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Containers|Discuss this section]]</small>

So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally visible
structure, but they're often uniformly sized. We could just string
them together in a rigid pre-determined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames though aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other metadata we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata, that is, data about the data, within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a container. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate and present the media. In general, any container can hold
any kind of data. And data can be put into any container.

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then--- Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:
"Boo Boo Coming", by Joel Forrester
Performed by the Microscopic Septet
Used by permission of Cuneiform Records.
Original source track All Rights Reserved.
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>
==The making of…==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#The_making_of.E2.80.A6|Discuss this section]]</small>
===Equipment===
====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking alot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===
====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orage audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-21T21:45:50Z

Mindspillage: /* Analog vs Digital */ sp, punct

<small>''Wiki edition''</small>
[[Image:Dmpfg_001.jpg|thumb|360px|right]]
This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

[[Image:Dmpfg_002.jpg|thumb|360px|right]]
It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|thumb|360px|right]]

Sound is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the 'scope down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's continuous in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is discrete in both value and time.
In the simplest and most common system, called Pulse Code Modulation,
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

[[Image:Dmpfg_006.jpg|thumb|360px|right]]
Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the Sampling Theorem says we can and tells us
how. Published in its most recognizable form by Claude Shannon in 1949
and built on the work of Nyquist, and Hartley, and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangeable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The telegraph predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... tickertape. Harry Nyquist of Bell Labs was
researching telegraph pulse transmission when he published his
description of what later became known as the Nyquist frequency, the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

[[Image:Dmpfg_007.jpg|thumb|360px|right]]
Audio had always been manipulated as an analog signal because... well,
gee, it's so much easier. A second-order low-pass filter, for example,
requires two passive components. An all-analog short-time Fourier
transform, a few hundred. Well, maybe a thousand if you want to build
something really fancy [bang on the 3585]. Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And since we
do, digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is a lot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds noise and distortion, usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up a lot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an inductor the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated, and transmitted without adding any noise
or distortion. We do use lossy algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern conversion stages are very, very good. By the
standards of our ears, we can consider them practically lossless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Wikipedia: [[wiki:Nyquist–Shannon_sampling_theorem|Nyquist–Shannon sampling theorem]]
*MIT OpenCourseWare [http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-003-signals-and-systems-spring-2010/lecture-notes/ Lecture notes from 6.003 signals and systems.]
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist, for example the
Sigma-Delta coding used by the SACD, which is a form of Pulse Density
Modulation. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.

===sample rate===

The first parameter is the sampling rate. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally band-limited voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate, the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like--- a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

[[Image:Dmpfg_008.jpg|thumb|360px|right]]
Stepping back for just a second, the French mathematician Jean
Baptiste Joseph Fourier showed that we can also think of signals like
audio as a set of component frequencies. This frequency domain
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it a different way. Here we see the
frequency domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a lowpass filter
before sampling, the sampling process will fold them down into the
representable frequency range as aliasing distortion.

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the lowpass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
20kHz but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the lowpass has an extra
octave or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.

===sample format===

The second fundamental PCM parameter is the sample format, that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was eight bit linear, encoded as an unsigned byte. The
dynamic range is limited to about 50dB and the quantization noise, as
you can hear, is pretty severe. Eight bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called A-law and mu-law. These formats encode a roughly
14 bit dynamic range into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16 or 24 bit two's-complement signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels and thus
beyond the maximum representable range are clipped.

In mixing and mastering, it's not unusual to use floating point
numbers for PCM instead of integers. A 32 bit IEEE754 float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in big or little endian order, and both
endiannesses are common. For example, Microsoft WAV files are little endian,
and Apple AIFC files tend to be big-endian. Be aware of it.

===channels===

The third PCM parameter is the number of channels. The convention in
raw PCM is to encode mutiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.

===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is _so easy_! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Video vegetables (they're good for you!)==

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. Raw CD audio is about 1.4 megabits
per second. Raw 1080i HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process and store per
second. By Moore's law... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty year old black and white television could still show a
normal analog television broadcast. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.

===resolution and aspect===

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of scanlines in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
bandwidth. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of 704 by 480, a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of 10:11, making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.

===frame rate and interlacing===

The second obvious video parameter is the frame rate, the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
interlacing.

In the very earliest days of broadcast video, engineers sought the
fastest practical framerate to smooth motion and to minimize flicker
on phosphor-based CRTs. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
deinterlace a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.

===gamma===

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary gamma correction circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard sRGB computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.

===color and colorspace===

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as additive primaries to
produce a full range of color output. The primary pigments in
printing are Cyan, Magenta, and Yellow for the same reason; pigments
are subtractive, and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be and sometimes is represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to luminosity than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution luma channel, the black & white, along with
additional, often lower resolution chroma channels, the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset and quantized for digital video, it's
usually more correctly called Y'CbCr, but the more generic term YUV is
widely used to decribe all the analog and digital variants of this
color model.

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually halved or even quartered in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are 4:4:4 video, which isn't actually subsampled at all, 4:2:2 video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, 4:2:0 video in which both the horizonal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, 4:1:1 and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, motion
JPEG, MPEG-1 video, MPEG-2 video, DV, Theora and WebM all use or can
use 4:2:0 subsampling, but they site the chroma pixels three different
ways.

Motion JPEG, MPEG1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizaare.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizonatal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least 50 different formats in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings due either to
convenience once-upon-a-time on some particular piece of hardware or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or fourcc code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. YV12
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of several YUV colorspace definitions.

===done!===

That wraps up our not so quick and yet very incomplete tour of raw
video. The good news is we can already get quite alot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
* Fixme
</div></center>

<br style="clear:both;"/>

==Containers==
So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally visible
structure, but they're often uniformly sized. We could just string
them together in a rigid pre-determined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames though aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other metadata we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata, that is, data about the data, within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a container. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate and present the media. In general, any container can hold
any kind of data. And data can be put into any container.

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then--- Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:
"Boo Boo Coming", by Joel Forrester
Performed by the Microscopic Septet
Used by permission of Cuneiform Records.
Original source track All Rights Reserved.
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>
==The making of…==

===Equipment===

====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking alot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===

====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orage audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-21T21:39:01Z

Mindspillage: /* Introduction */ punctuation

<small>''Wiki edition''</small>
[[Image:Dmpfg_001.jpg|thumb|360px|right]]
This first video from Xiph.Org presents the technical foundations of modern digital media via a half-hour firehose of information. One community member called it "a Uni lecture I never got but really wanted."

The program offers a brief history of digital media, a quick summary of the sampling theorem, and myriad details of low level audio and video characterization and formatting. It's intended for budding geeks looking to get into video coding, as well as the technically curious who want to know more about the media they wrangle for work or play.
<br/><br/><br/>
<center><font size="+2">[http://www.xiph.org/video/vid1.shtml Download or Watch online]</font></center>
<br style="clear:both;"/>
<hr/>

==Introduction==
<small>[[Talk:A_Digital_Media_Primer_For_Geeks_(episode_1)#Introduction|Discuss this section]]</small>

Workstations and high-end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

[[Image:Dmpfg_002.jpg|thumb|360px|right]]
It's no problem finding consumers for digital media. But here I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Analog vs Digital==
[[Image:Dmpfg_004.jpg|thumb|360px|right]]

Sound is the propagation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the 'scope down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's continuous in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is discrete in both value and time.
In the simplest and most common system, called Pulse Code Modulation,
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

[[Image:Dmpfg_006.jpg|thumb|360px|right]]
Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the Sampling Theorem says we can and tells us
how. Published in its most recognizable form by Claude Shannon in 1949
and built on the work of Nyquist, and Hartley, and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are fairly recent, analog signal technology must have
come first. Nope. Digital is actually older. The telegraph predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... Tickertape. Harry Nyquist of Bell Labs was
researching telegraph pulse transmission when he published his
description of what later became known as the Nyquist frequency, the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal technology progressed rapidly and side-by-side.

[[Image:Dmpfg_007.jpg|thumb|360px|right]]
Audio had always been manipulated as an analog signal because... well,
gee it's so much easier. A second-order lowpass filter, for example,
requires two passive components. An all-analog short-time Fourier
transform, a few hundred. Well, maybe a thousand if you want to build
something really fancy [bang on the 3585]. Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And since we
do, digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is alot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds noise and distortion, usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up alot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an inductor the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated and transmitted without adding any noise
or distortion. We do use lossy algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern conversion stages are very, very good. By the
standards of our ears, we can consider them practically losless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

<center><div style="background-color:#DDDDFF;border-color:#CCCCDD;border-style:solid;width:80%;padding:0 1em 1em 1em;text-align:left;">
'''Going deeper…'''
*Fixme: Some good HTML5 baseline codec debate article
*[http://diveintohtml5.org/video.html Dive into HTML5] web video tutorial
</div></center>

<br style="clear:both;"/>

==Raw (digital audio) meat==

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist, for example the
Sigma-Delta coding used by the SACD, which is a form of Pulse Density
Modulation. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.

===sample rate===

The first parameter is the sampling rate. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally band-limited voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate, the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like--- a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's virtually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

[[Image:Dmpfg_008.jpg|thumb|360px|right]]
Stepping back for just a second, the French mathematician Jean
Baptiste Joseph Fourier showed that we can also think of signals like
audio as a set of component frequencies. This frequency domain
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it a different way. Here we see the
frequency domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a lowpass filter
before sampling, the sampling process will fold them down into the
representable frequency range as aliasing distortion.

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the lowpass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
20kHz but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the lowpass has an extra
octave or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.

===sample format===

The second fundamental PCM parameter is the sample format, that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was eight bit linear, encoded as an unsigned byte. The
dynamic range is limited to about 50dB and the quantization noise, as
you can hear, is pretty severe. Eight bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called A-law and mu-law. These formats encode a roughly
14 bit dynamic range into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16 or 24 bit two's-complement signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels and thus
beyond the maximum representable range are clipped.

In mixing and mastering, it's not unusual to use floating point
numbers for PCM instead of integers. A 32 bit IEEE754 float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in big or little endian order, and both
endiannesses are common. For example, Microsoft WAV files are little endian,
and Apple AIFC files tend to be big-endian. Be aware of it.

===channels===

The third PCM parameter is the number of channels. The convention in
raw PCM is to encode mutiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.

===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is _so easy_! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

<br style="clear:both;"/>
==Video vegetables (they're good for you!)==

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. Raw CD audio is about 1.4 megabits
per second. Raw 1080i HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process and store per
second. By Moore's law... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatibility. Up until just last year in the
US, a sixty year old black and white television could still show a
normal analog television broadcast. That's actually a really neat
trick.

The downside to backward compatibility is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.

===resolution and aspect===

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of scanlines in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
bandwidth. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of 704 by 480, a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of 10:11, making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.

===frame rate and interlacing===

The second obvious video parameter is the frame rate, the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
interlacing.

In the very earliest days of broadcast video, engineers sought the
fastest practical framerate to smooth motion and to minimize flicker
on phosphor-based CRTs. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
deinterlace a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.

===gamma===

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary gamma correction circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard sRGB computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.

===color and colorspace===

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as additive primaries to
produce a full range of color output. The primary pigments in
printing are Cyan, Magenta, and Yellow for the same reason; pigments
are subtractive, and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be and sometimes is represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to luminosity than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution luma channel, the black & white, along with
additional, often lower resolution chroma channels, the color. The
luma channel, Y, is produced by weighting and then adding the separate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset and quantized for digital video, it's
usually more correctly called Y'CbCr, but the more generic term YUV is
widely used to decribe all the analog and digital variants of this
color model.

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually halved or even quartered in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are 4:4:4 video, which isn't actually subsampled at all, 4:2:2 video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, 4:2:0 video in which both the horizonal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, 4:1:1 and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, motion
JPEG, MPEG-1 video, MPEG-2 video, DV, Theora and WebM all use or can
use 4:2:0 subsampling, but they site the chroma pixels three different
ways.

Motion JPEG, MPEG1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG2 sites chroma pixels between lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizaare.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizonatal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in separate
planes stacked in order in the frame. There are at least 50 different formats in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings due either to
convenience once-upon-a-time on some particular piece of hardware or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or fourcc code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. YV12
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of several YUV colorspace definitions.

===done!===

That wraps up our not so quick and yet very incomplete tour of raw
video. The good news is we can already get quite alot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

<br style="clear:both;"/>
==Containers==
So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally visible
structure, but they're often uniformly sized. We could just string
them together in a rigid pre-determined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames though aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that separate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other metadata we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata, that is, data about the data, within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a container. Containers provide framing for the data blobs,
interleave and identify multiple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate and present the media. In general, any container can hold
any kind of data. And data can be put into any container.

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then--- Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:
"Boo Boo Coming", by Joel Forrester
Performed by the Microscopic Septet
Used by permission of Cuneiform Records.
Original source track All Rights Reserved.
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

<br style="clear:both;"/>
==The making of…==

===Equipment===

====Camera====
Canon HV40 HDV camera w/ wide-angle lens operating on a tripod.

The wide angle lens gives the camera a nice close macro mode, and approximately triples the amount of light coming into the sensor for a given zoom/aperture. Useful for shooting indoors at night.

No additional lighting kit was used.

====Audio====

Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker, run through a cheap Behringer portable mixer and into the camera's microphone input.

No additional audio kit was used.

====Sundries====

Whiteboard markers by 'Bic'

Drawing aids by Staedtler, McMaster Carr, and 'Generic'.

===Video shooting sequence===

Scenes were pre-scripted and memorized, usually with lots of on-the-fly revision. (In the future... I'm totally getting a teleprompter. Wh000. OTOH, I can totally rattle off the entire video script from beginning to end as a party trick, thus ensuring I'll not be invited to many parties --Monty).

Diagrams were drawn by hand on a physical whiteboard with whiteboard markers and magnetic T-squares, triangles, and yardsticks. Despite looking alot like greenscreen work, there is no image compositing in use (actually-- there are two small composites where an error in a whiteboard diagram was corrected by subtracting part of the original image and then adding a corrected version of the diagram).

Camera operated in 24F shutter priority mode (Tv set to "24") with exposure and white balance both calibrated to the white board (or a white piece of paper) and locked. Microphone attenuation setting active, with gain locked such that room noise peaked at -40dB (all the rooms in the shooting sequences were noisy due to the building's ventilation system, or active equipment). Lighting in the whiteboard rooms tended to be odd, with little relative light cast on a presenter standing just in front of the whiteboard; a presenter is practically standing in the room's only shadow. Most of the room light is focused on the table and walls. Additional fill lighting kit would have been useful, but for the first vid, I didn't want 'perfect' to be the enemy of 'good'.

Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.

Continuous capture to a Thinkpad with firewire input via a simple gstreamer script.

===Production sequence===

====All hail Cinelerra. You better hail, or Cinelerra will get pissy about it.====

Most of the production sequence hinged on making Cinelerra happy; it is a hulking rusty cast iron WWI tank of a program that can seem like it's composed entirely of compressed bugs. That said, it was neither particularly crashy nor did it ever accidentally corrupt or lose work. It was also the only FOSS editor with a working 2D compositor. It got the job done once I found a workflow it would cope with (and fixed a number of bugs; these fixes are available from my cinelerra Git repo at http://git.xiph.org/?p=users/xiphmont/cinelerraCV.git;a=summary)

====Choosing takes====

Each shooting session yielded four to six hours of raw video. The first step was to load the raw video into the cinelerra timeline, label each complete take, compare and choose the take to use, then render the chosen take out to a raw clip as a YUV4MPEG raw video file and a WAV raw audio file. Be careful that Settings->Align Cursor On Frames is set, else the audio and video renders won't start on the same boundary.

====Postprocessing====

At this point, the raw video clips were adjusted for gamma, contrast and saturation in gstreamer and mplayer. In the earlier shoots the camera was underexposing due to pilot error, which required quite a bit of gamma and saturation inflation to 'correct' (there is no real correction as the low-end data is gone, but it's possible to make it look better). Later shoots used saner settings and the adjustments were mostly to keep different shooting sessions more uniform. The whiteboard tends not to look white because it's mildly reflective, and picked up the color of the cyan and orage audio baffles in the room like a big diffuse mirror.

The audio was both noisy (due to the building's ventilation system which either sounded like a low loud rumble or a jet-engine taking off) and reverberant (the rooms were glass on two sides and plaster on the other two). Early takes used no additional sound absorbing material in the rooms, and the Postfish filtering and deverb was used heavily. It gives the early audio in the vid a slightly odd, processed feel (I had almost decided the original audio was simply unusable). Later takes used some big fleece 'soft flats' in the room to absorb some additional reverb, and the later takes are less heavily filtered.

The postfish filtering chain used declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (noise gating), single compand (for volume levelling) and EQ (the Crown mics are nice, but are very midrange heavy).

====Special Effects====

Audio special effects were one-offs, mostly done using Sox. The processed demo sections of audio were then spliced back into the original audio takes using Audactity.

Video special effects (eg, removing a color channel, etc) were done by writing quick, one-off filters in C for y4oi. A few effects were done by dumping a take as a directory full of PNGs and then batch-processing the PNGs again using a one-off C program, then reassembling with mplayer. Video effects were then stitched back into the original video takes in Cinelerra.

====Editing====

All editing was done in Cinelerra. This primarily consisted of stitching the individual takes back together with crossfades. All input and rendering output were done with raw YUV4MPEG and WAV files. Note that making this work well and correctly required several patches to the YUV4MPEG handler and colorspace conversion code.

====Encoding====

Encoding was done by hand external to Cinelerra using mplayer for final postprocessing, the example_encoder included with the [Ptalarbvorm] Theora source distribution, and ivfenc for WebM.

Sample Theora encode command line (note this is using an mplayer patched for y4o support; it could be done just as easily with a yuv4mpeg pipe):

# 360p, 128-ish (a4) audio + 500-ish (v50) video
mplayer -vf hqdn3d,scale=640:360,gradfun=1.5,unsharp=l3x3:.1,yuv4ogg complete2.m2v -fast -noconsolecontrols -vo null > /dev/null & ~/MotherfishSVN/theora-ptalarbvorm/examples/encoder_example -a 4 -v 50 -k 240 complete2.wav output.y4o -o A_Digital_Media_Primer_For_Geeks-360p-a4+v50.ogv

Videos/A Digital Media Primer For Geeks

2010-09-15T13:03:00Z

Mindspillage: /* Introduction */ sp

<small>''Wiki edition''</small>

<hr/>
*Watch
*Download
*Up to videos page
<hr/>

==Introduction==
Workstations and high end personal computers have been able to
manipulate digital audio pretty easily for about fifteen years now.
It's only been about five years that a decent workstation's been able
to handle raw video without a lot of expensive special purpose
hardware.

But today even most cheap home PCs have the processor power and
storage necessary to really toss raw video around, at least without
too much of a struggle. So now that everyone has all of this cheap media-capable hardware,
more people, not surprisingly, want to do interesting
things with digital media, especially streaming. YouTube was the first huge
success, and now everybody wants in.

Well good! Because this stuff is a lot of fun!

It's no problem finding consumers for digital media. But here, I'd
like to address the engineers, the mathematicians, the hackers, the
people who are interested in discovering and making things and
building the technology itself. The people after my own heart.

Digital media, compression especially, is perceived to be super-elite,
somehow incredibly more difficult than anything else in computer
science. The big industry players in the field don't mind this
perception at all; it helps justify the staggering number of very
basic patents they hold. They like the image that their media
researchers "are the best of the best, so much smarter than anyone
else that their brilliant ideas can't even be understood by mere
mortals. This is bunk.

Digital audio and video and streaming and compression offer endless
deep and stimulating mental challenges, just like any other
discipline. It seems elite because so few people have been been
involved. So few people have been involved perhaps because so few
people could afford the expensive, special-purpose equipment it
required. But today, just about anyone watching this video has a
cheap, general-purpose computer powerful enough to play with the big
boys. There are battles going on today around HTML5 and browsers and
video and open vs. closed. So now is a pretty good time to get
involved. The easiest place to start is probably understanding the
technology we have right now.

This is an introduction. Since it's an introduction, it glosses over a
ton of details so that the big picture's a little easier to see.
Quite a few people watching are going to be way past anything that I'm
talking about, at least for now. On the other hand, I'm probably
going to go too fast for folks who really are are brand new to all of
this, so if this is all new, relax. The important thing is to pick out
any ideas that really grab your imagination. Especially pay attention
to the terminology surrounding those ideas, because with those, and
Google, and Wikipedia, you can dig as deep as interests you.

So, without any further ado, welcome to one hell of a new hobby.

==Analog vs Digital==

Sound is the propogation of pressure waves through air, spreading out
from a source like ripples spread from a stone tossed into a pond. A
microphone, or the human ear for that matter, transforms these passing
ripples of pressure into an electric signal. Right, this is
middle school science class, everyone remembers this. Moving on.

That audio signal is a one-dimensional function, a single value
varying over time. If we slow the 'scope down a bit... that should be
a little easier to see. A few other aspects of the signal are
important. It's continuous in both value and time; that is, at any
given time it can have any real value, and there's a smoothly varying
value at every point in in time. No matter how much we zoom in, there
are no discontinuities, no singularities, no instantaneous steps or
points where the signal ceases to exist. It's defined
everywhere. Classic continuous math works very well on these signals.

A digital signal on the other hand is discrete in both value and time.
In the simplest and most common system, called Pulse Code Modulation,
one of a fixed number of possible values directly represents the
instantaneous signal amplitude at points in time spaced a fixed
distance apart. The end result is a stream of digits.

Now this looks an awful lot like this. It seems intuitive that we
should somehow be able to rigorously transform one into the other, and
good news, the Sampling Theorem says we can and tells us
how. Published in its most recognizable form by Claude Shannon in 1949
and built on the work of Nyquist, and Hartley, and tons of others, the
sampling theorem states that not only can we go back and
forth between analog and digital, but also lays
down a set of conditions for which conversion is lossless and the two
representations become equivalent and interchangable. When the
lossless conditions aren't met, the sampling theorem tells us how and
how much information is lost or corrupted.

Up until very recently, analog technology was the basis for
practically everything done with audio, and that's not because most
audio comes from an originally analog source. You may also think that
since computers are farily recent, analog signal technology must have
come first. Nope. Digital is actually older. The telegraph predates
the telephone by half a century and was already fully mechanically
automated by the 1860s, sending coded, multiplexed digital signals
long distances. You know... Tickertape. Harry Nyquist of Bell Labs was
researching telegraph pulse transmission when he published his
description of what later became known as the Nyquist frequency, the
core concept of the sampling theorem. Now, it's true the telegraph
was transmitting symbolic information, text, not a digitized analog
signal, but with the advent of the telephone and radio, analog and
digital signal techology progressed rapidly and side-by-side.

Audio had always been manipulated as an analog signal because... well,
gee it's so much easier. A second-order lowpass filter, for example,
requires two passive components. An all-analog short-time Fourier
transform, a few hundred. Well, maybe a thousand if you want to build
something really fancy [bang on the 3585]. Processing signals
digitally requires millions to billions of transistors running at
microwave frequencies, support hardware at very least to digitize and
reconstruct the analog signals, a complete software ecosystem for
programming and controlling that billion-transistor juggernaut,
digital storage just in case you want to keep any of those bits for
later...

So we come to the conclusion that analog is the only practical way to
do much with audio... well, unless you happen to have a billion
transistors and all the other things just lying around. And since we
do, digital signal processing becomes very attractive.

For one thing, analog componentry just doesn't have the flexibility of
a general purpose computer. Adding a new function to this
beast... yeah, it's probably not going to happen. On a digital
processor though, just write a new program. Software isn't trivial,
but it is alot easier.

Perhaps more importantly though every analog component is an
approximation. There's no such thing as a perfect transistor, or a
perfect inductor, or a perfect capacitor. In analog, every component
adds noise and distortion, usually not very much, but it adds up. Just
transmitting an analog signal, especially over long distances,
progressively, measurably, irretrievably corrupts it. Besides, all of
those single-purpose analog components take up alot of space. Two
lines of code on the billion transistors back here can implement a
filter that would require an inductor the size of a refrigerator.

Digital systems don't have these drawbacks. Digital signals can be
stored, copied, manipulated and transmitted without adding any noise
or distortion. We do use lossy algorithms from time to time, but the
only unavoidably non-ideal steps are digitization and reconstruction,
where digital has to interface with all of that messy analog. Messy
or not, modern conversion stages are very, very good. By the
standards of our ears, we can consider them practically losless as
well.

With a little extra hardware, then, most of which is now small and
inexpensive due to our modern industrial infrastructure, digital audio
is the clear winner over analog. So let us then go about storing it,
copying it, manipulating it, and transmitting it.

==Raw (digital audio) meat==

Pulse Code Modulation is the most common representation for
raw audio. Other practical representations do exist, for example the
Sigma-Delta coding used by the SACD, which is a form of Pulse Density
Modulation. That said, Pulse Code Modulation is far
and away dominant, mainly because it's so mathematically
convenient. An audio engineer can spend an entire career without
running into anything else.

PCM encoding can be characterized in three parameters, making it easy
to account for every possible PCM variant with mercifully little
hassle.

===sample rate===

The first parameter is the sampling rate. The highest frequency an
encoding can represent is called the Nyquist Frequency. The Nyquist
frequency of PCM happens to be exactly half the sampling rate.
Therefore the sampling rate directly determines the highest possible
frequency in the digitized signal.

Analog telephone systems traditionally band-limited voice channels to
just under 4kHz, so digital telephony and most classic voice
applications use an 8kHz sampling rate, the minimum sampling rate
necessary to capture the entire bandwidth of a 4kHz channel. This is
what an 8kHz sampling rate sounds like--- a bit muffled but perfectly
intelligible for voice. This is the lowest sampling rate that's ever
been used widely in practice.

From there, as power, and memory, and storage increased, consumer
computer hardware went to offering 11, and then 16, and then 22, and
then 32kHz sampling. With each increase in the sampling rate and the
Nyquist frequency, it's obvious that the high end becomes a little
clearer and the sound more natural.

The Compact Disc uses a 44.1kHz sampling rate, which is again slightly
better than 32kHz, but the gains are becoming less distinct. 44.1kHz
is a bit of an oddball choice, especially given that it hadn't been
used for anything prior to the compact disc, but the huge success of
the CD has made it a common rate.

The most common hi-fidelity sampling rate aside from the CD is 48kHz.
There's vitually no audible difference between the two. This video,
or at least the original version of it, was shot and produced with
48kHz audio, which happens to be the original standard for
high-fidelity audio with video.

Super-hi-fidelity sampling rates of 88, and 96, and 192kHz have also
appeared. The reason for the sampling rates beyond 48kHz isn't to
extend the audible high frequencies further. It's for a different
reason.

Stepping back for just a second, the French mathemetician Jean
Baptiste Joseph Fourier showed that we can also think of signals like
audio as a set of component frequencies. This frequency domain
representation is equivalent to the time representation; the signal is
exactly the same, we're just looking at it a different way. Here we see the
frequency domain representation of a hypothetical analog signal we
intend to digitally sample.

The sampling theorem tells us two essential things about the sampling
process. First, that a digital signal can't represent any
frequencies above the Nyquist frequency. Second, and this is the new
part, if we don't remove those frequencies with a lowpass filter
before sampling, the sampling process will fold them down into the
representable frequency range as aliasing distortion.

Aliasing, in a nutshell, sounds freakin' awful, so it's essential to
remove any beyond-Nyquist frequencies before sampling and after
reconstruction.

Human frequency perception is considered to extend to about 20kHz. In
44.1 or 48kHz sampling, the lowpass before the sampling stage has to
be extremely sharp to avoid cutting any audible frequencies below
20kHz but still not allow frequencies above the Nyquist to leak
forward into the sampling process. This is a difficult filter to
build and no practical filter succeeds completely. If the sampling
rate is 96kHz or 192kHz on the other hand, the lowpass has an extra
octave or two for its transition band. This is a much easier filter to
build. Sampling rates beyond 48kHz are actually one of those messy
analog stage compromises.

===sample format===

The second fundamental PCM parameter is the sample format, that is,
the format of each digital number. A number is a number, but a number
can be represented in bits a number of different ways.

Early PCM was eight bit linear, encoded as an unsigned byte. The
dynamic range is limited to about 50dB and the quantization noise, as
you can hear, is pretty severe. Eight bit audio is vanishingly rare
today.

Digital telephony typically uses one of two related non-linear eight
bit encodings called A-law and mu-law. These formats encode a roughly
14 bit dynamic range into eight bits by spacing the higher amplitude
values farther apart. A-law and mu-law obviously improve quantization
noise compared to linear 8-bit, and voice harmonics especially hide
the remaining quantization noise well. All three eight bit encodings,
linear, A-law, and mu-law, are typically paired with an 8kHz sampling
rate, though I'm demonstrating them here at 48kHz.

Most modern PCM uses 16 or 24 bit two's-complement signed integers to
encode the range from negative infinity to zero decibels in 16 or 24
bits of precision. The maximum absolute value corresponds to zero decibels.
As with all the sample formats so far, signals beyond zero decibels and thus
beyond the maximum representable range are clipped.

In mixing and mastering, it's not unusual to use floating point
numbers for PCM instead of integers. A 32 bit IEEE754 float, that's
the normal kind of floating point you see on current computers, has 24
bits of resolution, but a seven bit floating point exponent increases
the representable range. Floating point usually represents zero
decibels as +/-1.0, and because floats can obviously represent
considerably beyond that, temporarily exceeding zero decibels during
the mixing process doesn't cause clipping. Floating point PCM takes
up more space, so it tends to be used only as an intermediate
production format.

Lastly, most general purpose computers still read and
write data in octet bytes, so it's important to remember that samples
bigger than eight bits can be in big or little endian order, and both
endiannesses are common. For example, Microsoft WAV files are little endian,
and Apple AIFC files tend to be big-endian. Be aware of it.

===channels===

The third PCM parameter is the number of channels. The convention in
raw PCM is to encode mutiple channels by interleaving the samples of
each channel together into a single stream. Straightforward and extensible.

===done!===

And that's it! That describes every PCM representation ever. Done.
Digital audio is _so easy_! There's more to do of course, but at this
point we've got a nice useful chunk of audio data, so let's get some
video too.

==Video vegetables (they're good for you!)==

One could think of video as being like audio but with two additional
spatial dimensions, X and Y, in addition to the dimension of time.
This is mathematically sound. The Sampling Theorem applies to all
three video dimensions just as it does the single time dimension of
audio.

Audio and video are obviously quite different in practice. For one,
compared to audio, video is huge. Raw CD audio is about 1.4 megabits
per second. Raw 1080i HD video is over 700 megabits per second. That's
more than 500 times more data to capture, process and store per
second. By Moore's law... that's... let's see... roughly eight
doublings times two years, so yeah, computers requiring about an extra
fifteen years to handle raw video after getting raw audio down pat was
about right.

Basic raw video is also just more complex than basic raw audio. The
sheer volume of data currently necessitates a representation more
efficient than the linear PCM used for audio. In addition, electronic
video comes almost entirely from broadcast television alone, and the
standards committees that govern broadcast video have always been very
concerned with backward compatability. Up until just last year in the
US, a sixty year old black and white television could still show a
normal analog television broadcast. That's actually a really neat
trick.

The downside to backward compatability is that once a detail makes it
into a standard, you can't ever really throw it out again. Electronic
video has never started over from scratch the way audio has multiple
times. Sixty years worth of clever but obsolete hacks necessitated by
the passing technology of a given era have built up into quite a pile,
and because digital standards also come from broadcast television, all
these eldritch hacks have been brought forward into the digital
standards as well.

In short, there are a whole lot more details involved in digital video
than there were with audio. There's no hope of covering them
all completely here, so we'll cover the broad fundamentals.

===resolution and aspect===

The most obvious raw video parameters are the width and height of the
picture in pixels. As simple as that may sound, the pixel dimensions
alone don't actually specify the absolute width and height of the
picture, as most broadcast-derived video doesn't use square pixels.
The number of scanlines in a broadcast image was fixed, but the
effective number of horizontal pixels was a function of channel
bandwidth. Effective horizontal resolution could result in pixels that
were either narrower or wider than the spacing between scanlines.

Standards have generally specified that digitally sampled video should
reflect the real resolution of the original analog source, so a large
amount of digital video also uses non-square pixels. For example, a
normal 4:3 aspect NTSC DVD is typically encoded with a display
resolution of 704 by 480, a ratio wider than 4:3. In this case, the
pixels themselves are assigned an aspect ratio of 10:11, making them
taller than they are wide and narrowing the image horizontally to the
correct aspect. Such an image has to be resampled to show properly on
a digital display with square pixels.

===frame rate and interlacing===

The second obvious video parameter is the frame rate, the number of
full frames per second. Several standard frame rates are in active
use. Digital video, in one form or another, can use all of them. Or,
any other frame rate. Or even variable rates where the frame rate
changes adaptively over the course of the video. The higher the frame
rate, the smoother the motion and that brings us, unfortunately, to
interlacing.

In the very earliest days of broadcast video, engineers sought the
fastest practical framerate to smooth motion and to minimize flicker
on phosphor-based CRTs. They were also under pressure to use the
least possible bandwidth for the highest resolution and fastest frame
rate. Their solution was to interlace the video where the even lines
are sent in one pass and the odd lines in the next. Each pass is
called a field and two fields sort of produce one complete frame.
"Sort of", because the even and odd fields aren't actually from the
same source frame. In a 60 field per second picture, the source frame
rate is actually 60 full frames per second, and half of each frame,
every other line, is simply discarded. This is why we can't
deinterlace a video simply by combining two fields into one frame;
they're not actually from one frame to begin with.

===gamma===

The cathode ray tube was the only available display technology for
most of the history of electronic video. A CRT's output brightness is
nonlinear, approximately equal to the input controlling voltage raised
to the 2.5th power. This exponent, 2.5, is designated gamma, and so
it's often referred to as the gamma of a display. Cameras, though,
are linear, and if you feed a CRT a linear input signal, it looks a
bit like this.

As there were originally to be very few cameras, which were
fantastically expensive anyway, and hopefully many, many television
sets which best be as inexpensive as possible, engineers decided to
add the necessary gamma correction circuitry to the cameras rather
than the sets. Video transmitted over the airwaves would thus have a
nonlinear intensity using the inverse of the set's gamma exponent, so that
once a camera's signal was finally displayed on the CRT, the overall
response of the system from camera to set was back to linear again.

Almost.

There were also two other tweaks. A television camera actually uses a
gamma exponent that's the inverse of 2.2, not 2.5. That's just a
correction for viewing in a dim environment. Also, the exponential
curve transitions to a linear ramp near black. That's just an old
hack for suppressing sensor noise in the camera.

Gamma correction also had a lucky benefit. It just so happens that the
human eye has a perceptual gamma of about 3. This is relatively close
to the CRT's gamma of 2.5. An image using gamma correction devotes
more resolution to lower intensities, where the eye happens to have
its finest intensity discrimination, and therefore uses the available
scale resolution more efficiently. Although CRTs are currently
vanishing, a standard sRGB computer display still uses a nonlinear
intensity curve similar to television, with a linear ramp near black,
followed by an exponential curve with a gamma exponent of 2.4. This
encodes a sixteen bit linear range down into eight bits.

===color and colorspace===

The human eye has three apparent color channels, red, green, and blue,
and most displays use these three colors as additive primaries to
produce a full range of color output. The primary pigments in
printing are Cyan, Magenta, and Yellow for the same reason; pigments
are subtractive, and each of these pigments subtracts one pure color
from reflected light. Cyan subtracts red, magenta subtracts green, and
yellow subtracts blue.

Video can be and sometimes is represented with red, green, and blue
color channels, but RGB video is atypical. The human eye is far more
sensitive to luminosity than it is the color, and RGB tends to spread
the energy of an image across all three color channels. That is, the
red plane looks like a red version of the original picture, the green
plane looks like a green version of the original picture, and the blue
plane looks like a blue version of the original picture. Black and
white times three. Not efficient.

For those reasons and because, oh hey, television just happened to
start out as black and white anyway, video usually is represented as a
high resolution luma channel, the black & white, along with
additional, often lower resolution chroma channels, the color. The
luma channel, Y, is produced by weighting and then adding the seperate
red, green and blue signals. The chroma channels U and V are then
produced by subtracting the luma signal from blue and the luma signal
from red.

When YUV is scaled, offset and quantized for digital video, it's
usually more correctly called Y'CbCr, but the more generic term YUV is
widely used to decribe all the analog and digital variants of this
color model.

===chroma subsampling===

The U and V chroma channels can have the same resolution as the Y
channel, but because the human eye has far less spatial color
resolution than spatial luminosity resolution, chroma resolution is
usually halved or even quartered in the horizontal direction, the
vertical direction, or both, usually without any significant impact on the
apparent raw image quality. Practically every possible subsampling
variant has been used at one time or another, but the common choices
today are 4:4:4 video, which isn't actually subsampled at all, 4:2:2 video in
which the horizontal resolution of the U and V channels is halved, and
most common of all, 4:2:0 video in which both the horizonal and vertical
resolutions of the chroma channels are halved, resulting in U and V
planes that are each one quarter the size of Y.

The terms 4:2:2, 4:2:0, 4:1:1 and so on and so forth, aren't complete
descriptions of a chroma subsampling. There's multiple possible ways
to position the chroma pixels relative to luma, and again, several
variants are in active use for each subsampling. For example, motion
JPEG, MPEG-1 video, MPEG-2 video, DV, Theora and WebM all use or can
use 4:2:0 subsampling, but they site the chroma pixels three different
ways.

Motion JPEG, MPEG1 video, Theora and WebM all site chroma pixels
between luma pixels both horizontally and vertically.

MPEG2 sites chroma pixels betwwen lines, but horizontally aligned with
every other luma pixel. Interlaced modes complicate things somewhat,
resulting in a siting arrangement that's a tad bizaare.

And finally PAL-DV, which is always interlaced, places the chroma
pixels in the same position as every other luma pixel in the
horizonatal direction, and vertically alternates chroma channel on
each line.

That's just 4:2:0 video. I'll leave the other subsamplings as homework for the
viewer. Got the basic idea, moving on.

===pixel formats===

In audio, we always represent multiple channels in a PCM stream by
interleaving the samples from each channel in order. Video uses both
packed formats that interleave the color channels, as well as planar
formats that keep the pixels from each channel together in seperate
planes stacked in order in the frame. There are at least 50 different formats in
these two broad categories with possibly ten or fifteen in common use.

Each chroma subsampling and different bit-depth requires a different
packing arrangement, and so a different pixel format. For a given
unique subsampling, there are usually also several equivalent formats
that consist of trivial channel order rearrangements or repackings due either to
convenience once-upon-a-time on some particular piece of hardware or
sometimes just good old-fashioned spite.

Pixels formats are described by a unique name or fourcc code. There
are quite a few of these and there's no sense going over each one now.
Google is your friend. Be aware that fourcc codes for raw video
specify the pixel arrangement and chroma subsampling, but generally
don't imply anything certain about chroma siting or color space. YV12
video to pick one, can use JPEG, MPEG-2 or DV chroma siting, and any
one of several YUV colorspace definitions.

===done!===

That wraps up our not so quick and yet very incomplete tour of raw
video. The good news is we can already get quite alot of real work
done using that overview. In plenty of situations, a frame of video
data is a frame of video data. The details matter, greatly, when it
come time to write software, but for now I am satisfied that the
esteemed viewer is broadly aware of the relevant issues.

==Containers==
So. We have audio data. We have video data. What remains is the more
familiar non-signal data and straight up engineering that software
developers are used to, and plenty of it.

Chunks of raw audio and video data have no externally visible
structure, but they're often uniformly sized. We could just string
them together in a rigid pre-determined ordering for streaming and
storage, and some simple systems do approximately that. Compressed
frames though aren't necessarily a predictable size, and we usually want
some flexibility in using a range of different data types in streams.
If we string random formless data together, we lose the boundaries
that seperate frames and don't necessarily know what data belongs to
which streams. A stream needs some generalized structure to be
generally useful.

In addition to our signal data, we also have our PCM and video
parameters. There's probably plenty of other metadata we also want to
deal with, like audio tags and video chapters and subtitles, all
essential components of rich media. It makes sense to place this
metadata, that is, data about the data, within the media itself.

Storing and structuring formless data and disparate metadata is the
job of a container. Containers provide framing for the data blobs,
interleave and identify mutliple data streams, provide timing
information, and store the metadata necessary to parse, navigate,
manipulate and present the media. In general, any container can hold
any kind of data. And data can be put into any container.

==Credits==

In the past thirty minutes, we've covered digital audio, video, some
history, some math and a little engineering. We've barely scratched the
surface, but it's time for a well earned break.

There's so much more to talk about, so I hope you'll join me again in
our next episode. Until then--- Cheers!

Written by:
Christopher (Monty) Montgomery
and the Xiph.Org Community

Intro, title and credits music:
"Boo Boo Coming", by Joel Forrester
Performed by the Microscopic Septet
Used by permission of Cuneiform Records.
Original source track All Rights Reserved.
www.cuneiformrecords.com

This Video Was Produced Entirely With Free and Open Source Software

GNU
Linux
Fedora
Cinelerra
The Gimp
Audacity
Postfish
Gstreamer

CC BY-NC-SA
A Co-Production of Xiph.Org and Red Hat Inc.
(C) 2010, Some Rights Reserved

==The making of…==

===Equipment===

Video:
Canon HV40 HDV camera w/ wide-angle lens operating in 24F shutter priority mode on a Tiltall tripod
Shutter set to Tv "24", then exposure and white balance both calibrated to the white board (or a white piece of paper) and locked
Autofocus used for whiteboard scenes, manual focus used for several workshop scenes as the autofocus tended to hunt continuously in very low light.
No additional lighting kit.

Audio:
Two Crown PCC160 boundary microphones placed on a table approximately 4-8 feet in front of the speaker
Portable Behringer (yes, ugh) 602 board used as microphone preamp, stereo output fed directly into HDV camera

Realtime capture to laptop via firewire using a gstreamer script

Audio was postprocessed in Postfish using the declip (for the occasional overrange oops), deverb (remove room reverberation), multicompand (for noise gating), single compand (for volume levelling) and EQ filters.

Video was postprocessed in gstreamer to scale and adjust saturation, gamma, and contrast.

Takes were clipped out of the long HDV grabs using a patched Cinelerra and exported to working source files in raw YUV4MPEG and WAV format. Editing was done on these raw intermediates in Cinelerra.

Some video effects were performed by small one-off C programs that operated directly on the source clips; these effects were then stitched into the main video using Cinelerra.

Text, caption overlays, and still conversions perfromed in the Gimp.

Final renders encoded using libtheora (ptalarbvorm) and libvorbis via the Theora 'example encoder'.

IDABC Questionnaire 2009

2009-11-24T22:33:13Z

Mindspillage: /* Meeting and consultation */ "limited abilities" almost certainly means impairment here; changing language.

<strong>This is a draft document. A work in progress. A scratchpad for ideas. It should not be widely circulated in this form.</strong>

= Context =
We received [http://lists.xiph.org/pipermail/theora/2009-November/002996.html an e-mail] from a consultant studying the suitability of Theora for use in "eGovernment", on behalf of the [http://ec.europa.eu/idabc/ IDABC], an EU governmental agency responsible for "Interoperability" with an emphasis on open source. The investigation is in the context of [http://ec.europa.eu/idabc/en/document/7728 European Interoperability Framework], about which there has been [http://www.computerworlduk.com/community/blogs/index.cfm?entryid=2620&blogid=14&pn=1 some real controversy].

The method of assessment is the Common Assessment Method for Standards and Specifications, including the questions below.

= CAMSS Questions =
== Part 4: Market Criteria ==

This group of Market criteria analyses the formal specification in the scope of its market environment, and more precisely it examines the implementations of the formal specification and the market players. This implies identifying to which extent the formal specification benefits from market support and wide adoption, what are its level of maturity and its capacity of reusability.

Market support is evaluated through an analysis of how many products implementing the formal specification exist, what their market share is and who their end-users are. The quality and the completeness (in case of partitioning) of the implementations of the formal specification can also be analysed. Availability of existing or planned mechanisms to assess conformity of implementations to the standard or to the specification could also be identified. The existence of at least one reference implementation (i.e.: mentioning a recognized certification process) - and of which one is an open source implementation - can also be relevant to the assessment. Wide adoption can also be assessed across domains (i.e.: public and private sectors), in an open environment, and/or in a similar field (i.e.: best practices).

A formal specification is mature if it has been in use and development for long enough that most of its initial problems have been overcome and its underlying technology is well understood and well defined. Maturity is also assessed by identifying if all aspects of the formal specification are considered as validated by usage, (i.e.: if the formal specification is partitioned), and if the reported issues have been solved and documented.

Reusability of a formal specification is enabled if it includes guidelines for its implementation in a given context. The identification of successful implementations of the standard or specification should focus on good practices in a similar field. Its incompatibility with related standards or specifications should also be taken into account.

The ideas behind the Market Criteria can also be expressed in the form of the following questions:

=== Market support ===
* Does the standard have strong support in the marketplace?
: '''Yes. For example, among web browsers, support for Xiph's Ogg, Theora, and Vorbis standards is now included by default in Mozilla Firefox, Google Chrome, and the latest versions of Opera, representing hundreds of millions of installed users just in this market alone. On Windows, DirectShow filters exist which also enable all Windows applications that use the DirectShow framework to use Xiph's Ogg, Theora, and Vorbis standards. A QuickTime component exists which enables use of Xiph's Ogg, Theora, and Vorbis standards in all Mac OS X applications that make use of the QuickTime framework---which includes Safari/Webkit, iMovie, QuickTime, and many others.'''
* What products exist for this formal specification ?
: '''Theora is a video codec, and as such the required products are encoders, decoders, and transmission systems. All three types of products are widely available for Theora.'''
* How many implementations of the formal specification are there?
: '''Xiph does not require implementors to acquire any license before implementing the specification. Therefore, we do not have a definitive count of the number of implementations. In addition to the reference implementation, which has been ported to most modern platforms and highly optimized for x86 and ARM CPUs and TI C64x+ DSPs, we are aware of a number of independent, conformant or mostly-conformant implementations. These include two C decoders ([http://ffmpeg.org/ FFmpeg] and [http://sourceforge.jp/projects/qtheora/ QTheora]), a Java decoder ([http://www.theora.org/cortado/ Jheora]), a [http://www.wreckedgames.com/downloads/cSharpTheora.zip C# decoder], an [http://svn.xiph.org/trunk/theora-fpga/ FPGA decoder], and an [http://sourceforge.net/projects/elphel/ FPGA encoder].'''
* Are there products from different suppliers in the market that implement this formal specification ?
: '''Yes. Corporations such as Atari, Canonical, DailyMotion, Elphel, Fluendo, Google, Mozilla, Novell, Opera, Red Hat, Sun Microsystems, and Ubisoft have supplied products with an implementation of the Theora standard.'''
* Are there many products readily available from a variety of suppliers?
: '''Yes. Theora has been deployed in embedded devices, security cameras, video games, video conferencing systems, web browsers, home theater systems, and many other products. A complete, legal, open-source reference implementation can also be downloaded free of charge, including components for all major media frameworks (DirectShow, gstreamer, and Quicktime), giving most applications the ability to use the codec.'''
* What is the market share of the products implementing the formal specification, versus other implementations of competing formal specifications ?
: '''Theora playback is extremely widely available, covering virtually the entire market of personal computers. Theora is also increasingly available in mobile and embedded devices. Since we do not require licensing for products that implement the specification, we do not have market share numbers that can be compared with competing formal specifications. Because implementations are readily available and free, Theora is included in many products that support multiple codecs, and is sometimes the only video codec included in free software products.'''
* Who are the end-users of these products implementing the formal specification?
: '''The end users are television viewers, video gamers, web surfers, movie makers, business people, video distribution services, and anyone else who interacts with moving pictures.'''

=== Maturity ===
* Are there any existing or planned mechanisms to assess conformity of the implementations of the formal specification?
: '''Yes. In addition to a continuous peer review process, we maintain a suite of [http://v2v.cc/~j/theora_testsuite/ test vectors] that allow implementors to assess decoder conformity. We also provide free online developer support and testing for those attempting to make a conforming implementation. An [http://validator.xiph.org/ online validation service] is available.'''
* Is there a reference implementation (i.e.: mentioning a recognized certification process)?
: '''Yes. Xiph maintains a reference implementation called [http://downloads.xiph.org/releases/theora/ libtheora]. In addition to serving as a reference, libtheora is also highly optimized to achieve the maximum possible speed, accuracy, reliability, efficiency, and video quality. As a result, many implementors of Theora adopt the reference implementation.'''
* Is there an open source implementation?
: '''Yes. libtheora is made available under a permissive BSD-like license. Its open-source nature also contributes to its quality as a reference implementation, as implementors are welcome to contribute their improvements to the reference. There are also several other open source implementations in addition to the reference.'''
* Does the formal specification show wide adoption?
** across different domains? (I.e.: public and private)
: '''Yes. In addition to the private companies mentioned in the previous section, Theora has also been specified as the sole format supported by non-profit organizations such as Wikipedia, currently the 6th largest website in the world, and as one of a small number of preferred formats supported by other public institutions, such as the Norwegian government.'''
** in an open environment?
: '''Yes. On open/free operating systems such as those distributed by Novell/SuSE, Canonical, and Red Hat, Theora is the primary default video codec.'''
** in a similar field? (i.e.: can best practices be identified?)
: '''Yes. Most prominently, Theora has been used for eGovernment video distribution in the United States at [http://metavid.org Metavid]. Metavid is the most comprehensive, interactive archive of video footage from the United States legislature. Metavid not only distributes video; they also enable citizen engagement by allowing them to annotate videos and correct transcripts. Metavid distributes its entire archive in Theora format. Metavid's source code is entirely open and reusable for any purpose, providing instant access to best practices for eGovernment with Theora. Metavid's video display component is also available separately as [http://metavid.org/wiki/Mv_Embed mv_embed], which provides reusable best practices for easy Theora display on the web.'''
: '''Another important user of Theora is Wikipedia, which distributes video exclusively in Theora format. Wikipedia's best practices for Theora distribution are encapsulated in [http://www.mediawiki.org/wiki/Extension:OggHandler OggHandler], which can be freely reused by anyone using the open-source MediaWiki software.'''
* Has the formal specification been in use and development long enough that most of its initial problems have been overcome?
: '''Yes. Theora was derived from VP3, which was originally released in May 2000. The Theora specification was completed in 2004. Theora has now been used in a wide variety of applications, on the full spectrum of computing devices.'''
* Is the underlying technology of the standard well-understood? (e.g., a reference model is well defined, appropriate concepts of the technology are in widespread use, the technology may have been in use for many years, a formal mathematical model is defined, etc.)
: '''Yes. The underlying technology has been in use for nearly a decade, and most of the concepts have been in widespread use for even longer.'''
* Is the formal specification based upon technology that has not been well-defined and may be relatively new?
: '''No. The formal specification is based on technology from the On2 VP3 codec, which is substantially similar to simple block-transform codecs like H.261. This class of codecs is extremely well understood, and has been in active use for over 20 years.'''
* Has the formal specification been revised? (Yes/No, Nof)
: '''The formal specification of the Theora decoder has been stable for years. However, the text of the specification is continuously revised, based on user feedback, to improve the clarity and accuracy of the description of the technology.'''
* Is the formal specification under the auspices of an architectural board? (Yes/No)
: '''No. Although officially maintained by the Xiph.Org Foundation, anyone is free to join this organization, and one need not even be a member to make contributions. However, the core developers will review contributions and make sure they do not contradict the general architecture and they work well with the existing code and the test cases.'''
* Is the formal specification partitioned in its functionality? (Yes/No)
: '''No. Theora is very deliberately not partitioned, to avoid the confusion created by a "standard" composed of many incompatible "profiles". The Theora standard does not have any optional components. A compliant Theora decoder can correctly process any Theora stream.'''
** To what extent does each partition participate to its overall functionality? (NN%)
: '''N/A.'''
** To what extent is each partition implemented? (NN%) (cf market adoption)
: '''N/A.'''

=== Re-usability ===
* Does the formal specification provide guidelines for its implementation in a given organisation?
: '''Yes. For example, [http://theora.org/doc/Theora.pdf the Theora specification] provides "non-normative" advice and explanation for implementors of Theora decoders and encoders, including example algorithms for implementing required mathematical transforms. Xiph also maintains [http://wiki.xiph.org/Main_Page a documentation base] for implementors who desire more guidelines beyond the specification itself.'''
* Can other cases where similar systems implement the formal specification be considered as successful implementations and good practices?
: '''Xiph's standards have successfully been implemented by many organisations in a wide variety of environments. We maintain (non-exhaustive) [http://wiki.xiph.org/TheoraSoftwarePlayers lists] of products which implement Theora support, many of them open source, so that others may use them as a reference when preparing their own products.'''
* Is its compatibility with related formal specification documented?
: '''Yes. For example, [http://theora.org/doc/Theora.pdf the Theora specification] also documents the use of Theora within the [http://www.ietf.org/rfc/rfc3533.txt standard Ogg encapsulation format], and the [http://svn.xiph.org/trunk/theora/doc/draft-ietf-avt-rtp-theora-00.txt TheoraRTP draft specification] explains how to transmit Theora using the [http://tools.ietf.org/html/rfc3550 RTP standard]. In addition, the specification documents Theora's compatibility with ITU-R B.470, ITU-R B.601, ITU-R B.709, SMPTE-170M, [http://tools.ietf.org/html/rfc2044 UTF-8], ISO 10646, and [http://www.xiph.org/vorbis/doc/Vorbis_I_spec.pdf Ogg Vorbis].'''

== Part 5: Standardisation Criteria ==
From Idabc-camss

Note: Throughout this section, “Organisation” refers to the standardisation/fora/consortia body in charge of the formal specification.

Significant characteristics of the way the organisation operates are for example the way it gives the possibility to stakeholders to influence the evolution of the formal specification, or which conditions it attaches to the use of the formal specification or its implementation. Moreover, it is important to know how the formal specification is defined, supported, and made available, as well as how interaction with stakeholders is managed by the organisation during these steps. Governance of interoperability testing with other formal specifications is also indicative.

The standardisation criteria analyses therefore the following elements:

=== Availability of Documentation ===
The availability of documentation criteria is linked to cost and online availability. Access to all preliminary results documentation can be online, online for members only, offline, offline, for members only or not available. Access can be free or for a fee (which fee?).
: '''Every Xiph standard is permanently available online to everyone at no cost. For example, we invite everyone to download [http://theora.org/doc/Theora.pdf the most up-to-date copy of the Theora specification], and [http://xiph.org/vorbis/doc/Vorbis_I_spec.html the latest revision of Vorbis]. All previous revisions are available from Xiph's [http://svn.xiph.org/ revision control system].'''

=== Intellectual Property Right ===
The Intellectual Property Rights evaluation criteria relates to the ability for implementers to use the formal specification in products without legal or financial implications. The IPR policy of the organisation is therefore evaluated according to:
* the availability of the IPR or copyright policies of the organisation (available on-line or off-line, or not available);
: '''The reference implementations of each codec include all necessary IPR and copyright licenses for that codec, including all documentation, and are freely available to everyone.'''
* the organisation’s governance to disclose any IPR from any contributor (ex-ante, online, offline, for free for all, for a fee for all, for members only, not available);
: '''Xiph does not require the identification of specific patents that may be required to implement a standard; however, it does require an open-source compatible, royalty free license from a contributor for any such patents they may own before the corresponding technology can be included in a standard. These licenses are made available online, for free, to all parties.'''
* the level of IPR set "mandatory" by the organisation (no patent, royalty free patent, patent and RAND with limited liability , patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none);
: '''All standards, specifications, and software published by the Xiph.Org Foundation are required to have "open-source compatible" IPR. This means that a contribution must either be entirely clear of any known patents, or any patents that read upon the contribution must be available under a transferable, irrevocable public nonassertion agreement to all people everywhere. For example, see [http://svn.xiph.org/trunk/theora/LICENSE our On2 patent nonassertion warrant]. Other common "royalty free" patent licenses are either not transferable, are revocable under certain conditions (such as patent infringement litigation against the originating party), or otherwise impose restrictions that would prevent distribution under common [http://www.opensource.org/ OSI]-approved licenses. These would not be acceptable.'''
* the level of IPR "recommended" by the organisation (no patent, royalty free patent, patent and RAND with limited liability, patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none). [Note: RAND (Reasonable and Non Discriminatory License) is based on a "fairness" concept. Companies agree that if they receive any patents on technologies that become essential to the standard then they agree to allow other groups attempting to implement the standard to use these patents and they agree that the charges for the patents shall be reasonable. "RAND with limited availability" is a version of RAND where the "reasonable charges" have an upper limit.]
: '''Xiph's recommended IPR requirements are the same as our mandatory requirements.'''

=== Accessibility ===

The accessibility evaluation criteria describe the importance of equal and safe accessibility by the users of implementations of formal specifications. This aspect can be related to safety (physical safety and conformance safety) and accessibility of physical impaired people (design for all).

Focus is made particularly on accessibility and conformance safety. Conformance testing is testing to determine whether a system meets some specified formal specification. The result can be results from a test suite. Conformance validation is when the conformance test uniquely qualifies a given implementation as conformant or not. Conformance certification is a process that provides a public and easily visible "stamp of approval" that an implementation of a standard validates as conformant.

The following questions allow an assessment of accessibility and conformance safety:
* Does a mechanism that ensures disability support by a formal specification exist? (Y/N)
: '''Yes. Xiph ensures support for users with disabilities by providing specifications for accessible technologies independent of the codec itself. Notably, the Xiph [http://wiki.xiph.org/OggKate OggKate] codec for time-aligned text and image content provides support for subtitles for internationalisation, captions for the hearing-impaired, and textual audio descriptions for the visually impaired. Further, Ogg supports multiple tracks of audio and video content in one container, such that sign language tracks and audio descriptions can be included in one file. For this to work, Xiph has defined [http://wiki.xiph.org/Ogg_Skeleton Skeleton] which holds metadata about each track encapsulated within a single Ogg file. When Theora is transmitted or stored in an Ogg container, it is automatically compatible with these accessibility measures.'''
* Is conformance governance always part of a standard? (Y/N)
: '''Yes. Xiph's standards always precisely specify the requirements that an implementation must meet in order to be considered conformant.'''
* Is a conformance test offered to implementers? (Y/N)
: '''Yes. Xiph maintains a freely available suite of [http://v2v.cc/~j/theora_testsuite/ test vectors] and an [http://validator.xiph.org online validation service] that can be used by anyone to check confirm basic conformance, in addition to tools such as the oggz-validate program included with liboggz, which has been widely used for conformance testing.'''
* Is conformance validation available to implementers? (Y/N)
: '''Yes. Informal conformance testing is available to implementors upon request, and Xiph has provided such testing for a number of implementations in the past.'''
* Is conformance certification available? (Y/N)
: '''Yes. Xiph does not require certification, but maintains the right to withhold the use of our trademarks from implementors who act in bad faith. Implementors may, however, request explicit permission to use our trademarks with a conforming implementation.'''
* Is localisation of a formal specification possible? (Y/N)
: '''Yes. We welcome anyone who wishes to translate Xiph specifications into other languages. We have no policy requiring that the normative specification be written in English.'''

=== Interoperability governance ===
The interoperability governance evaluation criteria relates to how interoperability is identified and maintained between interoperable formal specifications. In order to do this, the organisation may provide governance for:
* open identification in formal specifications,
: '''Yes. The Xiph codecs can be precisely identified by [http://wiki.xiph.org/index.php/MIMETypesCodecs their MIME types], as formally defined by [http://tools.ietf.org/html/rfc5334 IETF RFC 5334], an open specification.'''
* open negotiation in formal specifications,
: '''Yes. For example, a [http://tools.ietf.org/html/draft-barbato-avt-rtp-theora-01 draft RTP specification] describes how Theora interoperates with the [http://tools.ietf.org/html/rfc3264 Session Description Protocol] (SDP), a mechanism for negotiating the parameters of RTP sessions.'''
* open selection in formal specifications.
: '''Yes. There are many open specifications that provide a mechanism for selecting Theora from among many codecs. One such specification is [http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#video HTML 5 video], which allows the user agent to select Theora based on its MIME type, using [http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#the-source-element the source element].'''

=== Meeting and consultation ===
The meeting and consultation evaluation criteria relates to the process of defining a formal specification. As formal specifications are usually defined by committees, and these committees normally consist of members of the organisation, this criteria studies how to become a member and which are the financial barriers for this, as well as how are non-members able to have an influence on the process of defining the formal specification. It analyses:
* if the organisation is open to all types of companies and organisations and to individuals;
: '''Yes. Xiph welcomes representatives from all companies and organizations as well as all individuals.'''
* if the standardisation process may specifically allow participation of members with limited abilities when relevant;
: '''Yes. Standardization occurs almost entirely in text-based, accessible internet communications channels, allowing participants with disabilities to engage fully in the standards development process.'''
* if meetings are open to all members;
: '''Xiph meetings are open to everyone. We charge no fee for and place no restrictions on attendance or participation. For example, anyone interested in contributing to the Theora specification may join [http://lists.xiph.org/pipermail/theora-dev/ the Theora development mailing list].'''
* if all can participate in the formal specification creation process;
: '''Yes. All people are welcome to participate in the specification creation process. No dues or fees are required to participate.'''
* if non-members can participate in the formal specification creation process.
: '''Yes. Xiph does not maintain an explicit list of members, and no one is excluded from contributing to specifications as they are developed.'''

=== Consensus ===
Consensus is decision making primarily with regard to the approval of formal specifications and review with interest groups (non-members). The consensus evaluation criterion is evaluated with the following questions:
* Does the organisation have a stated objective of reaching consensus when making decisions on standards?
: '''There is no explicitly stated objective of reaching consensus. However, when new contributions are made, the key specification developers will be able to veto the introduction of a new feature. Generally, differences are discussed openly and new features are adapted until they fit the overall architecture of the standard, at which stage they are introduced into the specification, standard and software.'''
* If consensus is not reached, can the standard be approved? (answers are: cannot be approved but referred back to working group/committee, approved with 75% majority, approved with 66% majority, approved with 51% majority, can be decided by a "director" or similar in the organisation).
: '''The standard can be approved without consensus via the decision of a "director" or similar.'''
* Is there a formal process for external review of standard proposals by interest groups (nonmembers)?
: '''Since anyone may participate in the development process and make proposals, there is no need for a separate formal process to include proposals by nonmembers.'''

=== Due Process ===
The due process evaluation criteria relates to the level of respect of each member of the organisation with regard to its rights. More specifically, it must be assured that if a member believes an error has been made in the process of defining a formal specification, it must be possible to appeal this to an independent, higher instance. The question is therefore: can a member formally appeal or raise objections to a procedure or to a technical specification to an independent, higher instance?

: '''Yes. Even if a member fails an appeal within the organization, because all of the technology Xiph standardizes is open and freely implementable, they are always free to develop their own, competing version. Such competing versions may even still be eligible for standardization under the Xiph umbrella.'''

=== Changes to the formal specification ===
The suggested changes made to a formal specification need to be presented, evaluated and approved in the same way as the formal specification was first defined. This criteria therefore applies the above criteria to the changes made to the formal specification(availability of documentation, Intellectual Property Right, accessibility, interoperability governance, meeting and consultation, consensus, due process).

: '''The exact same process is used for revisions to the standard as was used for the original development of the standard, and thus the answers to all of the above questions remain the same.'''

=== Support ===
It is critical that the organisation takes responsibility for the formal specification throughout its life span. This can be done in several ways such as for example a regular periodic review of the formal specification. The support criteria relates to the level of commitment the organisation has taken to support the formal specification throughout its life:
* does the organisation provide support until removal of the published formal specification from public domain (Including this process?)
: '''Xiph.Org standards are never removed from the public domain. Xiph endeavors to provide support for as long as the standard remains in use.'''
* does the organisation make the formal specification still available even when in non-maintenance mode?
: '''Yes. All Xiph.Org standards are freely licensed and will always be available.'''
* does the organisation add new features and keep the formal specification up-to-date?
: '''Yes. Xiph maintains its ecosystem of standards on a continuous basis.'''
* does the organisation rectify problems identified in initial implementations?
: '''Yes. Xiph maintains [https://trac.xiph.org/report a problem reporting system] that is open to the public, and invites everyone to submit suggestions for improvements. Improvements are made both to the standards documents and to the reference implementations.'''
* does the organisation only create the formal specification?
: '''No. Xiph also produces high-quality reusable reference implementations of its standards, released under an open license.'''

<strong>This is a draft document. A work in progress. A scratchpad for ideas. It should not be widely circulated in this form.</strong>

IDABC Questionnaire 2009

2009-11-24T22:21:30Z

Mindspillage: /* Intellectual Property Right */ punctuation

<strong>This is a draft document. A work in progress. A scratchpad for ideas. It should not be widely circulated in this form.</strong>

= Context =
We received [http://lists.xiph.org/pipermail/theora/2009-November/002996.html an e-mail] from a consultant studying the suitability of Theora for use in "eGovernment", on behalf of the [http://ec.europa.eu/idabc/ IDABC], an EU governmental agency responsible for "Interoperability" with an emphasis on open source. The investigation is in the context of [http://ec.europa.eu/idabc/en/document/7728 European Interoperability Framework], about which there has been [http://www.computerworlduk.com/community/blogs/index.cfm?entryid=2620&blogid=14&pn=1 some real controversy].

The method of assessment is the Common Assessment Method for Standards and Specifications, including the questions below.

= CAMSS Questions =
== Part 4: Market Criteria ==

This group of Market criteria analyses the formal specification in the scope of its market environment, and more precisely it examines the implementations of the formal specification and the market players. This implies identifying to which extent the formal specification benefits from market support and wide adoption, what are its level of maturity and its capacity of reusability.

Market support is evaluated through an analysis of how many products implementing the formal specification exist, what their market share is and who their end-users are. The quality and the completeness (in case of partitioning) of the implementations of the formal specification can also be analysed. Availability of existing or planned mechanisms to assess conformity of implementations to the standard or to the specification could also be identified. The existence of at least one reference implementation (i.e.: mentioning a recognized certification process) - and of which one is an open source implementation - can also be relevant to the assessment. Wide adoption can also be assessed across domains (i.e.: public and private sectors), in an open environment, and/or in a similar field (i.e.: best practices).

A formal specification is mature if it has been in use and development for long enough that most of its initial problems have been overcome and its underlying technology is well understood and well defined. Maturity is also assessed by identifying if all aspects of the formal specification are considered as validated by usage, (i.e.: if the formal specification is partitioned), and if the reported issues have been solved and documented.

Reusability of a formal specification is enabled if it includes guidelines for its implementation in a given context. The identification of successful implementations of the standard or specification should focus on good practices in a similar field. Its incompatibility with related standards or specifications should also be taken into account.

The ideas behind the Market Criteria can also be expressed in the form of the following questions:

=== Market support ===
* Does the standard have strong support in the marketplace?
: '''Yes. For example, among web browsers, support for Xiph's Ogg, Theora, and Vorbis standards is now included by default in Mozilla Firefox, Google Chrome, and the latest versions of Opera, representing hundreds of millions of installed users just in this market alone. On Windows, DirectShow filters exist which also enable all Windows applications that use the DirectShow framework to use Xiph's Ogg, Theora, and Vorbis standards. A QuickTime component exists which enables use of Xiph's Ogg, Theora, and Vorbis standards in all Mac OS X applications that make use of the QuickTime framework---which includes Safari/Webkit, iMovie, QuickTime, and many others.'''
* What products exist for this formal specification ?
: '''Theora is a video codec, and as such the required products are encoders, decoders, and transmission systems. All three types of products are widely available for Theora.'''
* How many implementations of the formal specification are there?
: '''Xiph does not require implementors to acquire any license before implementing the specification. Therefore, we do not have a definitive count of the number of implementations. In addition to the reference implementation, which has been ported to most modern platforms and highly optimized for x86 and ARM CPUs and TI C64x+ DSPs, we are aware of a number of independent, conformant or mostly-conformant implementations. These include two C decoders ([http://ffmpeg.org/ FFmpeg] and [http://sourceforge.jp/projects/qtheora/ QTheora]), a Java decoder ([http://www.theora.org/cortado/ Jheora]), a [http://www.wreckedgames.com/downloads/cSharpTheora.zip C# decoder], an [http://svn.xiph.org/trunk/theora-fpga/ FPGA decoder], and an [http://sourceforge.net/projects/elphel/ FPGA encoder].'''
* Are there products from different suppliers in the market that implement this formal specification ?
: '''Yes. Corporations such as Atari, Canonical, DailyMotion, Elphel, Fluendo, Google, Mozilla, Novell, Opera, Red Hat, Sun Microsystems, and Ubisoft have supplied products with an implementation of the Theora standard.'''
* Are there many products readily available from a variety of suppliers?
: '''Yes. Theora has been deployed in embedded devices, security cameras, video games, video conferencing systems, web browsers, home theater systems, and many other products. A complete, legal, open-source reference implementation can also be downloaded free of charge, including components for all major media frameworks (DirectShow, gstreamer, and Quicktime), giving most applications the ability to use the codec.'''
* What is the market share of the products implementing the formal specification, versus other implementations of competing formal specifications ?
: '''Theora playback is extremely widely available, covering virtually the entire market of personal computers. Theora is also increasingly available in mobile and embedded devices. Since we do not require licensing for products that implement the specification, we do not have market share numbers that can be compared with competing formal specifications. Because implementations are readily available and free, Theora is included in many products that support multiple codecs, and is sometimes the only video codec included in free software products.'''
* Who are the end-users of these products implementing the formal specification?
: '''The end users are television viewers, video gamers, web surfers, movie makers, business people, video distribution services, and anyone else who interacts with moving pictures.'''

=== Maturity ===
* Are there any existing or planned mechanisms to assess conformity of the implementations of the formal specification?
: '''Yes. In addition to a continuous peer review process, we maintain a suite of [http://v2v.cc/~j/theora_testsuite/ test vectors] that allow implementors to assess decoder conformity. We also provide free online developer support and testing for those attempting to make a conforming implementation. An [http://validator.xiph.org/ online validation service] is available.'''
* Is there a reference implementation (i.e.: mentioning a recognized certification process)?
: '''Yes. Xiph maintains a reference implementation called [http://downloads.xiph.org/releases/theora/ libtheora]. In addition to serving as a reference, libtheora is also highly optimized to achieve the maximum possible speed, accuracy, reliability, efficiency, and video quality. As a result, many implementors of Theora adopt the reference implementation.'''
* Is there an open source implementation?
: '''Yes. libtheora is made available under a permissive BSD-like license. Its open-source nature also contributes to its quality as a reference implementation, as implementors are welcome to contribute their improvements to the reference. There are also several other open source implementations in addition to the reference.'''
* Does the formal specification show wide adoption?
** across different domains? (I.e.: public and private)
: '''Yes. In addition to the private companies mentioned in the previous section, Theora has also been specified as the sole format supported by non-profit organizations such as Wikipedia, currently the 6th largest website in the world, and as one of a small number of preferred formats supported by other public institutions, such as the Norwegian government.'''
** in an open environment?
: '''Yes. On open/free operating systems such as those distributed by Novell/SuSE, Canonical, and Red Hat, Theora is the primary default video codec.'''
** in a similar field? (i.e.: can best practices be identified?)
: '''Yes. Most prominently, Theora has been used for eGovernment video distribution in the United States at [http://metavid.org Metavid]. Metavid is the most comprehensive, interactive archive of video footage from the United States legislature. Metavid not only distributes video; they also enable citizen engagement by allowing them to annotate videos and correct transcripts. Metavid distributes its entire archive in Theora format. Metavid's source code is entirely open and reusable for any purpose, providing instant access to best practices for eGovernment with Theora. Metavid's video display component is also available separately as [http://metavid.org/wiki/Mv_Embed mv_embed], which provides reusable best practices for easy Theora display on the web.'''
: '''Another important user of Theora is Wikipedia, which distributes video exclusively in Theora format. Wikipedia's best practices for Theora distribution are encapsulated in [http://www.mediawiki.org/wiki/Extension:OggHandler OggHandler], which can be freely reused by anyone using the open-source MediaWiki software.'''
* Has the formal specification been in use and development long enough that most of its initial problems have been overcome?
: '''Yes. Theora was derived from VP3, which was originally released in May 2000. The Theora specification was completed in 2004. Theora has now been used in a wide variety of applications, on the full spectrum of computing devices.'''
* Is the underlying technology of the standard well-understood? (e.g., a reference model is well defined, appropriate concepts of the technology are in widespread use, the technology may have been in use for many years, a formal mathematical model is defined, etc.)
: '''Yes. The underlying technology has been in use for nearly a decade, and most of the concepts have been in widespread use for even longer.'''
* Is the formal specification based upon technology that has not been well-defined and may be relatively new?
: '''No. The formal specification is based on technology from the On2 VP3 codec, which is substantially similar to simple block-transform codecs like H.261. This class of codecs is extremely well understood, and has been in active use for over 20 years.'''
* Has the formal specification been revised? (Yes/No, Nof)
: '''The formal specification of the Theora decoder has been stable for years. However, the text of the specification is continuously revised, based on user feedback, to improve the clarity and accuracy of the description of the technology.'''
* Is the formal specification under the auspices of an architectural board? (Yes/No)
: '''No. Although officially maintained by the Xiph.Org Foundation, anyone is free to join this organization, and one need not even be a member to make contributions. However, the core developers will review contributions and make sure they do not contradict the general architecture and they work well with the existing code and the test cases.'''
* Is the formal specification partitioned in its functionality? (Yes/No)
: '''No. Theora is very deliberately not partitioned, to avoid the confusion created by a "standard" composed of many incompatible "profiles". The Theora standard does not have any optional components. A compliant Theora decoder can correctly process any Theora stream.'''
** To what extent does each partition participate to its overall functionality? (NN%)
: '''N/A.'''
** To what extent is each partition implemented? (NN%) (cf market adoption)
: '''N/A.'''

=== Re-usability ===
* Does the formal specification provide guidelines for its implementation in a given organisation?
: '''Yes. For example, [http://theora.org/doc/Theora.pdf the Theora specification] provides "non-normative" advice and explanation for implementors of Theora decoders and encoders, including example algorithms for implementing required mathematical transforms. Xiph also maintains [http://wiki.xiph.org/Main_Page a documentation base] for implementors who desire more guidelines beyond the specification itself.'''
* Can other cases where similar systems implement the formal specification be considered as successful implementations and good practices?
: '''Xiph's standards have successfully been implemented by many organisations in a wide variety of environments. We maintain (non-exhaustive) [http://wiki.xiph.org/TheoraSoftwarePlayers lists] of products which implement Theora support, many of them open source, so that others may use them as a reference when preparing their own products.'''
* Is its compatibility with related formal specification documented?
: '''Yes. For example, [http://theora.org/doc/Theora.pdf the Theora specification] also documents the use of Theora within the [http://www.ietf.org/rfc/rfc3533.txt standard Ogg encapsulation format], and the [http://svn.xiph.org/trunk/theora/doc/draft-ietf-avt-rtp-theora-00.txt TheoraRTP draft specification] explains how to transmit Theora using the [http://tools.ietf.org/html/rfc3550 RTP standard]. In addition, the specification documents Theora's compatibility with ITU-R B.470, ITU-R B.601, ITU-R B.709, SMPTE-170M, [http://tools.ietf.org/html/rfc2044 UTF-8], ISO 10646, and [http://www.xiph.org/vorbis/doc/Vorbis_I_spec.pdf Ogg Vorbis].'''

== Part 5: Standardisation Criteria ==
From Idabc-camss

Note: Throughout this section, “Organisation” refers to the standardisation/fora/consortia body in charge of the formal specification.

Significant characteristics of the way the organisation operates are for example the way it gives the possibility to stakeholders to influence the evolution of the formal specification, or which conditions it attaches to the use of the formal specification or its implementation. Moreover, it is important to know how the formal specification is defined, supported, and made available, as well as how interaction with stakeholders is managed by the organisation during these steps. Governance of interoperability testing with other formal specifications is also indicative.

The standardisation criteria analyses therefore the following elements:

=== Availability of Documentation ===
The availability of documentation criteria is linked to cost and online availability. Access to all preliminary results documentation can be online, online for members only, offline, offline, for members only or not available. Access can be free or for a fee (which fee?).
: '''Every Xiph standard is permanently available online to everyone at no cost. For example, we invite everyone to download [http://theora.org/doc/Theora.pdf the most up-to-date copy of the Theora specification], and [http://xiph.org/vorbis/doc/Vorbis_I_spec.html the latest revision of Vorbis]. All previous revisions are available from Xiph's [http://svn.xiph.org/ revision control system].'''

=== Intellectual Property Right ===
The Intellectual Property Rights evaluation criteria relates to the ability for implementers to use the formal specification in products without legal or financial implications. The IPR policy of the organisation is therefore evaluated according to:
* the availability of the IPR or copyright policies of the organisation (available on-line or off-line, or not available);
: '''The reference implementations of each codec include all necessary IPR and copyright licenses for that codec, including all documentation, and are freely available to everyone.'''
* the organisation’s governance to disclose any IPR from any contributor (ex-ante, online, offline, for free for all, for a fee for all, for members only, not available);
: '''Xiph does not require the identification of specific patents that may be required to implement a standard; however, it does require an open-source compatible, royalty free license from a contributor for any such patents they may own before the corresponding technology can be included in a standard. These licenses are made available online, for free, to all parties.'''
* the level of IPR set "mandatory" by the organisation (no patent, royalty free patent, patent and RAND with limited liability , patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none);
: '''All standards, specifications, and software published by the Xiph.Org Foundation are required to have "open-source compatible" IPR. This means that a contribution must either be entirely clear of any known patents, or any patents that read upon the contribution must be available under a transferable, irrevocable public nonassertion agreement to all people everywhere. For example, see [http://svn.xiph.org/trunk/theora/LICENSE our On2 patent nonassertion warrant]. Other common "royalty free" patent licenses are either not transferable, are revocable under certain conditions (such as patent infringement litigation against the originating party), or otherwise impose restrictions that would prevent distribution under common [http://www.opensource.org/ OSI]-approved licenses. These would not be acceptable.'''
* the level of IPR "recommended" by the organisation (no patent, royalty free patent, patent and RAND with limited liability, patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none). [Note: RAND (Reasonable and Non Discriminatory License) is based on a "fairness" concept. Companies agree that if they receive any patents on technologies that become essential to the standard then they agree to allow other groups attempting to implement the standard to use these patents and they agree that the charges for the patents shall be reasonable. "RAND with limited availability" is a version of RAND where the "reasonable charges" have an upper limit.]
: '''Xiph's recommended IPR requirements are the same as our mandatory requirements.'''

=== Accessibility ===

The accessibility evaluation criteria describe the importance of equal and safe accessibility by the users of implementations of formal specifications. This aspect can be related to safety (physical safety and conformance safety) and accessibility of physical impaired people (design for all).

Focus is made particularly on accessibility and conformance safety. Conformance testing is testing to determine whether a system meets some specified formal specification. The result can be results from a test suite. Conformance validation is when the conformance test uniquely qualifies a given implementation as conformant or not. Conformance certification is a process that provides a public and easily visible "stamp of approval" that an implementation of a standard validates as conformant.

The following questions allow an assessment of accessibility and conformance safety:
* Does a mechanism that ensures disability support by a formal specification exist? (Y/N)
: '''Yes. Xiph ensures support for users with disabilities by providing specifications for accessible technologies independent of the codec itself. Notably, the Xiph [http://wiki.xiph.org/OggKate OggKate] codec for time-aligned text and image content provides support for subtitles for internationalisation, captions for the hearing-impaired, and textual audio descriptions for the visually impaired. Further, Ogg supports multiple tracks of audio and video content in one container, such that sign language tracks and audio descriptions can be included in one file. For this to work, Xiph has defined [http://wiki.xiph.org/Ogg_Skeleton Skeleton] which holds metadata about each track encapsulated within a single Ogg file. When Theora is transmitted or stored in an Ogg container, it is automatically compatible with these accessibility measures.'''
* Is conformance governance always part of a standard? (Y/N)
: '''Yes. Xiph's standards always precisely specify the requirements that an implementation must meet in order to be considered conformant.'''
* Is a conformance test offered to implementers? (Y/N)
: '''Yes. Xiph maintains a freely available suite of [http://v2v.cc/~j/theora_testsuite/ test vectors] and an [http://validator.xiph.org online validation service] that can be used by anyone to check confirm basic conformance, in addition to tools such as the oggz-validate program included with liboggz, which has been widely used for conformance testing.'''
* Is conformance validation available to implementers? (Y/N)
: '''Yes. Informal conformance testing is available to implementors upon request, and Xiph has provided such testing for a number of implementations in the past.'''
* Is conformance certification available? (Y/N)
: '''Yes. Xiph does not require certification, but maintains the right to withhold the use of our trademarks from implementors who act in bad faith. Implementors may, however, request explicit permission to use our trademarks with a conforming implementation.'''
* Is localisation of a formal specification possible? (Y/N)
: '''Yes. We welcome anyone who wishes to translate Xiph specifications into other languages. We have no policy requiring that the normative specification be written in English.'''

=== Interoperability governance ===
The interoperability governance evaluation criteria relates to how interoperability is identified and maintained between interoperable formal specifications. In order to do this, the organisation may provide governance for:
* open identification in formal specifications,
: '''Yes. The Xiph codecs can be precisely identified by [http://wiki.xiph.org/index.php/MIMETypesCodecs their MIME types], as formally defined by [http://tools.ietf.org/html/rfc5334 IETF RFC 5334], an open specification.'''
* open negotiation in formal specifications,
: '''Yes. For example, a [http://tools.ietf.org/html/draft-barbato-avt-rtp-theora-01 draft RTP specification] describes how Theora interoperates with the [http://tools.ietf.org/html/rfc3264 Session Description Protocol] (SDP), a mechanism for negotiating the parameters of RTP sessions.'''
* open selection in formal specifications.
: '''Yes. There are many open specifications that provide a mechanism for selecting Theora from among many codecs. One such specification is [http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#video HTML 5 video], which allows the user agent to select Theora based on its MIME type, using [http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#the-source-element the source element].'''

=== Meeting and consultation ===
The meeting and consultation evaluation criteria relates to the process of defining a formal specification. As formal specifications are usually defined by committees, and these committees normally consist of members of the organisation, this criteria studies how to become a member and which are the financial barriers for this, as well as how are non-members able to have an influence on the process of defining the formal specification. It analyses:
* if the organisation is open to all types of companies and organisations and to individuals;
: '''Yes. Xiph welcomes representatives from all companies and organizations.'''
* if the standardisation process may specifically allow participation of members with limited abilities when relevant;
: '''Yes. Standardization occurs almost entirely in internet communications channels, allowing participants with disabilities to engage fully in the standards development process. We also encourage nonexperts and students to assist us as they can, and to learn about Xiph technologies by participating in the standards development process.'''
* if meetings are open to all members;
: '''Xiph meetings are open to everyone. We charge no fee for and place no restrictions on attendance or participation. For example, anyone interested in contributing to the Theora specification may join [http://lists.xiph.org/pipermail/theora-dev/ the Theora development mailing list].'''
* if all can participate in the formal specification creation process;
: '''Yes. All people are welcome to participate in the specification creation process. No dues or fees are required to participate.'''
* if non-members can participate in the formal specification creation process.
: '''Yes. Xiph does not maintain an explicit list of members, and no one is excluded from contributing to specifications as they are developed.'''

=== Consensus ===
Consensus is decision making primarily with regard to the approval of formal specifications and review with interest groups (non-members). The consensus evaluation criterion is evaluated with the following questions:
* Does the organisation have a stated objective of reaching consensus when making decisions on standards?
: '''There is no explicitly stated objective of reaching consensus. However, when new contributions are made, the key specification developers will be able to veto the introduction of a new feature. Generally, differences are discussed openly and new features are adapted until they fit the overall architecture of the standard, at which stage they are introduced into the specification, standard and software.'''
* If consensus is not reached, can the standard be approved? (answers are: cannot be approved but referred back to working group/committee, approved with 75% majority, approved with 66% majority, approved with 51% majority, can be decided by a "director" or similar in the organisation).
: '''The standard can be approved without consensus via the decision of a "director" or similar.'''
* Is there a formal process for external review of standard proposals by interest groups (nonmembers)?
: '''Since anyone may participate in the development process and make proposals, there is no need for a separate formal process to include proposals by nonmembers.'''

=== Due Process ===
The due process evaluation criteria relates to the level of respect of each member of the organisation with regard to its rights. More specifically, it must be assured that if a member believes an error has been made in the process of defining a formal specification, it must be possible to appeal this to an independent, higher instance. The question is therefore: can a member formally appeal or raise objections to a procedure or to a technical specification to an independent, higher instance?

: '''Yes. Even if a member fails an appeal within the organization, because all of the technology Xiph standardizes is open and freely implementable, they are always free to develop their own, competing version. Such competing versions may even still be eligible for standardization under the Xiph umbrella.'''

=== Changes to the formal specification ===
The suggested changes made to a formal specification need to be presented, evaluated and approved in the same way as the formal specification was first defined. This criteria therefore applies the above criteria to the changes made to the formal specification(availability of documentation, Intellectual Property Right, accessibility, interoperability governance, meeting and consultation, consensus, due process).

: '''The exact same process is used for revisions to the standard as was used for the original development of the standard, and thus the answers to all of the above questions remain the same.'''

=== Support ===
It is critical that the organisation takes responsibility for the formal specification throughout its life span. This can be done in several ways such as for example a regular periodic review of the formal specification. The support criteria relates to the level of commitment the organisation has taken to support the formal specification throughout its life:
* does the organisation provide support until removal of the published formal specification from public domain (Including this process?)
: '''Xiph.Org standards are never removed from the public domain. Xiph endeavors to provide support for as long as the standard remains in use.'''
* does the organisation make the formal specification still available even when in non-maintenance mode?
: '''Yes. All Xiph.Org standards are freely licensed and will always be available.'''
* does the organisation add new features and keep the formal specification up-to-date?
: '''Yes. Xiph maintains its ecosystem of standards on a continuous basis.'''
* does the organisation rectify problems identified in initial implementations?
: '''Yes. Xiph maintains [https://trac.xiph.org/report a problem reporting system] that is open to the public, and invites everyone to submit suggestions for improvements. Improvements are made both to the standards documents and to the reference implementations.'''
* does the organisation only create the formal specification?
: '''No. Xiph also produces high-quality reusable reference implementations of its standards, released under an open license.'''

<strong>This is a draft document. A work in progress. A scratchpad for ideas. It should not be widely circulated in this form.</strong>

IDABC Questionnaire 2009

2009-11-04T19:27:55Z

Mindspillage: /* Consensus */ no need for separate process for nonmembers

= Context =
We received [http://lists.xiph.org/pipermail/theora/2009-November/002996.html an e-mail] from a consultant studying the suitability of Theora for use in "eGovernment", on behalf of the [http://ec.europa.eu/idabc/ IDABC], an EU governmental agency responsible for "Interoperability" with an emphasis on open source. The investigation is in the context of [http://ec.europa.eu/idabc/en/document/7728 European Interoperability Framework], about which there has been [http://www.computerworlduk.com/community/blogs/index.cfm?entryid=2620&blogid=14&pn=1 some real controversy].

The method of assessment is the Common Assessment Method for Standards and Specifications, including the questions below.

= CAMSS Questions =
== Part 4: Market Criteria ==

This group of Market criteria analyses the formal specification in the scope of its market environment, and more precisely it examines the implementations of the formal specification and the market players. This implies identifying to which extent the formal specification benefits from market support and wide adoption, what are its level of maturity and its capacity of reusability.

Market support is evaluated through an analysis of how many products implementing the formal specification exist, what their market share is and who their end-users are. The quality and the completeness (in case of partitioning) of the implementations of the formal specification can also be analysed. Availability of existing or planned mechanisms to assess conformity of implementations to the standard or to the specification could also be identified. The existence of at least one reference implementation (i.e.: mentioning a recognized certification process) - and of which one is an open source implementation - can also be relevant to the assessment. Wide adoption can also be assessed across domains (i.e.: public and private sectors), in an open environment, and/or in a similar field (i.e.: best practices).

A formal specification is mature if it has been in use and development for long enough that most of its initial problems have been overcome and its underlying technology is well understood and well defined. Maturity is also assessed by identifying if all aspects of the formal specification are considered as validated by usage, (i.e.: if the formal specification is partitioned), and if the reported issues have been solved and documented.

Reusability of a formal specification is enabled if it includes guidelines for its implementation in a given context. The identification of successful implementations of the standard or specification should focus on good practices in a similar field. Its incompatibility with related standards or specifications should also be taken into account.

The ideas behind the Market Criteria can also be expressed in the form of the following questions:

=== Market support ===
* Does the standard have strong support in the marketplace?
: Yes. For example, among web browsers, support for Xiph's Ogg, Theora, and Vorbis standards is now included by default in Mozilla Firefox, Google Chrome, and the latest versions of Opera, representing many millions of installed users just in this market alone.
* What products exist for this formal specification ?
: Theora is a video codec, and as such the required products are encoders, decoders, and transmission systems. All three types of products are widely available for Theora.
* How many implementations of the formal specification are there?
: Xiph.org does not require implementors to acquire any license before implementing the specification. Therefore, we do not know how many implementations have been created. From voluntary notification given to us by implementors, we know that Theora has at least been implemented in embedded devices, security cameras, video games, video conferencing systems, web browsers, home theater systems, and many other applications.
* Are there products from different suppliers in the market that implement this formal specification ?
: Yes. Corporations such as Novell, Opera, Google, Mozilla, Red Hat, Sun Microsystems, Canonical, DailyMotion, Elphel, and countless others have supplied implementations of the Theora standard.
* Are there many products readily available from a variety of suppliers?
: Yes. Theora is extremely readily available: any user with a modern home computer can have a complete, legal, open source implementation, free of charge, downloaded and running in less than 30 seconds.
* What is the market share of the products implementing the formal specification, versus other implementations of competing formal specifications ?
: Theora playback is extremely widely available, covering virtually the entire market of personal computers. Theora is also increasingly available in mobile and embedded devices.
* Who are the end-users of these products implementing the formal specification?
: The end users are television viewers, movie makers, business people, video distribution services, and anyone else who interacts with moving pictures.

=== Maturity ===
* Are there any existing or planned mechanisms to assess conformity of the implementations of the formal specification?
: Yes. In addition to a continuous peer review process, we maintain a suite of [http://v2v.cc/~j/theora_testsuite/ test vectors] that allow implementors to assess decoder conformity.
* Is there a reference implementation (i.e.: mentioning a recognized certification process)?
: Yes. Xiph.org maintains a constantly updated reference implementation called [http://downloads.xiph.org/releases/theora/ libtheora]. libtheora is maintained not only as a reference, but also to achieve the maximum possible speed, accuracy, reliability, efficiency, and video quality. As a result, many implementors of Theora adopt the reference implementation.
* Is there an open source implementation?
: Yes. libtheora is made available under a completely permissive BSD-like license. Its open-source nature also contributes to its quality as a reference implementation, as implementors are welcome to contribute their improvements to the reference. There are also several other open source implementations.
* Does the formal specification show wide adoption?
** across different domains? (I.e.: public and private)
** in an open environment?
: Yes. On open/free operating systems such as those distributed by Novell/SuSE, Canonical, and Red Hat, Theora is the primary default video codec.
** in a similar field? (i.e.: can best practices be identified?)
* Has the formal specification been in use and development long enough that most of its initial problems have been overcome?
: Yes. Theora has now been used in a wide variety of applications, on the full spectrum of computing devices.
* Is the underlying technology of the standard well-understood? (e.g., a reference model is welldefined, appropriate concepts of the technology are in widespread use, the technology may have been in use for many years, a formal mathematical model is defined, etc.)
* Is the formal specification based upon technology that has not been well-defined and may be relatively new?
: No. The formal specification is based on technology from the On2 VP3 codec, which is substantially similar to simple block-transform codecs like H.261. This class of codecs is extremely well understood, and has been actively in use for over 20 years.
* Has the formal specification been revised? (Yes/No, Nof)
: Yes. The specification is continuously revised to improve clarity and accuracy.
* Is the formal specification under the auspices of an architectural board? (Yes/No)
* Is the formal specification partitioned in its functionality? (Yes/No)
: No. Theora is very deliberately not partitioned, to avoid the confusion created by a "standard" composed of many incompatible "profiles". The Theora standard does not have any optional components. A compliant Theora decoder can correctly process any Theora stream.
** To what extent does each partition participate to its overall functionality? (NN%)
** To what extent is each partition implemented? (NN%) (cf market adoption)

=== Re-usability ===
* Does the formal specification provide guidelines for its implementation in a given organisation?
: Yes. For example, [http://theora.org/doc/Theora.pdf the Theora specification document] provides "non-normative" advice and explanation for implementors of Theora decoders and encoders, including example algorithms for implementing required mathematical transforms. Xiph also maintains [http://wiki.xiph.org/Main_Page a documentation base] for implementors who desire more guidelines beyond the specification itself.
* Can other cases where similar systems implement the formal specification be considered as successful implementations and good practices?
: Xiph.org standards have successfully been implemented by many organisations in a wide variety of environments. We encourage independent implementations and unexpected applications of our standards.
* Is its compatibility with related formal specification documented?
: Yes. For example, [http://theora.org/doc/Theora.pdf the Theora specification] also documents the use of Theora within the [http://www.ietf.org/rfc/rfc3533.txt standard Ogg encapsulation format], and the [http://svn.xiph.org/trunk/theora/doc/draft-ietf-avt-rtp-theora-00.txt TheoraRTP draft specification] explains how to transmit Theora using the [http://tools.ietf.org/html/rfc3550 RTP standard].

== Part 5: Standardisation Criteria ==
From Idabc-camss

Note: Throughout this section, “Organisation” refers to the standardisation/fora/consortia body in charge of the formal specification.

Significant characteristics of the way the organisation operates are for example the way it gives the possibility to stakeholders to influence the evolution of the formal specification, or which conditions it attaches to the use of the formal specification or its implementation. Moreover, it is important to know how the formal specification is defined, supported, and made available, as well as how interaction with stakeholders is managed by the organisation during these steps. Governance of interoperability testing with other formal specifications is also indicative.

The standardisation criteria analyses therefore the following elements:

=== Availability of Documentation ===
The availability of documentation criteria is linked to cost and online availability. Access to all preliminary results documentation can be online, online for members only, offline, offline, for members only or not available. Access can be free or for a fee (which fee?).
: Every Xiph.org standard is permanently available online to everyone at no cost. For example, we invite everyone to download [http://theora.org/doc/Theora.pdf the most up-to-date copy of the Theora specification], and [http://xiph.org/vorbis/doc/Vorbis_I_spec.html the latest revision of Vorbis].

=== Intellectual Property Right ===
The Intellectual Property Rights evaluation criteria relates to the ability for implementers to use the formal specification in products without legal or financial implications. The IPR policy of the organisation is therefore evaluated according to:
* the availability of the IPR or copyright policies of the organisation (available on-line or off-line, or not available);
* the organisation’s governance to disclose any IPR from any contributor (ex-ante, online, offline, for free for all, for a fee for all, for members only, not available);
* the level of IPR set "mandatory" by the organisation (no patent, royalty free patent, patent and RAND with limited liability , patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none);
: Xiph.org standards have a mandatory requirement that our standards be absolutely free of IPR encumbrance. This means that a standard must either be entirely clear of any known patents, or any patents that read upon the standard must be available under an irrevocable public nonassertion agreement to all people everywhere. For example, see [http://svn.xiph.org/trunk/theora/LICENSE our On2 patent nonassertion warrant].
* the level of IPR "recommended" by the organisation (no patent, royalty free patent, patent and RAND with limited liability, patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none). [Note: RAND (Reasonable and Non Discriminatory License) is based on a "fairness" concept. Companies agree that if they receive any patents on technologies that become essential to the standard then they agree to allow other groups attempting to implement the standard to use these patents and they agree that the charges for the patents shall be reasonable. "RAND with limited availability" is a version of RAND where the "reasonable charges" have an upper limit.]
: Xiph's mandatory IPR requirements are the most stringent possible, and so our IPR recommendation is necessarily the same as the mandatory requirements.

=== Accessibility ===

The accessibility evaluation criteria describe the importance of equal and safe accessibility by the users of implementations of formal specifications. This aspect can be related to safety (physical safety and conformance safety) and accessibility of physical impaired people (design for all).

Focus is made particularly on accessibility and conformance safety. Conformance testing is testing to determine whether a system meets some specified formal specification. The result can be results from a test suite. Conformance validation is when the conformance test uniquely qualifies a given implementation as conformant or not. Conformance certification is a process that provides a public and easily visible "stamp of approval" that an implementation of a standard validates as conformant.

The following questions allow an assessment of accessibility and conformance safety:
* Does a mechanism that ensures disability support by a formal specification exist? (Y/N)
: Yes. Xiph.org ensures disability supporting by providing specifications for accessible technologies independent of the codec itself. Notable Xiph specifications include [http://wiki.xiph.org/OggKate OggKate] and [http://wiki.xiph.org/index.php/CMML CMML], which provide subtitles for the hearing-impaired, as well as [http://wiki.xiph.org/Ogg_Skeleton Skeleton], which provides metadata that can be used to specify scene description audio tracks for the visually impaired. When Theora is transmitted or stored in an Ogg container, it is automatically compatible with these accessibility measures.
* Is conformance governance always part of a standard? (Y/N)
* Is a conformance test offered to implementers? (Y/N)
* Is conformance validation available to implementers? (Y/N)
* Is conformance certification available? (Y/N)
* Is localisation of a formal specification possible? (Y/N)
: Yes. We welcome anyone who wishes to translate Xiph specifications into other languages. We have no policy requiring that the normative specification be written in English.

=== Interoperability governance ===
The interoperability governance evaluation criteria relates to how interoperability is identified and maintained between interoperable formal specifications. In order to do this, the organisation may provide governance for:
* open identification in formal specifications,
* open negotiation in formal specifications,
* open selection in formal specifications.

=== Meeting and consultation ===
The meeting and consultation evaluation criteria relates to the process of defining a formal specification. As formal specifications are usually defined by committees, and these committees normally consist of members of the organisation, this criteria studies how to become a member and which are the financial barriers for this, as well as how are non-members able to have an influence on the process of defining the formal specification. It analyses:
* if the organisation is open to all types of companies and organisations and to individuals;
: Yes. Xiph welcomes representatives from all companies and organizations.
* if the standardisation process may specifically allow participation of members with limited abilities when relevant;
: Yes. Standardization occurs almost entirely in internet communications channels, allowing participants with disabilities to engage fully in the standards development process. We also encourage nonexperts and students to assist us as they can, and to learn about Xiph technologies by participating in the standards development process.
* if meetings are open to all members;
: Xiph meetings are open to everyone. We place no restrictions on attendance or participation. For example, anyone interesting in contributing to the Theora specification may join [http://lists.xiph.org/pipermail/theora-dev/ the Theora development mailing list].
* if all can participate in the formal specification creation process;
: Yes. All people are welcome to participate in the specification creation process. No dues or fees are required to participate
* if non-members can participate in the formal specification creation process.
: Yes. Xiph does not maintain an explicit list of members, and no one is excluded from contributing to specifications as they are developed.

=== Consensus ===
Consensus is decision making primarily with regard to the approval of formal specifications and review with interest groups (non-members). The consensus evaluation criterion is evaluated with the following questions:
* Does the organisation have a stated objective of reaching consensus when making decisions on standards?
* If consensus is not reached, can the standard be approved? (answers are: cannot be approved but referred back to working group/committee, approved with 75% majority, approved with 66% majority, approved with 51% majority, can be decided by a "director" or similar in the organisation).
* Is there a formal process for external review of standard proposals by interest groups (nonmembers)?
** Since anyone may participate in the development process and make proposals, there is no need for a separate formal process to include proposals by nonmembers.

=== Due Process ===
The due process evaluation criteria relates to the level of respect of each member of the organisation with regard to its rights. More specifically, it must be assured that if a member believes an error has been made in the process of defining a formal specification, it must be possible to appeal this to an independent, higher instance. The question is therefore: can a member formally appeal or raise objections to a procedure or to a technical specification to an independent, higher instance?

=== Changes to the formal specification ===
The suggested changes made to a formal specification need to be presented, evaluated and approved in the same way as the formal specification was first defined. This criteria therefore applies the above criteria to the changes made to the formal specification(availability of documentation, Intellectual Property Right, accessibility, interoperability governance, meeting and consultation, consensus, due process).

=== Support ===
It is critical that the organisation takes responsibility for the formal specification throughout its life span. This can be done in several ways such as for example a regular periodic review of the formal specification. The support criteria relates to the level of commitment the organisation has taken to support the formal specification throughout its life:
* does the organisation provide support until removal of the published formal specification from public domain (Including this process?
: Xiph.org standards are never removed from the public domain. Xiph endeavors to provide support for as long as the standard remains in use.
* does the organisation make the formal specification still available even when in non-maintenance mode?
: Yes. All Xiph.org standards are freely licensed and will always be available.
* does the organisation add new features and keep the formal specification up-to-date?
: Yes. Xiph.org maintains its ecosystem of standards on a continuous basis.
* does the organisation rectify problems identified in initial implementations?
: Yes. Xiph.org maintains [https://trac.xiph.org/report a problem reporting system] that is open to the public, and invites everyone to submit suggestions for improvements. Improvements are made both to the standards documents and to the reference implementations.
* does the organisation only create the formal specification?
: No. Xiph.org also produces high-quality reusable reference implementations of its standards, released under an open license.