2.4. Quantization#

If you recall the introduction to this chapter, we saw that digitized audio has two properties: the sampling rate (already covered in this chapter), and the precision. This section concerns the precision of digital audio, but what exactly does this mean? To understand this, we’ll first need to take a detour to see how computers represent numerical data.

2.4.1. Background: digital computers and integers#

These days, most humans use the Hindu-Arabic (or decimal) numeral system to represent numbers. With decimal digits, we use the ten symbols \(0,1,2,\cdots,9\) to encode numbers as combinations of powers of ten (the base or radix of the system). For example, a number like \(132\) can be expanded out in terms of powers of 10:

\[ 132= \red{1} \cdot 10^2 + \red{3} \cdot 10^1 + \red{2} \cdot 10^0. \]

There’s nothing magical about the number 10 here: it was probably chosen to match up with the number of fingers (ahem, digits) most people possess. Any other base can work too.

Of course, computers don’t have fingers, so they might find decimal to be difficult. Computers do have logic gates though, which can represent true and false values, which we can interpret as \(1\) and \(0\) respectively. This leads us to binary numbers, which only use two symbols to encode numbers as combinations of powers of 2, rather than combinations of powers of 10.

In our example above, the number \(132\) could be represented as

\[\begin{align*} 132 &= \red{1} \cdot 128 + \red{0} \cdot 64 + \red{0} \cdot 32 + \red{0} \cdot 16 + \red{0} \cdot 8 + \red{1} \cdot 4 + \red{0} \cdot 2 + \red{0} \cdot 1\\ &= \red{1} \cdot 2^7 + \red{0} \cdot 2^6 + \red{0} \cdot 2^5 + \red{0} \cdot 2^4 + \red{0} \cdot 2^3 + \red{1} \cdot 2^2 + \red{0} \cdot 2^1 + \red{0} \cdot 2^0, \end{align*}\]

or, more compactly, as \(10000100_2\) (where the subscript lets us know we’re in binary). We refer to each position as a bit (short for binary digit).

For various technical reasons, computers don’t generally support arbitrarily large numbers. Instead, integers come in a few different “sizes” depending on how many bits we’ll need: usually, 8, 16, 32, or 64 bits. The example above is an 8-bit number, but it could just as easily have been written in 16-, 32- or 64-bit representation by using leading zeros: \(0000000010000100_2\) for 132 in 16-bit form.

2.4.1.1. Negative numbers#

An \(n\)-bit number can represent \(2^n\) distinct numbers, but which numbers? We can interpret the bit representation as the numbers \(0, 1, \cdots, 2^n-1\), but this doesn’t provide an obvious way to represent negative numbers.

There’s an elegant solution to this problem if we imagine arranging binary numbers around a circle, as illustrated below for 3-bit numbers. We can think of counter-clockwise movement as incrementing by one, and clockwise movement as decrementing by one. In this view, the numbers beginning with 1 can be seen as negative numbers: \(111 = -1, 110 = -2, \dots\), and the numbers beginning with 0 are the non-negative numbers as discussed above. It’s beyond our scope here, but this representation of integers, known as two’s complement, has many nice properties, and is implemented by almost every modern computer for doing integer arithmetic.

Visualization of two's complement for 3-bit integers

Fig. 2.6 Two’s complement representation of 3-bit integers (-4, -3, …, 3).#

To summarize, an \(n\)-bit two’s-complement integer can represent \(2^{n-1}\) distinct non-negative numbers (\(0, 1, \dots, 2^{n-1}-1\)), and \(2^{n-1}\) distinct negative numbers (\(-1, -2, \dots, -2^{n-1}\)). For example, an 8-bit number can take \(2^8 = 256\) distinct values: \(-128, -127, \dots, -2, -1, 0, 1, 2, \dots, 127\).

This is a relatively minor detail in the bigger picture of digital signals, but it can help to understand why quantized signals look the way they do (as illustrated below).

2.4.2. Defining precision and quantization#

Precision, also known as bit depth, refers to how many bits are used to represent each sample in a digital signal. While we typically think of signals as taking on continuous real values, computers quantize these values to be drawn from a fixed, finite set of numbers.

High precision means that we have more distinct values, and can therefore faithfully represent smaller differences and get a more accurate representation of the underlying continuous signal. However, doing so comes at a cost: higher precision means we’re storing and transmitting more data. There’s a trade-off to be made between storage cost and the perceptual fidelity of the quantized signal. Investigating this thoroughly is beyond the scope of this text, but interested readers are encouraged to look into perceptual coding and lossy audio compression to learn more.

2.4.3. The effects of quantization#

Fig. 2.7 illustrates the effects of varying levels of quantization on samples from a continuous waveform. High precision values (16-bit) provide a good approximation to the original wave, but this approximation deteriorates as we reduce the precision. In the extreme case (1-bit quantization), each sample can take only one of two values (-1 or 0), which results in a highly distorted signal.