Contents

1 Spectral decorrelation

Most image and video compressors exploit the statistical (and also perceptual¹) correlation between the \(\text {RGB}\) color components² of the pixels, using a color transform.

Color transforms are pixel-wise operators. As a result, each pixel is represented in a different domain where (usually) three new coefficients³ express the same⁴ information but in a different color domain.

2 Luma and chromas

Most color transforms are designed to split the color information of a pixel into luminance (luma) and chrominance (chroma). The luma is basically the low frequency⁵ information of the pixel, and the chroma (logically) high frequency information.

For example, in JPEG and H.264/AVC the color information of each pixel is transformed from the \(\text {RGB}\) color space to the \(\text {YCrCb}\) color space, and in JPEG XR, the color space \(\text {YCoCg}\)is used. In these luma-based color spaces, \(\text {Y}\) represents the luma (coefficient) of the pixel. The other two coefficients form the chroma. 1Note that the chrominance of a pixel is determined by two chromas.

Channels and subbands (a word about notation)

Apart from using the terms of component and coefficient, we will use the word channel to refer to the same index component of all the pixels of an image (or video), and subband to denote the same index coefficient generated after the transformation of all pixels of an image (or video). For example, the \(\mathbf {R}\) channel of a color image corresponds to the monochromatic image⁶ generated by the \(\text {R}\) component of all pixels of the image, and, for example, the \(\mathbf {Y}\) subband of a transformed (\(\text {RGB}\)) image corresponds to the \(\text {Y}\) coefficients of all pixels (also a “monochromatic” image).

In image and video coding, most color transforms map 3 channels (\(\text {RGB}\)) into 3 subbands.

3 Benefits of color transforms

Color transforms applied to natural visual information generally have two key advantages:

1. Energy concentration: In general, transforms “move” the energy⁷ between subbands, accumulating most of the energy in a reduced number of them (aspect related to the so-called coding gain of the transform [5]). In our case, where the transformations are between color spaces, in the transform domain most of the energy is concentrated in the \(\text {Y}\) subband. As a consequence of this, usually, the entropy [2] is decreased and the dynamic range of the signal is increased. The first fact means that we will compress more⁸, and the second that we will be able to use a higher range of quantization step sizes [3, 7], increasing also the number of feasible points in the RD curve [2].
2. Luma/croma analysis: Our visual system is more sensitive in terms of spatial resolution to the luma (“black-and-white”) information than to the chroma (“color”) information, which basically means that we can quantize more the chroma⁹ without generating perceptual distortion.

4 Scalar quantization in the color transform domain

If the color transform is orthogonal or biorthogonal, that is, the luma and the cromas are independent, the quantization noise generated in the subbands is additive [1]. Therefore, from a pure RD point of view, the quantization step sizes for each subband should be selected using the same RD slope in all subbands (see the notebook Scalar Quantization of RGB images. Notice that this implies to compute the RD curves.

Therefore, taking a generic luma / croma transform \(\text {YUV}\), we would expect that \begin {equation} \lambda ^{\text {Y}} \approx \lambda ^{\text {U}} \approx \lambda ^{\text {V}} \label {eq:optimal_lambda} \end {equation} for a given quantization step size \(\Delta \), the RDO [2] can be ignored. In the notebook Scalar Quantization of RGB images we can explore (at least visually) the grade of compilance of Eq. \eqref{eq:optimal_lambda}.

Resources

See the notebooks Removing RGB redundancy with the DCT, Removing RGB redundancy with the \(\text {YCoCg}\) transform and Removing RGB redundancy with the \(\text {YCrCb}\) transform.

5 Vector quantization applied to the \(\text {RGB}\) domain

SQ (Scalar Quantization) [3, 7] would be an optimal solution only if the image colors are uniformly distributed within the RGB cube. However, the typical color distribution in natural images is anything but uniform, with some regions of the color space being densely populated and many potentially used colors completely missing. In this case, depending on the quantization step size [4], SQ could be suboptimal because the colors used may not be sampled with sufficient density, while at the same time the encoding system considers colors that do not appear in the image at all [1].

On the other hand, VQ (Vector Quantization) [6, 7] applied to the color domain does not treat the individual \(\text {RGB}\) components separately as does scalar quantization, but each color vector used \({\mathbf C}_i = (\text {R}_i, \text {G}_i, \text {B}_i )\) in the image is treated as a minimum structure. VQ determines a code-book of \(K\) code-vectors (centroids) that minimizes the distortion between the original image and the reconstructed one. Notice that the code-book must be known by the decoder to find a reconstruction.

Resources

See the notebooks Vector Quantization (in the color domain) of a RGB image and Vector Quantization (in the 2D domain) of a color (RGB) image.

6 To-Do

1. Modify VCF to allow the use of the (color) \(\text {DCT}\) in the compression pipeline. Complexity 1.
2. Modify VCF to allow the use of the \(\text {YCrCb}\) transform in the compression pipeline. Complexity 1.
3. Modify VCF to allow the use of VQ (Vector Quantization) (applied to the color domain) in the compression pipeline. Complexity 4.

7 References