Zero to Photon: Image Pipeline

Zero to Photon:

Image Pipeline

2024 . 5 . 18

Viewing Photon's raw image data would depict a monochrome image with a stark checkerboard pattern, so it's Photon Transfer's job to convert the raw image data to an acceptable color image:

Photon Transfer image processing in a nutshell

Note the checkerboard pattern visible in the RAW image, caused by the Bayer pattern of the image sensor

Image Pipeline Overview

The image pipeline contains the following stages:

The stages of Photon Transfer's image pipeline

Left column: the automatic stages that aren't configurable
Right column: the user-controlled "artistic" stages

The input to the image pipeline is raw image data. Viewed directly, the raw image data appears as a 2304 x 1296 pixel monochome image that reflects the bayer pattern of the image sensor, where each pixel is sensitive to a single primary color (red, green, or blue).

Photon Transfer's image pipeline is implemented in C++ and Metal shaders so that the heavy lifting can be performed efficiently on the GPU. All of the source is available on GitHub; you can follow along with this blog post by looking at ImagePipeline.mm.

Illuminant Estimation

Source on GitHub

Since the image pipeline doesn't know the light source that's illuminating a scene, it's the job of the first stage to estimate it so that later stages (specifically, highlight reconstruction and white balancing) can use this information.

There are many techniques to perform illuminant estimation and it's an active topic of academic research. Photon Transfer uses Fast Fourier Color Constancy (FFCC) for illuminant estimation, which is fast (~2.6ms per image) and delivers good-quality output.

Highlight Reconstruction

When capturing a photo there are typically regions that are over-exposed — highlights — which causes one or more of the RGB channels to clip to the maximum pixel value. (Photon uses 12-bit pixels, so highlights are clipped to 2¹²-1 = 4095.)

Without explicit processing for these clipped pixels, chromatic distortions occur in the final output image. The green channel is most prone to clipping, so the distortions typically appear as magenta regions:

A clipped green channel results in magenta regions in the final image, because information was lost from the green channel. In other words, there's less green in the output image than there should be, which results in a magenta hue.

The goal of the Highlight Reconstruction stage is to reconstruct the information that was lost due to one or more of the channels "hitting the rails", restoring the pixel value to what it would have been had it not been clipped:

The three channels of a highlight pixel before and after highlight reconstruction

Photon Transfer implements a rudimetary highlight reconstruction algorithm that replaces highlights with the color of the illuminant (determined in the Illuminant Estimation stage). It performs decently in some situations, but fails in others:

With the current implementation of highlight reconstruction, the car benefits, but the sky does not.

Improving highlight reconstruction is on the todo list!

Debayer

Source on GitHub

The Debayer stage is responsible for converting the single-channel RAW image to a 3-channel RGB image:

The easiest way to implement debayering is to translate each 2x2 group of input RAW pixels to a single 1x1 output RGB pixel (where the two green pixels are averaged). The limitation of this approach is that the output image has half the width/height of the input image, so this technique typically isn't used in practice.

Debayering is an active topic of academic research, and Photon Transfer uses Linear Minimum Mean Square-Error Estimation (LMMSE) to perform debayering.

White Balance

White balancing adjusts the colors of the image so that neutral colors actually appear neural, effectively removing the green tint of the debayered image:

The result of white-balancing

Note how the sidewalk is transformed from green-tinted to gray.

White balancing works by simply dividing every RGB pixel of the image by the RGB illuminant color (which we know from the Illuminant Estimation stage):

This works because objects in the real-life scene that have a neutral color will (by definition) assume the illuminant color in the RAW image. Therefore dividing every pixel by the illuminant color will set these objects to a RGB value with equal component values, ie a neutral color.

Color Correction

The final stage of the "automatic" image pipeline is color correction:

The result of color-correction

This stage applies a linear transformation to the RGB pixels to correct the image sensor's perception of colors to match a ground-truth color-calibration chart:

A color-calibration chart used to compute the color-correction matrix

The circles on the color-calibration chart are placed manually in order the compute the color-correction matrix shown in the panel on the right.

This linear transformation involves multiplying every pixel by a 3x3 color-correction matrix (CCM):

We can compute the CCM using a photo of the color-calibration chart and a dusting of linear algebra. Using the standard matrix equation Ax=b, we'll set x to the image sensor's perceived color of the 24 calibration squares, and we'll set b to the ground truth color of the 24 calibration squares. In other words, x is what we have, b is what we want, and A is the transformation that'll get us there:

Now we just need to solve for the CCM A. We'll rearrange the equation into the standard form Ax=b:

Now that the equation is in the standard form Ax=b with respect to what we want to solve for (ie we want A^T which is in the position of x in the standard equation), we can do a least-squares regression, transpose the result, and we have A!

White Balance Stability

An important property of the CCM is that it doesn't disturb the white balance of the image that was established in the previous stage. In other words, a neutral color multiplied by the CCM must remain a neutral color.

To ensure that the CCM has this property, we'll perform one last step: we'll divide each row of the CCM by the sum of its three components. This ensures that each row of the CCM sums to 1, and therefore a neutral color multiplied by the CCM will remain a neutral color.

User-Controlled Stages

The remaining stages of the image pipeline allow for manual image adjustment to suit the user's preferences:

Image adjustment panel in Photon Transfer

Performing these image processing stages requires transforming the color data between different color spaces and white points (specifically ProPhotoRGB.D50, XYZ.D50, XYZ.D65, XYY.D50, Lab.D50, and LSRGB.D65). Other than that, these stages are pretty standard image processing blocks and don't warrant too much discussion.