Image Processing Blocks

Processes for desktop and cell phone are not covered. The processes are generally ordered by their position in the pipeline. Those earlier in the pipeline are listed first.

Defective Pixels Correction corrects for pixels which are either short circuits (white), and open circuits (black). The Himax HM1246 camera “identifies and corrects hot (white) and cold (black) pixels by comparing each pixel to neighboring same color pixels.”
Video Cropping works much like photo cropping. It is a bit more complex, because the process has to be synchronized with the Vertical (VSync) and horizontal (HSync) synchronization ignals.

Black Level Calibration compensates for image sensors output voltage which have a non-zero offset when no light is shining on them. Many cameras have a blackened area to measure and deduct the average offset value from each measurement's nominal value.

Lense Shade Correction fixes visible hue changes caused by the lens and its housing at the edges of the sensor array.

Exposure Controls sets the length of time for which charge is accumulated on the sensor, based on the lighting condition.

Binning averages input measurements prior to debayering. The OV5640 supports 1x2, 2x1 and 2x2 binning.

Debayering converts a grid of red, green and blue measurements into an image, where each pixel has red green and blue components.  Most cameras have a single sensor backplane, with a grid of color filters in front of it. This original Bayer pattern is half green, one quarter red and one quarter blue. Debayering converts those measurements into one quarter as many pixels, each with 3 colors.

Lens correction compensates for errors in the lens manufacturing process.  Lenses have a designed shape, but the actual manufactured shape may differ from the designed shape.

White balance ensures that a white wall will look white.  The frequency response of pixels in a camera, or on a display do not match the frequency response of the human eye,   White balance rotates the RGB vector using a matrix multiply, so that the colors are more accurate. Some colors are more important than others. While an inaccurate shade of green on a tree may be acceptable, an incorrect skin tone is immediately noticeable and undesirable, making accurate white balance critical for applications involving people.

50/60 Hz Luminance Detection prevents bands in the image caused by fluorescent lights flickering at 50 or 60 hz.   This happens because most commercial cameras use a rolling shutter. Wikipedia says that: “rolling shutter is a process of image capture in which … each frame of a video (in a video camera) is captured not by taking a snapshot of the entire scene at a single instant in time but rather by scanning across the scene rapidly.” So the lines recorded during the dark part of the fluorescent flicker will look different from the lines recorded during the bright part of the flicker. OV5640 and other cameras detect and correct for this effect.

Color Correction compensates for the fact that the red, green and blue pixels all respond a little to all of 3 colors. This step in the image pipeline multiplies the debayered pixels by a 3x3 matrix to extract the true colors.

Gamma Correction compensates for human’s logarithmic perception. We can only detect a 10% change in light amplitude. If the brightness is on a scale of 0-127, we can just barely distinguish between 127 and (127 -12.7= 114). All those numbers between 114 and 127 are wasted. Cameras and display devices may also display similar non-linearities. Gamma correction attempts to compensate for all of this. The correction is typically done with a power-law function (e.g., V_out = V_in^γ). In the process it also compresses the data. One could achieve even greater data compressions using lookup tables. The Himax documentation details how they implemented this using a piecewise linear function.

Color Space Conversion usually transforms the image data between RGB and YCrCb (1 luminance channel and 2 chrominance channels), which is an affine transform. The human eye is less sensitive to color, than brightness, so low order bits are often dropped from the YCrCb representation, saving space.

Denoise reduces the noise by averaging pixels. Denoise can happen in the RGP color space or on Himax cameras, in the YCrCb color space.  It is also possible to do denoise across multiple frames.

Alpha Blending Mixtures display multiple overlaid images. The transparency of the images in front can be dynamically adjusted to achieve the desired effect.

JPEG uses a discrete cosine transform (DCT) on 8x8 pixel blocks to compress the image. The Brilliant Lab’s glasses Open Source Verilog implements this to shrink still photographs to a size which can rapidly be transmitted over Bluetooth. M-JPEG  uses JPEG to compress each still image separately.

H.264 does compression across multiple still images. H.264 compression is more efficient than M-JPEG, but increases lag time, a key issue for drones.

Edge Detection finds “a line or border at which a surface terminates”. For the purpose of edge detection, Wikipedia defines edges as “curves in a digital image at which the image brightness changes sharply or more formally has discontinues.” While common algorithms like Canny and Sobel only use brightness (luminance), on FPGAs more computationally intensive algorithms could use color to better identify material and surface changes.

Frame buffers store the image for future display. Typically an image pipeline will have two Frame buffers. One is being written to, while the other is being displayed.

Encryption protects privacy and ensures security. Advanced Encryption Standard (AES) is a popular encryptions algorithm.   For securing data in transit over networks, protocols like TLS are used on top of TCP/IP, while WiFi uses WPA2/WPA3 for link security.

Computer Control of the image processing pipeline includes both high and low level controls.   The many complex registers need to be configured correctly to achieve the user’s simple objectives. Libcamera hides the complexity, making cameras easy to use. While processing video, the 'auto' in auto-white-balance, auto-focus and auto-exposure need to be executed, usually by the camera chip itself.

Display. The last stage in a video processing pipeline is often to display the video to the user.