Open Source Video Processing Wiki

Introduction.    View      

This is a high level overview of low-cost open-source video processing pipelines. It is intended primarily for those who are new to the field, but in this fast evolving field even experts in this area are likely to learn something new.

An image processing pipeline involves a lens, a sensor, analog to digital conversion, processes applied to the data, transmission and display. The parts can be aggregated into a camera, a microcontroller, an ASIC, or an FPGA all communicating with each other using some protocols. Commercially viable image processing pipelines can include both open and closed source RTL, software, hardware and protocols.

There are so many options, it is hard to know what to choose. This article reviews the options available, paying particular attention to the open-source less-expensive alternatives. There is also an emphasis on European supplied components. Desktop and cell phone applications will not be covered.

Throughout the wiki, particular attention is paid to cost and supply chains, providing practical advice on how to choose cameras and FPGAs. The goal for this brief article, on a large topic, is not to cover everything, but to map the landscape and point you towards the most important resources to learn more.

Applications.    View      

Here are some of the applicaitons which could be built using video processing pipelines.

Open Source FPGAs are great for building innovative applications which do not already exist and for reducing the cost of existing applications.

Hardware.    View      

Video processing on Microcontrollers, Asics and FPGAs. This also covers protocol bridges.

Communication Protocols.    View      

This area covers communication protocols used by image sensors, display protocols, and longer range protocols. Bridge chips, which convert between protocols are also covered.

Image Processing Blocks.    View      

Here are common image processing functions which can implemented in image sensors, ASICs and on FPGAs.

Processes for desktop and cell phone are not covered. The processes are generally ordered by their position in the pipeline. Those earlier in the pipeline are listed first.

Recommendations.    View      

General advice on which technology to use when doing image processing. It depends strongly on your bandwidth requirements.

If an ASIC or application exists which does what you need, use it.

For applications with low bandwidth, and small computational loads, a mass produced microcontroller is the cheapest solution. Libmpix running on the Zephyr operating system is a good choice. Once they get vector processing integrated it will be a better choice.

The only Open Source FPGAs with MIPI are the Lattice NX17/40 chips.

If Lattice NX17/40 is not an option, but you want to use Open Source FPGAs, and you need less than 1.66 Mpixels, use a Digital Video Port camera.

If a DVP camera does not meet your needs, consider using a microcontroller as a bridge with MIPI-CSI in and a parallel DPI or RGB port out. Examples include Raspberry Pi Zero Bridge,  and maybe ESP-p4, CH32H417 and Stm-32 N6 bridges. .  You will need CDC circuits, available as Open Source Verilog, or provided in hard core by GateMate.  You will be limited to 1.66 MPixels.

For applications over 1.66 MegaPixels, a closed source FPGA with MIPI is needed.

To easily connect an Open Source FPGA to USB, use EZ-USB FX3 with single-ended wires. FX5 also supports LVDS which can run faster.

For a lower cost USB display, use DVI output to a DVI→ USB Dongle.

For even lower unit cost USB connections, use a microcontroller as a bridge, but the engineering costs will be vastly higher. H.264 or M-JPEG will improve the video performance.

For bandwidths up to 5 Gbits/s and cable lengths up to 15 meters use the [MIPI A-Phy standard(/mipi-a-phy). While MIPI C-Phy is proprietary, the A-Phy standard is more open. A rapidly increasing number of products should be available.

For bandwidths up to 5Gbits/s, cable lengths up to 15 meters, and simpler engineering, use one of the protocols not designed by a committee. FPD-link or GMSL. GMSL even allows the control signals to pass via a different side channel.

For displays, the Gowin 1K and GateMate support 640 * 480 DVI @60 FPS. The Lattice ECP5 supports 1920x1080 DVI @ 30FPS. Gowin 20K supports (3840 × 2160s) DVI. FGPAs can drive TFT LCD screens. Otherwise one of the above mentioned microcontrollers can be used to drive DVI.

Military applications should also consider the supply chains used in producing their video processing pipeines.