Other Applications

Edge Detectors.    View      

Canny and Sobel edge detection algorithms are designed for microcontrollers, a different approach is needed for FPGAs.

Canny and Sobel edge detectors use 3x3 matrices , and assume that the middle row or column is the edge. Dividing by 3 or 9 is hard on an FPGA. Much better to use 2x2 or 4x4 or 8x8 edge detectors, and then one can average over multiple pixels, and divide by 2, 4 or 8. One could even use all of them at the same time on an FPGA, and thus detect both fine edges, and thick noisy edges.

There is a class of vision impaired people who would benefit from good edge detection algorithms. Thick black lines on their glasses would help them navigate through doors, and follow the sidewalk. The Canny and other edge detector algorithms detect fine lines, but are not so good at recognizing the major features needed to support this functionality. FPGAs can use more computationally intensive algorithms to accomplish this task.

Stereoscopic vision currently uses computationally expensive disparity maps. Edge detection followed by stereoscopic analysis may reduce computational complexity, and run on less expensive FPGAs.

NeTV2.    View      

An open video development board in a PCI express form factor that supports overlaying content on encrypted video signals. It can alw

The NeTV2 is an FPGA-based development board optimized for open digital video applications, and just like any other development board

Classic mode allows the NeTV2 to add encrypted pixels to an encrypted video stream, without ever decrypting the video stream. It does this by observing the initial cryptographic handshake between the the video source and video sink. So, for example, using NeTV Classic Mode, you could add an opaque text overlay to a live video stream, but you could not add a translucent text overlay, as that would require decrypting the original video stream to compute the alpha blending.

Libre mode works only with unencrypted video feeds, but has full access to the entire video stream. This lets you arbitrarily manipulate pixels in real time, either onboard or by plugging into a host using the PCI-express 2.0 x4 interface for offboarding compute to a companion GPU or cloud service.

NuEyes NuLoups.    View      

A beautiful example of what can be done with FPGA based video processing pipelines. I know the people who wrote the Verilog. Sadly not open source.

Engineered for dental and medical use and designed in conjunction with health care experts, NuLoupes are built for real world use cases. NuLoupes brings surgical loupes into the 21st century with live 3D stereoscopic imaging for depth perception, continuous magnification from 1x to 13x, all in a light weight ergonomic design for comfort and all day use. Launching in late 2025.

Teaching Verilog.    View      

Video Pipelines may be the best way to teach Verilog.

It is fun to see things move on the screen. More importantly debugging FPGAs is often difficult, but if you can see the generated image, the bug is often quite obvious. If the refresh rate is too fast, you can use the $5 DVI to USB Dongle to capture the video stream on the desktop, record it, play it back slowly, zoom in on the pixels, and do calculations in Python on the image to ensure that it is correct.

The first thing that you need is a graphical Verilog simulator to debug the Verilog designs before synthesis. Then you need boards. You could use the low cost Gowin boards, but I think you are better off using the IcePi Zero board; it has access to the whole Raspberry Pi ecosystem.

Once you have the display working, all kinds of graphics can be generated. Moving squares, swirls, flashing lights and anything else you can image. That is lots of fun.

The. low end FPGA boards do not support MIPI, so the next step is to connect a camera using DVP, or you can use a $10 -$15 Raspberry Pi zero to bridge from propritary mipi to DVP.

Once the basic camera -> DVI pipeline is working, there are lots of possible simple projects. Input two cameras, and switch between them. Display on two monitors, and switch between them. Split the two cameras on the display. Fade from one camera to the other. Generate a grey scale image. Change the colors, like you are on drugs. Shake the image, like it is in an earthquake. Memory permitting you may be able to tilt the image, particularly if it is a grey scale image. Distort the video. Here are a list of more traditional video processes which can be applied to video.

Tiny Videos.    View      

TimVideos.us is a group of exciting projects which together create a system for doing both recording and live event streaming for conferences, meetings, user groups and other presentations.

https://code.timvideos.us/

ECPi Camera.    View      

The open source FPGAs with MIPI CSi hard cores are back ordered for 2 years. A $10 Raspberry Pi Zero can read from a CSI camera and drive a 24 bit Display Parallel Interface (DPI) output. The goal of this project was to feed that DPI into an ECP5 FPGA. Sadly it is not so simple.

There are several problems with this approach.

What follows is the description of this open source project.

ECPi camera lets you connect any Raspberry Pi 2 or 3 camera to an ECP5 FPGA and display it on DVI or USB. You can control the camera using the RPI Zero. Parts are available.

ECPi Camera uses a Raspberry Pi Zero and a Rasperry Pi Camera to drive an IcePi Zero using a Lattice ECP5 FPGA. You can use either the $10 Raspberry Pi Zero (one core) or the $15 Raspberry Pi Zero 2W (four cores, Bluetooth and WiFi). This is an entirely open source tool chain, athough obviously the chips themselves are all proprietary.

Assembly

You need a Raspberry Pi Zero, an IcePi Zero available from CrowdSupply or Electrow, any Raspberry Pi Camera version 2 or 3, a camera cable and a male and a female header. The camera cable should convert between the standard 24 pin camera flexible printed cable socket and the Raspberry Pi Zero's 15 pin FPC socket. The standard header configuration is male headers on the RPI Zero facing up, and female headers on the IcePi Zero facing down. You can solder them, or if you are not comfortable with your soldering skills, what I am using is the Pimoni Hammer headers. Here is a blog posing about how to do it, scroll all the way down for how to do the female hat header.

If you are concerned about using hammer headers, here is an article reviewing the product. Basically it and the comments say that it works fine, and has some plusses and minuses. "Solder is not glue". In the automotive industry the fittings are press fit, solder does not survive temperature cycling. But the holes are a particular size, and the gold platting is a little thicker. Pimoroni and the article tested the electrical characteristics and all is fine. Be careful to apply equal pressure everywhere and to not bend the board. A vice woudl be the best way to do it.

FirmWare

The camera video can be displayed on the screen and the screen can actually be a 24 bit + controls Display Parallel Interface driving an FPGA. This page explains how to do the wiring . At 60Mhz, the display would be 1 MegaPixel. Maybe as much as a 2 MegaPixel display is possible.

GateWare

I am working on it. There is an open source ECP5 color bar gateware. I need to instead drive it from the camera input.

USB Output

For development it is fine to use the ECP5 DVI output. to show it off, or capture images or video, it is nice to have a DVI to USB dongle shown below. Make sure to buy the USB 3 dongles, not the USB 2 dongles.

Code Repository

As soon as it is all working github and codeberg repositories will be created.

Questions???

Please send me an email

ECP-Z.    View      

ECP-Z is a project to connect an ECP5 FPGA with the Zephyr Operating System.

It is already possible to connect an ECP5 FPGA with the Zephyr operating system. IcePi Zero offers a 40 pin Raspberry Pi form factor header with 12 Power and Ground pins, and 28 signal pins. ULX3S offers 2 headers with 56 available pins.

The goal of this project is to enable video preprocessing on an ECP5 connected to a Zephyr microcontroller. For that a new board is needed with 24 pin FPC connectors. The ECP5 and Zephyr would be connected using I2C, Quad SPI, JTag and Uart. They would share an oscillator.