DOCS
RPi Camera FMC (OP068)

Example Designs

The example designs for the RPi Camera FMC are open source under the MIT license and maintained on Github. We strongly encourage community contributions to the example designs; if you would like to help but don’t know where to start, please contact us .

Example Designs
RPi Camera Example DesignMore infoGit repoDocs
Vision AI apps for ZCU10x Ubuntu 22.04 LTSMore infoGit repoDocs
Multi-cam ZynqMP and Hailo-8 AI Acceleration DemoMore infoGit repoDocs
MIPI over Chip2Chip Example DesignMore infoGit repoDocs

RPi Camera FMC example design

Description

This example design demonstrates use of the AMD Xilinx MIPI CSI Controller Subsystem IP and a video pipeline for receiving video streams simultaneously from up to 4x Raspberry Pi cameras and displaying the videos on a single monitor. The designs fall into two architectures:

  • Zynq UltraScale+ designs: four capture pipelines with the ISP Pipeline IP, a Video Mixer based display pipeline to a DisplayPort monitor and the Video Codec Unit (VCU) where the device has one. They run embedded Linux built with PetaLinux or Yocto (AMD EDF), with libcamera support.
  • FPGA designs: a simpler video pipeline for processor-less boards, controlled by a bare-metal application running on a MicroBlaze, with output to an HDMI monitor.

Supported carrier boards

The Git repo contains an example design for each of the following carrier boards.

  • AMD Xilinx ZCU104 Zynq UltraScale+ Development board (4 cameras)
  • AMD Xilinx ZCU102 Zynq UltraScale+ Development board (4 cameras on HPC0, 2 cameras on HPC1)
  • AMD Xilinx ZCU106 Zynq UltraScale+ Development board (4 cameras)
  • TUL PYNQ-ZU Zynq UltraScale+ Development board (2 cameras)
  • Avnet UltraZed EV Carrier Zynq UltraScale+ Development board (4 cameras)
  • Tria AUBoard 15P Artix UltraScale+ Development board (2 cameras, FPGA design)

Supported cameras

Note that the RPi Camera FMC is designed to support all cameras with the standard 15-pin Raspberry Pi camera interface, however this example design currently only has the software to support the following:

Support for more cameras will be added soon. We’re working hard on this and if you’d like to help us, please do. We accept pull requests on the Git repository.

Block diagrams

Zynq UltraScale+ designs

The top level block diagram of the Zynq UltraScale+ designs is illustrated below. RPi Camera FMC example design

The main features illustrated in the block diagram are:

  • Each camera has its own capture pipeline (mipi_0 to mipi_3), a sub-block composed of multiple IPs which is described below. Targets with only two cameras build only the pipelines they use.
  • The capture pipelines write frames into the PS DDR memory through an AXI SmartConnect connected to one of the high-performance slave interfaces of the PS (S_AXI_HPC0_FPD).
  • The display pipeline reads the camera frames back from DDR into a Video Mixer ( AMD Xilinx Video Mixer IP ) which combines them into a single 1080p video stream. The combined stream is passed through the AMD Xilinx AXI4-Stream to Video Out IP to generate the live video signals that are fed to the DisplayPort interface of the Zynq UltraScale+.
  • Timing signals for the AXIS to Video Out IP are generated by the AMD Xilinx Video Timing Controller IP , and a Clocking Wizard generates the pixel clock.
  • All of the IPs are controlled from the PS over AXI-Lite, and the Linux drivers (V4L2 media controller and DRM) manage the pipelines.
  • The Video Codec Unit (VCU) is included on the boards that have one (ZCU104, ZCU106, UltraZed-EV) for H.264/H.265 encoding and decoding.

Capture pipeline

The capture pipeline sub-block (one per camera) is illustrated below. RPi Camera FMC capture pipeline

The main features of the capture pipeline are:

  • The main input to the block is the MIPI CSI-2 interface from the camera. This connects to the AMD Xilinx MIPI CSI Controller Subsystem IP (2-lane D-PHY) which generates a RAW10 video stream on its AXI-Streaming interface.
  • An AXI4-Stream Data FIFO decouples the CSI-2 receiver from the rest of the pipeline.
  • The ISP Pipeline (a Vitis Vision HLS core) performs black level correction, gain, demosaic, auto white balance, colour correction and gamma, converting the raw stream to RGB.
  • The AMD Xilinx Video Processor Subsystem IP scales the video stream and converts the colour space to YUV 4:2:2.
  • The Frame Buffer Write IP writes the video frames into the external DDR memory.
  • Each capture pipeline has its own AXI IIC for the camera’s I2C bus and its own AXI GPIO for the camera’s IO0 and IO1 pins (see GPIO direction and output enable).

Display pipeline

The display pipeline is illustrated below. RPi Camera FMC display pipeline

FPGA designs

The FPGA designs (for the processor-less AUBoard 15P ) use a simpler video pipeline and output to an HDMI monitor. The display pipeline and the video pipeline of these designs are illustrated below. RPi Camera FMC HDMI display pipeline RPi Camera FMC simple video pipeline

Vision AI apps for ZCU10x Ubuntu 22.04 LTS

Description

These example designs demonstrate two smart vision apps that run on Certified Ubuntu 22.04 LTS for ZCU10x boards (for the ZCU10x boards only) or PetaLinux . These applications were designed by AMD Xilinx for the Kria KV260 platform and we ported them to the ZCU10x boards for use with the RPi Camera FMC. More detailed documentation regarding the applications can be found on the Kria KV260 Vision AI Starter Kit application documentation.

  • Smartcam: An accelerated face detection application that supports RTSP network streaming output or DisplayPort output. This application can be used with a single Raspberry Pi camera connected to port CAM1 of the RPi Camera FMC.
  • NLP Smart Vision: An accelerated face detection, object detection and plate detection application that uses Natural Language Processing to switch between AI tasks using verbal commands. This application can be used with one of two Raspberry Pi cameras connected to ports CAM1 and CAM2 of the RPi Camera FMC.

Supported carrier boards

The Git repo contains an example design for each of the following carrier boards.

  • AMD Xilinx ZCU104 Zynq UltraScale+ Development board
  • AMD Xilinx ZCU102 Zynq UltraScale+ Development board
  • AMD Xilinx ZCU106 Zynq UltraScale+ Development board

Supported cameras

These example designs currently support the following cameras:

Further documentation

The following tutorials have been written to help you get started building and using these example designs:

Multi-cam ZynqMP and Hailo-8 AI Acceleration Demo

Description

This example design demonstrates a multi-camera YOLOv5 implementation that runs on the Zynq UltraScale+ and the Hailo-8 AI accelerator. The RPi Camera FMC connects 4x Raspberry Pi Camera Module 2 , and the Hailo-8 M.2 AI Acceleration Module connects through the M.2 M-key Stack FMC (with the RPi Camera FMC stacked on top of it) or, on the ZCU106, through the FPGA Drive FMC Gen4 on a second FMC connector. The camera frames are captured into DDR through the PL, inference runs on the Hailo-8 over PCIe using HailoRT, and the results are displayed on a DisplayPort monitor.

Block diagram

ZynqMP Hailo AI example design block diagram

Supported carrier boards

The Git repo contains an example design for each of the following carrier boards.

  • AMD Xilinx ZCU104 Zynq UltraScale+ Development board
  • AMD Xilinx ZCU106 Zynq UltraScale+ Development board
  • TUL PYNQ-ZU Zynq UltraScale+ Development board
  • Avnet UltraZed EV Carrier Zynq UltraScale+ Development board

The designs for some of these boards are limited to 2x cameras and some of them can only support 1x Hailo-8 module. For more information about these limitations, refer to the reference design documentation.

MIPI over Chip2Chip

Description

This example design connects MIPI CSI-2 cameras that are attached to a processor-less FPGA board to a Zynq UltraScale+ host running Linux, using the AMD Xilinx AXI Chip2Chip IP over an Aurora 64B/66B serial link on an SFP+ cable. The host sees the remote video pipelines as ordinary memory-mapped peripherals, and the captured frames are written straight into the host’s DDR memory.

It is the proof of concept of a universal MIPI CSI-2 FMC: an FMC card with its own FPGA. Putting the camera pipelines on a small FPGA on the mezzanine fixes the MIPI pin assignment there, and the interface to the carrier becomes a generic serial link that almost any FPGA board can provide. Here the roles are played by off-the-shelf hardware: the host is a ZCU106 and a AUBoard 15P carrying the RPi Camera FMC with two Raspberry Pi cameras stands in for the mezzanine card. It is therefore a two-board system: the two target designs of the repository are the two halves of one system, and both must be built.

Supported carrier boards

  • AMD Xilinx ZCU106 Zynq UltraScale+ Development board (host)
  • Tria AUBoard 15P Artix UltraScale+ Development board (mezzanine, carries the RPi Camera FMC)
← EMC integration notes
Board Files →