Virtual Channel (V4L2 Sensor)

Important

All the bash commands shown here are based on Genio 510/700-EVK. For Genio 510-EVK, users can follow the same step to do the whole setup by changing the naming term from 700 to 510.

Note

All command operations presented in this chapter are based on the IoT Yocto v24.0, Genio 700-EVK. You might get different operation results depending on the platform you use.

This chapter shows how to use virtual channel sensors (e.g. SerDes) under V4L2 Sensor architecture on Genio 510/700-EVK.

Introduction To Virtual Channel And SerDes

Virtual Channel

The MIPI Camera Serial Interface (CSI) is a standard communication protocol used primarily for transmitting data between cameras and host processors. One of the key features of the MIPI CSI protocol is Virtual Channel. This feature is particularly important as it enhances the flexibility and scalability of camera systems in embedded applications.

Virtual channels are logical channels that allow multiple video streams to be transmitted over a single physical connection. This means that a single MIPI CSI interface can handle data from multiple camera sources simultaneously, without the need for additional physical lanes. Each virtual channel is identified by a unique ID, which is used to distinguish between different streams of data.

When a camera captures an image, the data is packaged into frames and tagged with the virtual channel ID. The host processor (e.g. MT8370, MT8390, MT8395) then uses these IDs to route and process each stream appropriately. This process is seamless and transparent to the end-user but requires careful configuration and synchronization to ensure data integrity and timely delivery.

The benefits of virtual channels are

  1. Efficiency: Maximize the use of the physical lanes available, allowing for more data throughput without the need for extra hardware.

  2. Flexibility: Transmit multiple streams from different cameras, which can be useful in advanced applications that require data from multiple sensors.

  3. Cost-effective: Reduce the number of physical connections needed for multiple camera setups to lower the overall system cost and complexity.

SerDes

SerDes, short for Serializer/Deserializer, is a crucial technology used in high-speed communications. It facilitates the transfer of data over a differential pair of wires or a single high-speed data lane. This technology is fundamental in various applications, including data centers, consumer electronics, automotive systems, and telecommunications.

The serializer converts parallel data into a serial stream, reducing the number of data paths and thus minimizing the number of wires needed for data transmission. Conversely, the deserializer performs the opposite function. It converts the serial data back into parallel form, making it usable by the receiving system.

In a typical SerDes setup, the serializer (e.g. MAX96705) receives parallel data from the data source along with a clock signal. It aligns the data bits into a serial stream and sends it over a high-speed transmission medium. At the receiving end, the deserializer (e.g. MAX9286) extracts the clock and realigns the incoming serial data into parallel format. Finally, the deserializer transmits the parallel data to the host receiver.

The benefits of using SerDes are

  1. Reduced Complexity: By minimizing the number of physical connections required for data transmission, SerDes simplifies board layout and reduces hardware complexity.

  2. Increased Speed: Enables high-speed data transmission that is essential for modern applications, such as video streaming, high-performance computing, and high-speed networking.

  3. Improved Signal Integrity: Helps in maintaining the quality of the signal over longer distances by using differential signaling.

Camera Daughter Board

The SerDes camera DTB for Genio 510/700-EVK is CAM DTB-D7. It contains a Maxim MAX9286 deserializer.

../../../../_images/sw_yocto_app-dev_camera_cam-dtb-d7.png

Camera DTB D7 - Maxim MAX9286 Deserializer

Important

The SerDes Camera DTB CAM DTB-D7 needs an additional 12V power supply for PoC (Power Over Coax). Make sure the power is set correctly before using them.

The camera module is ABEO AG190C. It contains an OV10640 sensor, an OV490 ISP, and a MAX96705 serializer.

../../../../_images/sw_yocto_app-dev_camera_ag190c.png

Camera Module - ABEO AG190C

Connect The SerDes To The EVK

There are 2 MIPI-CSI sockets on Genio 510/700-EVK, which are CSI0 and CSI1. In the following figure, the camera DTB is connected to the CSI0 socket.

../../../../_images/sw_yocto_app-dev_camera_g700_connect-single-d7.png

Genio 510/700-EVK with 1x CAM-DTB-D7 and 4x ABEO AG190C sensors

The hardware connection is AG190C ---> MAX9286 ---> MIPI-CSI On EVK.

../../../../_images/sw_yocto_app-dev_camera_single-serdes-diagram.svg

Hardware Block Diagram of Single SerDes and SoC

After finishing the above steps, the hardware connection is complete.

Select Camera Device Tree Blob Overlay

The camera device is inactive by default, so the kernel has to load a specific device tree blob overlay to enable it. Please refer to the bl33 (u-boot) section for how to load DTBO.

The following table shows the supported SerDes DTBO in V4L2 sensor architecture.

SerDes Camera DTBO for Genio 510/700-EVK

Platform

CSI0

CSI1

DTBO

Usage

Genio 510/700-EVK

MAX9286+AG190C

x

camera-ag190c-max9286-csi0-std.dtbo

Enable 4-lane CAM-DTB-D7 w/ ABEO AG190C on CSI0

Genio 510/700-EVK

MAX9286+AG190C

MAX9286+AG190C

camera-ag190c-max9286-dual-std.dtbo

Enable 4-lane CAM-DTB-D7 w/ ABEO AG190C on CSI0 and CSI1

 
genio-flash --list-dtbo
List of available DTBO:
        - camera-ag190c-max9286-csi0-std.dtbo
        - camera-ag190c-max9286-dual-std.dtbo
        - ...
genio-flash --load-dtbo camera-ag190c-max9286-csi0-std.dtbo --load-dtbo gpu-mali.dtbo --load-dtbo apusys.dtbo --load-dtbo video.dtbo

Media Device

The media device represents the internal topology of a media system like ISP. The topology setting including links, pads, and entities can be set through the media device. Therefore, the first step is to find the target media device node.

export MEDIA_DEV=$(v4l2-ctl -z platform:$(basename `find /sys/bus/platform/devices/ -name "*camisp"`) --list-devices | grep media)
echo ${MEDIA_DEV}
/dev/media3

Warning

The assigned node number for the media device may vary depending on the sequence of the probing process during the boot-up phase. As such, it is crucial to ensure the accuracy of the media device node before using the camera.

Video Device

The video device is the interface for users to obtain the images. On Genio 510/700-EVK, they are called mtk-cam camsv-* main-stream. The media-ctl is a useful tool to list video devices.

declare -a VIDEO_DEV=(`for i in {0..7}; do media-ctl -d ${MEDIA_DEV} --entity "mtk-cam camsv-$i main-stream"; done | tr "\n" " "`)
echo ${VIDEO_DEV[*]}
/dev/video53 /dev/video54 /dev/video55 /dev/video56 /dev/video57 /dev/video58 /dev/video59 /dev/video60

Warning

The assigned node number for the video device may vary depending on the sequence of the probing process during the boot-up phase. As such, it is crucial to ensure the accuracy of the video device node before using the camera.

Launch Camera

The initial step is to link all the necessary components within the media device. The structure of the YUV camera pipeline is quite straightforward. It fundamentally incorporates only three components: the sensor, SENINF, and CAMSV.

# Set the name of MAX9286
MAX9286_1_NAME="max9286 5-0048"

media-ctl -d ${MEDIA_DEV} -l "'${MAX9286_1_NAME}':4 -> 'seninf-0':0 [1]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-0':1 -> 'mtk-cam camsv-0':0 [5]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-0':2 -> 'mtk-cam camsv-1':0 [5]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-0':3 -> 'mtk-cam camsv-2':0 [5]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-0':4 -> 'mtk-cam camsv-3':0 [5]"

The second step is to assign the correct format and size to each component incorporated in the camera pipeline.

media-ctl -d ${MEDIA_DEV} -V "'seninf-0':0 [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-0':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-0':2 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-0':3 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-0':4 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-0':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-1':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-2':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-3':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"

For the third step, it’s necessary to identify the correct video device node. For further information on this, please refer to Video Device.

declare -a VIDEO_DEV=(`for i in {0..7}; do media-ctl -d ${MEDIA_DEV} --entity "mtk-cam camsv-$i main-stream"; done | tr "\n" " "`)
echo ${VIDEO_DEV[*]}
/dev/video53 /dev/video54 /dev/video55 /dev/video56 /dev/video57 /dev/video58 /dev/video59 /dev/video60

Important

You have to select the video node that corresponds to the CAMSV instance linked in the second step. In this given example, mtk-cam camsv-0/1/2/3 are the linked instances, hence the video device node should be ${VIDEO_DEV[0]}, ${VIDEO_DEV[1]}, ${VIDEO_DEV[2]}, ${VIDEO_DEV[3]} or /dev/video{53,54,55,56}.

Lastly, to activate the camera, the user can use v4l2-ctl, gst-launch-1.0, or other V4L2 applications.

  • Use v4l2-ctl

    v4l2-ctl -d ${VIDEO_DEV[0]} --set-fmt-video=width=1280,height=720,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc0.yuv --verbose
v4l2-ctl -d ${VIDEO_DEV[1]} --set-fmt-video=width=1280,height=720,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc1.yuv --verbose
v4l2-ctl -d ${VIDEO_DEV[2]} --set-fmt-video=width=1280,height=720,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc2.yuv --verbose
v4l2-ctl -d ${VIDEO_DEV[3]} --set-fmt-video=width=1280,height=720,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc3.yuv --verbose

  • Use gst-launch-1.0

    gst-launch-1.0 v4l2src device=${VIDEO_DEV[0]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[1]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[2]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[3]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false &
    ../../../../_images/sw_yocto_app-dev_camera_4xserdes-show-on-weston.png

    Show 4x SerDes stream on Weston

Note

To use MediaTek video codec and MDP hardware, the user has to load video.dtbo. For more details, please refer to Genio 700-EVK Video Codec.

Dual Maxim SerDes MAX9286 & ABEO AG190C

Hardware Setup

  1. Prepare 2x CAM-DTB-D7 and 8x ABEO AG190C sensors

  2. Connect the DTB to CSI0 and CSI1 slots

    ../../../../_images/sw_yocto_app-dev_camera_g700_connect-dual-d7.png

    Genio 510/700-EVK with 2x CAM-DTB-D7 and 8x ABEO AG190C sensors

    ../../../../_images/sw_yocto_app-dev_camera_dual-serdes-diagram.svg

    Hardware Block Diagram of Dual SerDes and SoC

Software Setup

  1. Select the dtbo camera-ag190c-max9286-dual-std.dtbo through genio-flash

    genio-flash --load-dtbo camera-ag190c-max9286-dual-std.dtbo --load-dtbo gpu-mali.dtbo --load-dtbo video.dtbo

  2. Find the media device of the camera system

    export MEDIA_DEV=$(v4l2-ctl -z platform:$(basename `find /sys/bus/platform/devices/ -name "*camisp"`) --list-devices | grep media)
echo ${MEDIA_DEV}
/dev/media3

  3. Set the media topology and the format on the EVK

    # Set the name of MAX9286
MAX9286_1_NAME="max9286 5-0048"
MAX9286_2_NAME="max9286 3-0048"

# Set CSI0 pipeline
media-ctl -d ${MEDIA_DEV} -l "'${MAX9286_1_NAME}':4 -> 'seninf-0':0 [1]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-0':1 -> 'mtk-cam camsv-0':0 [5]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-0':2 -> 'mtk-cam camsv-1':0 [5]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-0':3 -> 'mtk-cam camsv-2':0 [5]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-0':4 -> 'mtk-cam camsv-3':0 [5]"

media-ctl -d ${MEDIA_DEV} -V "'seninf-0':0 [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-0':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-0':2 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-0':3 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-0':4 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-0':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-1':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-2':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-3':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"

# Set CSI1 pipeline
media-ctl -d ${MEDIA_DEV} -l "'${MAX9286_2_NAME}':4 -> 'seninf-1':0 [1]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-1':1 -> 'mtk-cam camsv-4':0 [5]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-1':2 -> 'mtk-cam camsv-5':0 [5]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-1':3 -> 'mtk-cam camsv-6':0 [5]"
media-ctl -d ${MEDIA_DEV} -l "'seninf-1':4 -> 'mtk-cam camsv-7':0 [5]"

media-ctl -d ${MEDIA_DEV} -V "'seninf-1':0 [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-1':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-1':2 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-1':3 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'seninf-1':4 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-4':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-5':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-6':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"
media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-7':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]"

  4. Find the video device of the camera system

    declare -a VIDEO_DEV=(`for i in {0..7}; do media-ctl -d ${MEDIA_DEV} --entity "mtk-cam camsv-$i main-stream"; done | tr "\n" " "`)

  5. Launch dual Maxim SerDes MAX9286 pipeline to show images on the Weston desktop

    gst-launch-1.0 v4l2src device=${VIDEO_DEV[0]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[1]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[2]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[3]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[4]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[5]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[6]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${VIDEO_DEV[7]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false &
    ../../../../_images/sw_yocto_app-dev_camera_8xserdes-show-on-weston.png

    Show 8x SerDes stream on Weston