.. include:: /keyword.rst .. _feature-g720-cam-v4l2-vc-sensor: ============================= Virtual Channel (V4L2 Sensor) ============================= .. contents:: Sections :local: :depth: 2 .. important:: All the bash commands shown here are based on |G720-EVK-REF-BOARD|, MAX96724 EVKit, and AG190H. For |G520-EVK|, users can follow the same steps to perform the setup by changing the name from `720` to `520`. You might get different operation results depending on the platform you use. This chapter shows how to use virtual channel sensors (e.g., SerDes) under **V4L2 Sensor** architecture on |G520-G720-EVK|. Introduction to Virtual Channel and SerDes ------------------------------------------ Virtual Channel ~~~~~~~~~~~~~~~ The MIPI Camera Serial Interface (CSI) is a standard communication protocol used primarily for transmitting data between cameras and host processors. One of the key features of the MIPI CSI protocol is Virtual Channel. This feature is particularly important as it enhances the flexibility and scalability of camera systems in embedded applications. Virtual channels are logical channels that allow multiple video streams to be transmitted over a single physical connection. This means that a single MIPI CSI interface can handle data from multiple camera sources simultaneously, without the need for additional physical lanes. Each virtual channel is identified by a unique ID, which is used to distinguish between different streams of data. When a camera captures an image, the data is packaged into frames and tagged with the virtual channel ID. The host processor (e.g., MT8371, MT8391) then uses these IDs to route and process each stream appropriately. This process is seamless and transparent to the end-user but requires careful configuration and synchronization to ensure data integrity and timely delivery. The benefits of virtual channels include: #. Efficiency: Maximize the use of the physical lanes available, allowing for more data throughput without the need for extra hardware. #. Flexibility: Transmit multiple streams from different cameras, which can be useful in advanced applications that require data from multiple sensors. #. Cost-effective: Reduce the number of physical connections needed for multiple camera setups to lower the overall system cost and complexity. SerDes ~~~~~~ SerDes, short for Serializer/Deserializer, is a crucial technology used in high-speed communications. It facilitates the transfer of data over a differential pair of wires or a single high-speed data lane. This technology is fundamental in various applications, including data centers, consumer electronics, automotive systems, and telecommunications. The serializer converts parallel data into a serial stream, reducing the number of data paths and thus minimizing the number of wires needed for data transmission. Conversely, the deserializer performs the opposite function. It converts the serial data back into parallel form, making it usable by the receiving system. In a typical SerDes setup, the serializer (e.g., MAX96705, MAX9295) receives parallel data from the data source along with a clock signal. It aligns the data bits into a serial stream and sends it over a high-speed transmission medium. At the receiving end, the deserializer (e.g. MAX9286, MAX96724) extracts the clock and realigns the incoming serial data into parallel format. Finally, the deserializer transmits the parallel data to the host receiver. The benefits of using SerDes include: #. Reduced Complexity: By minimizing the number of physical connections required for data transmission, SerDes simplifies board layout and reduces hardware complexity. #. Increased Speed: Enables high-speed data transmission that is essential for modern applications, such as video streaming, high-performance computing, and high-speed networking. #. Improved Signal Integrity: Helps in maintaining the quality of the signal over longer distances by using differential signaling. Camera Daughter Board --------------------- The SerDes camera DTB for |G520-G720-EVK| is ``IoT-CAM-DTB-D10``. It contains a ``MAX96724-BAK-EVK#`` and a ``GMSL EVKIT RPi Adaptor``. .. figure:: /_asset/sw_yocto_app-dev_camera_cam-dtb-d10.jpg :align: center :scale: 30% Camera DTB D10 - MAX96724-BAK-EVK# and GMSL EVKIT Rpi Adaptor The camera module is ``ABEO AG190H``. It contains an OmniVision OX03C sensor, an OmniVision OAX4000 ISP, and an ADI MAX9295 serializer. .. figure:: /_asset/sw_yocto_app-dev_camera_ag190h.jpg :align: center :scale: 30% Camera Module - ABEO AG190H Connect the SerDes to the EVK ----------------------------- There are 2 MIPI-CSI sockets on |G720-EVK|, which are ``CAM0`` and ``CAM1``. In the following figure, the camera DTB is connected to the ``CAM0`` socket. .. figure:: /_asset/sw_yocto_app-dev_camera_g720-single-serdes-connection.png :align: center :scale: 30% |G720-EVK| with 1x MAX96724-BAK-EVK and 4x ABEO AG190H sensors The FPC cable between the |G720-EVK| and MAX96724 EVK MUST be twisted to ensure correct orientation. .. figure:: /_asset/sw_yocto_app-dev_camera_g720-serdes-fpc-connection.png :align: center :scale: 30% FPC Connection Between |G720-EVK| and MAX96724-BAK-EVK The hardware connection is ``AG190H ---> MAX96724-BAK-EVK ---> MIPI-CSI On Genio 720-EVK``. .. figure:: /_asset/sw_yocto_app-dev_camera_g720-single-serdes-diagram.svg :align: center :scale: 80% Hardware Block Diagram of Single SerDes and SoC After finishing the above steps, the hardware connection is complete. Configure the MAX96724 EVKIT ---------------------------- The MAX96724 EVKIT has various flexible configurations, including jumpers, switches, and programmable settings. To replicate the supported configuration, users MUST follow the steps below to configure the MAX96724 EVK. 1. GMSL EVKIT RPi Adaptor - Set switch ``S1`` to ``CAM2``. - Set switch ``S3`` to the left. - Connect the FPC cable to ``P6 (CAM2)``. .. figure:: /_asset/sw_yocto_app-dev_camera_max96724-rpi-adaptor-setting.png :align: center :scale: 50% The Setting of GMSL EVKIT RPi Adaptor 2. MAX96724-BAK-EVK - Set the jumpers as shown in the following figure .. figure:: /_asset/sw_yocto_app-dev_camera_max96724-jumper-setting.png :align: center :scale: 80% The Jumper Setting of MAX96724-BAK-EVK - Use the ``GMSL-SerDesEV`` tool (provided by ADI) to set the CFG level of MAX96724. The ``CFG0`` and ``CFG1`` should be set to 0. For more details about the tool, please contact ADI. .. figure:: /_asset/sw_yocto_app-dev_camera_max96724-GMSL-cfg-setting.png :align: center :scale: 80% The CFG Setting of MAX96724-BAK-EVK Select Camera Device Tree Blob Overlay -------------------------------------- The camera device is inactive by default, so the kernel must load a specific device tree blob overlay to enable it. Please refer to the :ref:`bl33 (u-boot) section ` for how to load DTBO. The following table shows the supported SerDes DTBO in V4L2 sensor architecture. .. csv-filter:: SerDes Camera DTBO for |G520-G720-EVK| :header-rows: 1 :file: ../../../../_asset/tables/camera-platform-sensor-dtbo.csv :include: {0: 'Genio 520\/720-EVK'} :exclude: {5: '[^5.*]'} :included_cols: 0,1,2,4,7 .. prompt:: bash > auto > genio-flash --list-dtbo List of available DTBO: camera-ag190h-max96724-csi0-std.dtbo camera-ag190h-max96724-csi1-std.dtbo camera-ag190h-max96724-dual-std.dtbo ... > genio-flash --load-dtbo camera-ag190h-max96724-csi0-std.dtbo Media Device ------------ The media device represents the internal topology of a media system like ISP. Topology settings, including links, pads, and entities, can be configured through the media device. Therefore, the first step is to find the target media device node. .. prompt:: bash # auto # export MEDIA_DEV=$(v4l2-ctl -z platform:$(basename `find /sys/bus/platform/devices/ -name "*seninf-top"`) --list-devices | grep media) # echo ${MEDIA_DEV} /dev/media1 .. warning:: The assigned node number for the media device may vary depending on the sequence of the probing process during the boot-up phase. As such, it is crucial to ensure the accuracy of the media device node before using the camera. Video Device ------------ The video device is the interface for users to obtain the images. On |G520-G720-EVK|, they are called ``1a09[2-7]000\.camsv[2-7] video stream``. ``media-ctl`` is a useful tool to list video devices. .. prompt:: bash # auto # declare -a VIDEO_DEV=($(for i in `find /sys/bus/platform/devices/ -name "*camsv*" | sort`; do media-ctl -d ${MEDIA_DEV} --entity "`basename $i` video stream" ; done)) # echo ${VIDEO_DEV[*]} /dev/video5 /dev/video6 /dev/video7 /dev/video8 /dev/video9 /dev/video10 .. warning:: The assigned node number for the video device may vary depending on the sequence of the probing process during the boot-up phase. As such, it is crucial to ensure the accuracy of the video device node before using the camera. Launch Camera ------------- The initial step is to link all the necessary components within the media device. The structure of the YUV camera pipeline is quite straightforward. It consists of only three components: the sensor, SENINF, and CAMSV. .. prompt:: bash # auto # # Set the name of MAX96724 # MAX96724_1_NAME="max96724 7-0027:0" # # media-ctl -d ${MEDIA_DEV} -l "'${MAX96724_1_NAME}':0 -> 'seninf-0':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':1 -> '1a092000.camsv2':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':2 -> '1a093000.camsv3':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':3 -> '1a094000.camsv4':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':4 -> '1a095000.camsv5':0 [1]" The second step is to assign the correct format and size to each component within the camera pipeline. .. prompt:: bash # auto # media-ctl -d ${MEDIA_DEV} -V "'${MAX96724_1_NAME}':0 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':2 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':3 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':4 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a092000.camsv2':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a093000.camsv3':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a094000.camsv4':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a095000.camsv5':1 [fmt:YUYV8_1X16/1920x1080]" Lastly, to activate the camera, the user can use ``v4l2-ctl``, ``gst-launch-1.0``, or other V4L2 applications. - Use ``v4l2-ctl`` .. prompt:: bash # auto # v4l2-ctl -d ${VIDEO_DEV[0]} --set-fmt-video=width=1920,height=1080,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc0.yuv --verbose # v4l2-ctl -d ${VIDEO_DEV[1]} --set-fmt-video=width=1920,height=1080,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc1.yuv --verbose # v4l2-ctl -d ${VIDEO_DEV[2]} --set-fmt-video=width=1920,height=1080,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc2.yuv --verbose # v4l2-ctl -d ${VIDEO_DEV[3]} --set-fmt-video=width=1920,height=1080,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc3.yuv --verbose - Use ``gst-launch-1.0`` .. prompt:: bash # auto # gst-launch-1.0 v4l2src device=${VIDEO_DEV[0]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[1]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[2]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[3]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false & .. figure:: /_asset/sw_yocto_app-dev_camera_g720-4xserdes-show-on-weston.png :align: center :scale: 30% Show 4x SerDes streams on Weston .. note:: To use MediaTek video codec and MDP hardware, users MUST load ``video.dtbo``. For more details, please refer to :doc:`Genio 720-EVK Video Codec `. Dual ADI SerDes MAX96724 & ABEO AG190H -------------------------------------- Hardware Setup ~~~~~~~~~~~~~~ #. Prepare 2x CAM-DTB-D10 and 6x ABEO AG190H sensors #. Connect the DTB to CAM0 and CAM1 slots .. figure:: /_asset/sw_yocto_app-dev_camera_g720-dual-serdes-connection.png :align: center :scale: 50% |G520-G720-EVK| with 2x CAM-DTB-D10 and 6x ABEO AG190H sensors .. figure:: /_asset/sw_yocto_app-dev_camera_g720-dual-serdes-diagram.svg :align: center :scale: 80% Hardware Block Diagram of Dual SerDes and SoC Software Setup ~~~~~~~~~~~~~~ #. Select the `dtbo` ``camera-ag190h-max96724-dual-std.dtbo`` through ``genio-flash`` .. prompt:: bash > auto > genio-flash --load-dtbo camera-ag190h-max96724-dual-std.dtbo #. Find the media device of the camera system .. prompt:: bash # auto # export MEDIA_DEV=$(v4l2-ctl -z platform:$(basename `find /sys/bus/platform/devices/ -name "*seninf-top"`) --list-devices | grep media) # echo ${MEDIA_DEV} /dev/media3 #. Set the media topology and the format on the EVK .. prompt:: bash # auto # # Set the name of MAX96724 # MAX96724_1_NAME="max96724 7-0027:0" # MAX96724_2_NAME="max96724 8-0027:0" # # # Set CAM0 pipeline # media-ctl -d ${MEDIA_DEV} -l "'${MAX96724_1_NAME}':0 -> 'seninf-0':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':1 -> '1a092000.camsv2':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':2 -> '1a093000.camsv3':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':3 -> '1a094000.camsv4':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':4 -> '1a095000.camsv5':0 [1]" # # media-ctl -d ${MEDIA_DEV} -V "'${MAX96724_1_NAME}':0 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':2 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':3 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':4 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a092000.camsv2':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a093000.camsv3':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a094000.camsv4':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a095000.camsv5':1 [fmt:YUYV8_1X16/1920x1080]" # # # Set CAM1 pipeline # media-ctl -d ${MEDIA_DEV} -l "'${MAX96724_2_NAME}':0 -> 'seninf-1':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-1':1 -> '1a096000.camsv6':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-1':2 -> '1a097000.camsv7':0 [1]" # # media-ctl -d ${MEDIA_DEV} -V "'${MAX96724_2_NAME}':0 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-1':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-1':2 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a096000.camsv6':1 [fmt:YUYV8_1X16/1920x1080]" # media-ctl -d ${MEDIA_DEV} -V "'1a097000.camsv7':1 [fmt:YUYV8_1X16/1920x1080]" #. Find the video device of the camera system .. prompt:: bash # auto # declare -a VIDEO_DEV=($(for i in `find /sys/bus/platform/devices/ -name "*camsv*" | sort`; do media-ctl -d ${MEDIA_DEV} --entity "`basename $i` video stream" ; done)) # echo ${VIDEO_DEV[*]} /dev/video5 /dev/video6 /dev/video7 /dev/video8 /dev/video9 /dev/video10 #. Launch dual ADI SerDes MAX96724 pipeline to show images on the Weston desktop .. prompt:: bash # auto # gst-launch-1.0 v4l2src device=${VIDEO_DEV[0]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=480,height=270 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[1]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=480,height=270 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[2]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=480,height=270 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[3]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=480,height=270 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[4]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=480,height=270 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[5]} ! video/x-raw,width=1920,height=1080,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=480,height=270 ! waylandsink sync=false & .. figure:: /_asset/sw_yocto_app-dev_camera_g720-6xserdes-show-on-weston.png :align: center :scale: 30% Show 6x SerDes streams on Weston