.. include:: /keyword.rst .. _feature-g700-cam-v4l2-vc-sensor: ============================= Virtual Channel (V4L2 Sensor) ============================= .. important:: All the bash commands shown here are based on |G510-G700-EVK|. For |G510-EVK|, users can follow the same step to do the whole setup by changing the naming term from `700` to `510`. .. contents:: Sections :local: :depth: 2 .. note:: All command operations presented in this chapter are based on the IoT Yocto v24.0, |G700-EVK-REF-BOARD|. You might get different operation results depending on the platform you use. This chapter shows how to use virtual channel sensors (e.g. SerDes) under **V4L2 Sensor** architecture on |G510-G700-EVK|. Introduction To Virtual Channel And SerDes ------------------------------------------ Virtual Channel ~~~~~~~~~~~~~~~ The MIPI Camera Serial Interface (CSI) is a standard communication protocol used primarily for transmitting data between cameras and host processors. One of the key features of the MIPI CSI protocol is Virtual Channel. This feature is particularly important as it enhances the flexibility and scalability of camera systems in embedded applications. Virtual channels are logical channels that allow multiple video streams to be transmitted over a single physical connection. This means that a single MIPI CSI interface can handle data from multiple camera sources simultaneously, without the need for additional physical lanes. Each virtual channel is identified by a unique ID, which is used to distinguish between different streams of data. When a camera captures an image, the data is packaged into frames and tagged with the virtual channel ID. The host processor (e.g. MT8370, MT8390, MT8395) then uses these IDs to route and process each stream appropriately. This process is seamless and transparent to the end-user but requires careful configuration and synchronization to ensure data integrity and timely delivery. The benefits of virtual channels are #. Efficiency: Maximize the use of the physical lanes available, allowing for more data throughput without the need for extra hardware. #. Flexibility: Transmit multiple streams from different cameras, which can be useful in advanced applications that require data from multiple sensors. #. Cost-effective: Reduce the number of physical connections needed for multiple camera setups to lower the overall system cost and complexity. SerDes ~~~~~~ SerDes, short for Serializer/Deserializer, is a crucial technology used in high-speed communications. It facilitates the transfer of data over a differential pair of wires or a single high-speed data lane. This technology is fundamental in various applications, including data centers, consumer electronics, automotive systems, and telecommunications. The serializer converts parallel data into a serial stream, reducing the number of data paths and thus minimizing the number of wires needed for data transmission. Conversely, the deserializer performs the opposite function. It converts the serial data back into parallel form, making it usable by the receiving system. In a typical SerDes setup, the serializer (e.g. MAX96705) receives parallel data from the data source along with a clock signal. It aligns the data bits into a serial stream and sends it over a high-speed transmission medium. At the receiving end, the deserializer (e.g. MAX9286) extracts the clock and realigns the incoming serial data into parallel format. Finally, the deserializer transmits the parallel data to the host receiver. The benefits of using SerDes are #. Reduced Complexity: By minimizing the number of physical connections required for data transmission, SerDes simplifies board layout and reduces hardware complexity. #. Increased Speed: Enables high-speed data transmission that is essential for modern applications, such as video streaming, high-performance computing, and high-speed networking. #. Improved Signal Integrity: Helps in maintaining the quality of the signal over longer distances by using differential signaling. Camera Daughter Board --------------------- The SerDes camera DTB for |G510-G700-EVK| is ``CAM DTB-D7``. It contains a Maxim MAX9286 deserializer. .. figure:: /_asset/sw_yocto_app-dev_camera_cam-dtb-d7.png :align: center :scale: 30% Camera DTB D7 - Maxim MAX9286 Deserializer .. important:: The SerDes Camera DTB ``CAM DTB-D7`` needs an additional 12V power supply for PoC (Power Over Coax). Make sure the power is set correctly before using them. The camera module is ``ABEO AG190C``. It contains an OV10640 sensor, an OV490 ISP, and a MAX96705 serializer. .. figure:: /_asset/sw_yocto_app-dev_camera_ag190c.png :align: center :scale: 30% Camera Module - ABEO AG190C Connect The SerDes To The EVK ----------------------------- There are 2 MIPI-CSI sockets on |G510-G700-EVK|, which are ``CSI0`` and ``CSI1``. In the following figure, the camera DTB is connected to the ``CSI0`` socket. .. figure:: /_asset/sw_yocto_app-dev_camera_g700_connect-single-d7.png :align: center :scale: 30% |G510-G700-EVK| with 1x CAM-DTB-D7 and 4x ABEO AG190C sensors The hardware connection is ``AG190C ---> MAX9286 ---> MIPI-CSI On EVK``. .. figure:: /_asset/sw_yocto_app-dev_camera_single-serdes-diagram.svg :align: center :scale: 80% Hardware Block Diagram of Single SerDes and SoC After finishing the above steps, the hardware connection is complete. Select Camera Device Tree Blob Overlay -------------------------------------- The camera device is inactive by default, so the kernel has to load a specific device tree blob overlay to enable it. Please refer to the :ref:`bl33 (u-boot) section ` for how to load DTBO. The following table shows the supported SerDes DTBO in V4L2 sensor architecture. .. csv-filter:: SerDes Camera DTBO for |G510-G700-EVK| :header-rows: 1 :file: ../../../../_asset/tables/camera-platform-sensor-dtbo.csv :include: {0: 'Genio 510\/700-EVK'} :exclude: {5: '[^5.*]'} :included_cols: 0,1,2,4,7 .. prompt:: bash > auto > genio-flash --list-dtbo List of available DTBO: - camera-ag190c-max9286-csi0-std.dtbo - camera-ag190c-max9286-dual-std.dtbo - ... > genio-flash --load-dtbo camera-ag190c-max9286-csi0-std.dtbo --load-dtbo gpu-mali.dtbo --load-dtbo apusys.dtbo --load-dtbo video.dtbo Media Device ------------ The media device represents the internal topology of a media system like ISP. The topology setting including links, pads, and entities can be set through the media device. Therefore, the first step is to find the target media device node. .. prompt:: bash # auto # export MEDIA_DEV=$(v4l2-ctl -z platform:$(basename `find /sys/bus/platform/devices/ -name "*camisp"`) --list-devices | grep media) # echo ${MEDIA_DEV} /dev/media3 .. warning:: The assigned node number for the media device may vary depending on the sequence of the probing process during the boot-up phase. As such, it is crucial to ensure the accuracy of the media device node before using the camera. Video Device ------------ The video device is the interface for users to obtain the images. On |G510-G700-EVK|, they are called ``mtk-cam camsv-* main-stream``. The ``media-ctl`` is a useful tool to list video devices. .. prompt:: bash # auto # declare -a VIDEO_DEV=(`for i in {0..7}; do media-ctl -d ${MEDIA_DEV} --entity "mtk-cam camsv-$i main-stream"; done | tr "\n" " "`) # echo ${VIDEO_DEV[*]} /dev/video53 /dev/video54 /dev/video55 /dev/video56 /dev/video57 /dev/video58 /dev/video59 /dev/video60 .. warning:: The assigned node number for the video device may vary depending on the sequence of the probing process during the boot-up phase. As such, it is crucial to ensure the accuracy of the video device node before using the camera. Launch Camera ------------- The initial step is to link all the necessary components within the media device. The structure of the YUV camera pipeline is quite straightforward. It fundamentally incorporates only three components: the sensor, SENINF, and CAMSV. .. prompt:: bash # auto # # Set the name of MAX9286 # MAX9286_1_NAME="max9286 5-0048" # # media-ctl -d ${MEDIA_DEV} -l "'${MAX9286_1_NAME}':4 -> 'seninf-0':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':1 -> 'mtk-cam camsv-0':0 [5]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':2 -> 'mtk-cam camsv-1':0 [5]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':3 -> 'mtk-cam camsv-2':0 [5]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':4 -> 'mtk-cam camsv-3':0 [5]" The second step is to assign the correct format and size to each component incorporated in the camera pipeline. .. prompt:: bash # auto # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':0 [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':2 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':3 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':4 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-0':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-1':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-2':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-3':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" For the third step, it's necessary to identify the correct video device node. For further information on this, please refer to `Video Device`_. .. prompt:: bash # auto # declare -a VIDEO_DEV=(`for i in {0..7}; do media-ctl -d ${MEDIA_DEV} --entity "mtk-cam camsv-$i main-stream"; done | tr "\n" " "`) # echo ${VIDEO_DEV[*]} /dev/video53 /dev/video54 /dev/video55 /dev/video56 /dev/video57 /dev/video58 /dev/video59 /dev/video60 .. important:: You have to select the video node that corresponds to the CAMSV instance linked in the second step. In this given example, ``mtk-cam camsv-0/1/2/3`` are the linked instances, hence the video device node should be ``${VIDEO_DEV[0]}, ${VIDEO_DEV[1]}, ${VIDEO_DEV[2]}, ${VIDEO_DEV[3]}`` or ``/dev/video{53,54,55,56}``. Lastly, to activate the camera, the user can use ``v4l2-ctl``, ``gst-launch-1.0``, or other V4L2 applications. - Use ``v4l2-ctl`` .. prompt:: bash # auto # v4l2-ctl -d ${VIDEO_DEV[0]} --set-fmt-video=width=1280,height=720,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc0.yuv --verbose # v4l2-ctl -d ${VIDEO_DEV[1]} --set-fmt-video=width=1280,height=720,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc1.yuv --verbose # v4l2-ctl -d ${VIDEO_DEV[2]} --set-fmt-video=width=1280,height=720,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc2.yuv --verbose # v4l2-ctl -d ${VIDEO_DEV[3]} --set-fmt-video=width=1280,height=720,pixelformat=YUYV --stream-mmap --stream-count=10 --stream-to=/tmp/serdes_vc3.yuv --verbose - Use ``gst-launch-1.0`` .. prompt:: bash # auto # gst-launch-1.0 v4l2src device=${VIDEO_DEV[0]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[1]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[2]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[3]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=640,height=480 ! waylandsink sync=false & .. figure:: /_asset/sw_yocto_app-dev_camera_4xserdes-show-on-weston.png :align: center :scale: 30% Show 4x SerDes stream on Weston .. note:: To use MediaTek video codec and MDP hardware, the user has to load ``video.dtbo``. For more details, please refer to :doc:`Genio 700-EVK Video Codec `. Dual Maxim SerDes MAX9286 & ABEO AG190C --------------------------------------- Hardware Setup ~~~~~~~~~~~~~~ #. Prepare 2x CAM-DTB-D7 and 8x ABEO AG190C sensors #. Connect the DTB to CSI0 and CSI1 slots .. figure:: /_asset/sw_yocto_app-dev_camera_g700_connect-dual-d7.png :align: center :scale: 30% |G510-G700-EVK| with 2x CAM-DTB-D7 and 8x ABEO AG190C sensors .. figure:: /_asset/sw_yocto_app-dev_camera_dual-serdes-diagram.svg :align: center :scale: 80% Hardware Block Diagram of Dual SerDes and SoC Software Setup ~~~~~~~~~~~~~~ #. Select the `dtbo` ``camera-ag190c-max9286-dual-std.dtbo`` through ``genio-flash`` .. prompt:: bash > auto > genio-flash --load-dtbo camera-ag190c-max9286-dual-std.dtbo --load-dtbo gpu-mali.dtbo --load-dtbo video.dtbo #. Find the media device of the camera system .. prompt:: bash # auto # export MEDIA_DEV=$(v4l2-ctl -z platform:$(basename `find /sys/bus/platform/devices/ -name "*camisp"`) --list-devices | grep media) # echo ${MEDIA_DEV} /dev/media3 #. Set the media topology and the format on the EVK .. prompt:: bash # auto # # Set the name of MAX9286 # MAX9286_1_NAME="max9286 5-0048" # MAX9286_2_NAME="max9286 3-0048" # # # Set CSI0 pipeline # media-ctl -d ${MEDIA_DEV} -l "'${MAX9286_1_NAME}':4 -> 'seninf-0':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':1 -> 'mtk-cam camsv-0':0 [5]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':2 -> 'mtk-cam camsv-1':0 [5]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':3 -> 'mtk-cam camsv-2':0 [5]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-0':4 -> 'mtk-cam camsv-3':0 [5]" # # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':0 [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':2 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':3 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-0':4 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-0':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-1':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-2':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-3':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # # # Set CSI1 pipeline # media-ctl -d ${MEDIA_DEV} -l "'${MAX9286_2_NAME}':4 -> 'seninf-1':0 [1]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-1':1 -> 'mtk-cam camsv-4':0 [5]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-1':2 -> 'mtk-cam camsv-5':0 [5]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-1':3 -> 'mtk-cam camsv-6':0 [5]" # media-ctl -d ${MEDIA_DEV} -l "'seninf-1':4 -> 'mtk-cam camsv-7':0 [5]" # # media-ctl -d ${MEDIA_DEV} -V "'seninf-1':0 [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-1':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-1':2 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-1':3 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'seninf-1':4 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-4':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-5':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-6':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" # media-ctl -d ${MEDIA_DEV} -V "'mtk-cam camsv-7':1 [fmt:YUYV8_1X16/1280x720 field:none colorspace:srgb]" #. Find the video device of the camera system .. prompt:: bash # auto # declare -a VIDEO_DEV=(`for i in {0..7}; do media-ctl -d ${MEDIA_DEV} --entity "mtk-cam camsv-$i main-stream"; done | tr "\n" " "`) #. Launch dual Maxim SerDes MAX9286 pipeline to show images on the Weston desktop .. prompt:: bash # auto # gst-launch-1.0 v4l2src device=${VIDEO_DEV[0]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[1]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[2]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[3]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[4]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[5]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[6]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false & # gst-launch-1.0 v4l2src device=${VIDEO_DEV[7]} ! video/x-raw,width=1280,height=720,format=YUY2 ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=320,height=240 ! waylandsink sync=false & .. figure:: /_asset/sw_yocto_app-dev_camera_8xserdes-show-on-weston.png :align: center :scale: 30% Show 8x SerDes stream on Weston