.. include:: /keyword.rst .. _feature-g350-cam-v4l2-raw-sensor: ======================== RAW Sensor (V4L2 Sensor) ======================== .. important:: All the bash commands shown here are based on |G350-EVK|. .. contents:: Sections :local: :depth: 2 .. note:: All command operations presented in this chapter are based on the IoT Yocto v24.1, |G350-EVK-REF-BOARD| and Onsemi AR0430 sensor. You might get different operation results depending on the platform you use. This chapter shows how to receive RAW (bayer) sensor data and directly dump it to the DRAM under **V4L2 Sensor** architecture on |G350-EVK|. Camera Daughter Board --------------------- The RAW camera DTB for |G350-EVK| is ``AIoT CAM DTB D1V2``. It contains an Onsemi AP1302 ISP and an Onsemi AR0430 Sensor. We will bypass the AP1302 ISP for RAW mode. The camera DTB connects with the |G350-EVK| through a miniSAS cable. For more details about the ISP and the sensor, please refer to: - `Onsemi AR0430 `_ Connect The Camera to the EVK ----------------------------- There are two MIPI-CSI ports on the EVK, which means you can connect up to two cameras. .. figure:: /_asset/sw_rity_app-dev_camera_mipicsi-port.png :align: center :scale: 50% Two MIPI CSI camera ports on the EVK The hardware connection for RAW mode is ``AR0430 ---> MIPI-CSI On EVK``. The camera daughter board can be configured in CAM+ISP mode and CAM mode. In CAM+ISP mode, the images from the AR0430 camera module are passed to AP1302 ISP before being received by the EVK. In CAM mode, the images from the AR0430 camera module are transferred to the EVK directly. By default, |IOT-YOCTO| supports both, CAM+ISP and CAM mode. In this section, we will capture RAW frames using the CAM mode. First, configure camera DTB in CAM mode. You will need 6 jumpers and 1 camera sensor configured as the below figure. Make sure you have the correct jumper setting and that the camera sensor is properly connected to the CAM+ISP slot. .. figure:: /_asset/sw_rity_app-dev_camera_cam-mode.png :align: center CAM mode Second, connect the camera DTB with a miniSAS cable. .. figure:: /_asset/sw_rity_app-dev_camera_miniSAS.jpeg :align: center Connect miniSAS cable After finishing the above steps, the hardware connection is complete. .. warning:: Please make sure all the above settings, including the jumpers, camera slot, and miniSAS cable, are correct. Otherwise, the I2C connection between the sensor, ISP, and SoC will fail. Select Camera Device Tree Blob Overlay -------------------------------------- The camera is inactive by default. The kernel has to load a specific device tree blob overlay to enable the camera. Please refer to the :ref:`bl33 (u-boot) section ` for more details. .. csv-filter:: RAW Camera DTBO for |G350-EVK| :header-rows: 1 :file: ../../../../_asset/tables/camera-platform-sensor-dtbo.csv :include: {0: 'Genio 350-EVK'} :exclude: {5: '[^0.* ^2.*]', 6: '[^1.* ^3.*]'} :included_cols: 0,1,2,4,7 .. prompt:: bash > auto > genio-flash --list-dtbo List of available DTBO: - camera-ap1302-ar0430-dual.dtbo - camera-ap1302-ar0430-single-csi0.dtbo - camera-ap1302-ar0430-single-csi1.dtbo - camera-ar0430-dual.dtbo - camera-ar0430-single-csi0.dtbo - camera-ar0430-single-csi1.dtbo > genio-flash --load-dtbo camera-ar0430-dual.dtbo # Dual RAW camera on MIPI-CSI0 and MIPICSI1 > genio-flash --load-dtbo camera-ar0430-single-csi0.dtbo # Single RAW camera on MIPI-CSI0 > genio-flash --load-dtbo camera-ar0430-single-csi1.dtbo # Single RAW camera on MIPI-CSI1 .. warning:: Please select the DTBO according to the CSI slot to which the camera sensor is connected. For example, if the camera is connected to the ``CSI0`` slot, please load the `dtbo` ``camera-ar0430-single-csi0.dtbo``. Otherwise, the camera initialization will fail. Supported Formats and Sizes --------------------------- .. csv-table:: :header: "Platform", "Sensor", "Stream Type", "Size", "Frame rate", "Format", "MIPI Lanes" :widths: 10, 20, 5, 5, 5, 5, 5 :align: left "|G350-EVK|", "Onsemi AR0430", "Preview", "2304x1296", "30", "SGRBG10", "4" .. note:: The supported format, resolution, and frame rate are related to the capability of the sensor and SoC. Set Camera Properties through `media-ctl` ----------------------------------------- Before using the camera, you should set the resolution and the pixel format through ``media-ctl``. All of the example in this section uses the dual camera. Onsemi AR0430 sensor captures the image with ``2304 * 1296, SGRBG10`` format. Therefore, you have to configure the format for MIPI-CSI0 and MIPI-CSI1 cameras. The media device is the key to setting up the camera. You should find the media device with the module name, ``mtk-camsys-3.0``. You can operate this media device to set the format, the resolution, and the enablement. The index of the media device may differ depending on the probing order of drivers and the number of devices. The command ``v4l2-ctl --list-devices`` can show the device information. .. prompt:: bash # auto # v4l2-ctl --list-devices mtk-mdp (platform:14004000.mdp_rdma0): /dev/media1 mtk-camsys-3.0 (platform:15040000.seninf): /dev/media0 mtk-camsv-isp30 (platform:15050000.camsv): /dev/video0 mtk-camsv-isp30 (platform:15050800.camsv): /dev/video1 platform:mt8167 (platform:mt8167): /dev/video2 /dev/video4 14004000.mdp_rdma0 (platform:mt8173): /dev/video3 For example, the media device for ``mtk-camsys-3.0`` is ``/dev/media0``. Please replace ``/dev/media`` with the actual node on the platform. To set the format of the MIPI-CSI0 camera: .. prompt:: bash # auto # media-ctl -d /dev/media -V "'ar0430 2-0036':0 [fmt:SGRBG10_1X10/2304x1296]" # media-ctl -d /dev/media -V "'15040000.seninf':4 [fmt:SGRBG10_1X10/2304x1296]" # media-ctl -d /dev/media -V "'15050000.camsv':1 [fmt:SGRBG10_1X10/2304x1296]" To set the format of the MIPI-CSI1 camera: .. prompt:: bash # auto # media-ctl -d /dev/media -V "'ar0430 3-0036':0 [fmt:SGRBG10_1X10/2304x1296]" # media-ctl -d /dev/media -V "'15040000.seninf':5 [fmt:SGRBG10_1X10/2304x1296]" # media-ctl -d /dev/media -V "'15050800.camsv':1 [fmt:SGRBG10_1X10/2304x1296]" .. important:: Before launching the camera, you have to use ``media-ctl`` to set the format and the resolution of the camera pipeline. Otherwise, the camera pipeline may fail due to the unknown format. You can use ``media-ctl`` tool to dump the current state of the device topology. The name and the number in the above commands correspond to the entity and the pad. For more details, you can use ``media-ctl --help``. .. prompt:: bash # auto # media-ctl -d /dev/media -p Media controller API version 5.15.47 Media device information ------------------------ driver mtk-seninf model mtk-camsys-3.0 serial bus info platform:15040000.seninf hw revision 0x0 driver version 5.15.47 Device topology - entity 1: 15040000.seninf (8 pads, 4 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb] <- "ar0430 2-0036":0 [ENABLED,IMMUTABLE] pad1: Sink [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb] <- "ar0430 3-0036":0 [ENABLED,IMMUTABLE] pad2: Sink [fmt:SGRBG10_1X10/1920x1080 field:none colorspace:srgb quantization:full-range] pad3: Sink [fmt:SGRBG10_1X10/1920x1080 field:none colorspace:srgb quantization:full-range] pad4: Source [fmt:SGRBG10_1X10/2304x1296 field:none] -> "15050000.camsv":0 [ENABLED,IMMUTABLE] pad5: Source [fmt:SGRBG10_1X10/2304x1296 field:none] -> "15050800.camsv":0 [ENABLED,IMMUTABLE] pad6: Source [fmt:SGRBG10_1X10/1920x1080 field:none colorspace:srgb quantization:full-range] pad7: Source [fmt:SGRBG10_1X10/1920x1080 field:none colorspace:srgb quantization:full-range] - entity 10: 15050000.camsv (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb quantization:full-range] <- "15040000.seninf":4 [ENABLED,IMMUTABLE] pad1: Source [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb quantization:full-range] -> "15050000.camsv video stream":0 [ENABLED,IMMUTABLE] - entity 13: 15050000.camsv video stream (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video0 pad0: Sink <- "15050000.camsv":1 [ENABLED,IMMUTABLE] - entity 21: 15050800.camsv (2 pads, 2 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb quantization:full-range] <- "15040000.seninf":5 [ENABLED,IMMUTABLE] pad1: Source [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb quantization:full-range] -> "15050800.camsv video stream":0 [ENABLED,IMMUTABLE] - entity 24: 15050800.camsv video stream (1 pad, 1 link) type Node subtype V4L flags 0 device node name /dev/video1 pad0: Sink <- "15050800.camsv":1 [ENABLED,IMMUTABLE] - entity 32: ar0430 2-0036 (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev3 pad0: Source [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb crop.bounds:(0,0)/2316x1746 crop:(12,231)/2304x1296] -> "15040000.seninf":0 [ENABLED,IMMUTABLE] - entity 36: ar0430 3-0036 (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev4 pad0: Source [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb crop.bounds:(0,0)/2316x1746 crop:(12,231)/2304x1296] -> "15040000.seninf":1 [ENABLED,IMMUTABLE] You can also dump the media graph for a better view by ``media-ctl --print-dot``. The graph shows the connection between each camera subdevice. As the figure shows below, there are two AR0430 sensors connected to ``seninf`` which receives data from the two sensors and sends it to two ``camsv`` nodes. .. figure:: /_asset/sw_rity_app-dev_raw_camera_media-device.svg :align: center The media graph of camera subsystem .. _i350-evk-raw-capture-frames: Confirm Streaming ----------------- After setting the camera pipeline using ``media-ctl``, you can stream RAW frames in SGRBG10 packed format. The packed SGRBG10 format for ISP3.0's P1 is called ``MBgA`` (V4L2_PIX_FMT_MTISP_SGRBG10). Replace by the device node for camsv video stream. .. prompt:: bash # auto # v4l2-ctl -d --set-fmt-video=width=2304,height=1296,pixelformat=MBgA --stream-mmap --stream-count=10 --verbose VIDIOC_QUERYCAP: ok VIDIOC_G_FMT: ok VIDIOC_S_FMT: ok Format Video Capture Multiplanar: Width/Height : 2304/1296 Pixel Format : 'MBgA' (10-bit Bayer GRBG MTISP Packed) Field : None Number of planes : 1 Flags : Colorspace : sRGB Transfer Function : Default YCbCr/HSV Encoding: Default Quantization : Full Range Plane 0 : Bytes per Line : 2880 Size Image : 3732480 VIDIOC_REQBUFS returned 0 (Success) VIDIOC_QUERYBUF returned 0 (Success) VIDIOC_QUERYBUF returned 0 (Success) VIDIOC_QUERYBUF returned 0 (Success) VIDIOC_QUERYBUF returned 0 (Success) VIDIOC_QBUF returned 0 (Success) VIDIOC_QBUF returned 0 (Success) VIDIOC_QBUF returned 0 (Success) VIDIOC_QBUF returned 0 (Success) VIDIOC_STREAMON returned 0 (Success) cap dqbuf: 0 seq: 0 bytesused: 3732480 ts: 413.092186 (ts-monotonic, ts-src-eof) cap dqbuf: 1 seq: 1 bytesused: 3732480 ts: 413.125523 delta: 33.337 ms (ts-monotonic, ts-src-eof) cap dqbuf: 2 seq: 2 bytesused: 3732480 ts: 413.158861 delta: 33.338 ms (ts-monotonic, ts-src-eof) cap dqbuf: 3 seq: 3 bytesused: 3732480 ts: 413.192198 delta: 33.337 ms (ts-monotonic, ts-src-eof) cap dqbuf: 0 seq: 4 bytesused: 3732480 ts: 413.225535 delta: 33.337 ms fps: 30.00 (ts-monotonic, ts-src-eof) cap dqbuf: 1 seq: 5 bytesused: 3732480 ts: 413.258872 delta: 33.337 ms fps: 30.00 (ts-monotonic, ts-src-eof) cap dqbuf: 2 seq: 6 bytesused: 3732480 ts: 413.292210 delta: 33.338 ms fps: 30.00 (ts-monotonic, ts-src-eof) cap dqbuf: 3 seq: 7 bytesused: 3732480 ts: 413.325548 delta: 33.338 ms fps: 30.00 (ts-monotonic, ts-src-eof) cap dqbuf: 0 seq: 8 bytesused: 3732480 ts: 413.358885 delta: 33.337 ms fps: 30.00 (ts-monotonic, ts-src-eof) cap dqbuf: 1 seq: 9 bytesused: 3732480 ts: 413.392222 delta: 33.337 ms fps: 30.00 (ts-monotonic, ts-src-eof)