RAW Sensor (V4L2 Sensor)

Important

All the bash commands shown here are based on Genio 350-EVK.

Note

All command operations presented in this chapter are based on the IoT Yocto v24.1, Genio 350-EVK and Onsemi AR0430 sensor. You might get different operation results depending on the platform you use.

This chapter shows how to receive RAW (bayer) sensor data and directly dump it to the DRAM under V4L2 Sensor architecture on Genio 350-EVK.

Camera Daughter Board

The RAW camera DTB for Genio 350-EVK is AIoT CAM DTB D1V2. It contains an Onsemi AP1302 ISP and an Onsemi AR0430 Sensor. We will bypass the AP1302 ISP for RAW mode. The camera DTB connects with the Genio 350-EVK through a miniSAS cable. For more details about the ISP and the sensor, please refer to:

• Onsemi AR0430

Connect The Camera to the EVK

There are two MIPI-CSI ports on the EVK, which means you can connect up to two cameras.

Two MIPI CSI camera ports on the EVK

The hardware connection for RAW mode is AR0430 ---> MIPI-CSI On EVK. The camera daughter board can be configured in CAM+ISP mode and CAM mode. In CAM+ISP mode, the images from the AR0430 camera module are passed to AP1302 ISP before being received by the EVK. In CAM mode, the images from the AR0430 camera module are transferred to the EVK directly. By default, IoT Yocto supports both, CAM+ISP and CAM mode. In this section, we will capture RAW frames using the CAM mode.

First, configure camera DTB in CAM mode. You will need 6 jumpers and 1 camera sensor configured as the below figure. Make sure you have the correct jumper setting and that the camera sensor is properly connected to the CAM+ISP slot.

CAM mode

Second, connect the camera DTB with a miniSAS cable.

Connect miniSAS cable

After finishing the above steps, the hardware connection is complete.

Warning

Please make sure all the above settings, including the jumpers, camera slot, and miniSAS cable, are correct. Otherwise, the I2C connection between the sensor, ISP, and SoC will fail.

Select Camera Device Tree Blob Overlay

The camera is inactive by default. The kernel has to load a specific device tree blob overlay to enable the camera. Please refer to the bl33 (u-boot) section for more details.

RAW Camera DTBO for Genio 350-EVK

Platform	CSI0	CSI1	DTBO	Usage
Genio 350-EVK	AR0430	x	camera-ar0430-single-csi0.dtbo	Enable 4-lane AIoT CAM DTB w/ Onsemi AR0430 on CSI0
Genio 350-EVK	x	AR0430	camera-ar0430-single-csi1.dtbo	Enable 4-lane AIoT CAM DTB w/ Onsemi AR0430 on CSI1
Genio 350-EVK	AR0430	AR0430	camera-ar0430-dual.dtbo	Enable 4-lane AIoT CAM DTB w/ Onsemi AR0430 on CSI0 and CSI1

genio-flash --list-dtbo
List of available DTBO:
    - camera-ap1302-ar0430-dual.dtbo
    - camera-ap1302-ar0430-single-csi0.dtbo
    - camera-ap1302-ar0430-single-csi1.dtbo
    - camera-ar0430-dual.dtbo
    - camera-ar0430-single-csi0.dtbo
    - camera-ar0430-single-csi1.dtbo
genio-flash --load-dtbo camera-ar0430-dual.dtbo # Dual RAW camera on MIPI-CSI0 and MIPICSI1
genio-flash --load-dtbo camera-ar0430-single-csi0.dtbo # Single RAW camera on MIPI-CSI0
genio-flash --load-dtbo camera-ar0430-single-csi1.dtbo # Single RAW camera on MIPI-CSI1

Warning

Please select the DTBO according to the CSI slot to which the camera sensor is connected. For example, if the camera is connected to the CSI0 slot, please load the dtbo camera-ar0430-single-csi0.dtbo. Otherwise, the camera initialization will fail.

Supported Formats and Sizes

Platform	Sensor	Stream Type	Size	Frame rate	Format	MIPI Lanes
Genio 350-EVK	Onsemi AR0430	Preview	2304x1296	30	SGRBG10	4

Note

The supported format, resolution, and frame rate are related to the capability of the sensor and SoC.

Set Camera Properties through media-ctl

Before using the camera, you should set the resolution and the pixel format through media-ctl. All of the example in this section uses the dual camera. Onsemi AR0430 sensor captures the image with 2304 * 1296, SGRBG10 format. Therefore, you have to configure the format for MIPI-CSI0 and MIPI-CSI1 cameras.

The media device is the key to setting up the camera. You should find the media device with the module name, mtk-camsys-3.0. You can operate this media device to set the format, the resolution, and the enablement. The index of the media device may differ depending on the probing order of drivers and the number of devices. The command v4l2-ctl --list-devices can show the device information.

v4l2-ctl --list-devices
mtk-mdp (platform:14004000.mdp_rdma0):
        /dev/media1

mtk-camsys-3.0 (platform:15040000.seninf):
        /dev/media0

mtk-camsv-isp30 (platform:15050000.camsv):
        /dev/video0

mtk-camsv-isp30 (platform:15050800.camsv):
        /dev/video1

platform:mt8167 (platform:mt8167):
        /dev/video2
        /dev/video4

14004000.mdp_rdma0 (platform:mt8173):
        /dev/video3

For example, the media device for mtk-camsys-3.0 is /dev/media0. Please replace /dev/media<N> with the actual node on the platform. To set the format of the MIPI-CSI0 camera:

media-ctl -d /dev/media<N> -V "'ar0430 2-0036':0 [fmt:SGRBG10_1X10/2304x1296]"
media-ctl -d /dev/media<N> -V "'15040000.seninf':4 [fmt:SGRBG10_1X10/2304x1296]"
media-ctl -d /dev/media<N> -V "'15050000.camsv':1 [fmt:SGRBG10_1X10/2304x1296]"

To set the format of the MIPI-CSI1 camera:

media-ctl -d /dev/media<N> -V "'ar0430 3-0036':0 [fmt:SGRBG10_1X10/2304x1296]"
media-ctl -d /dev/media<N> -V "'15040000.seninf':5 [fmt:SGRBG10_1X10/2304x1296]"
media-ctl -d /dev/media<N> -V "'15050800.camsv':1 [fmt:SGRBG10_1X10/2304x1296]"

Important

Before launching the camera, you have to use media-ctl to set the format and the resolution of the camera pipeline. Otherwise, the camera pipeline may fail due to the unknown format.

You can use media-ctl tool to dump the current state of the device topology. The name and the number in the above commands correspond to the entity and the pad. For more details, you can use media-ctl --help.

media-ctl -d /dev/media<N> -p
Media controller API version 5.15.47

Media device information
------------------------
driver          mtk-seninf
model           mtk-camsys-3.0
serial
bus info        platform:15040000.seninf
hw revision     0x0
driver version  5.15.47

Device topology
- entity 1: 15040000.seninf (8 pads, 4 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb]
                <- "ar0430 2-0036":0 [ENABLED,IMMUTABLE]
        pad1: Sink
                [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb]
                <- "ar0430 3-0036":0 [ENABLED,IMMUTABLE]
        pad2: Sink
                [fmt:SGRBG10_1X10/1920x1080 field:none colorspace:srgb quantization:full-range]
        pad3: Sink
                [fmt:SGRBG10_1X10/1920x1080 field:none colorspace:srgb quantization:full-range]
        pad4: Source
                [fmt:SGRBG10_1X10/2304x1296 field:none]
                -> "15050000.camsv":0 [ENABLED,IMMUTABLE]
        pad5: Source
                [fmt:SGRBG10_1X10/2304x1296 field:none]
                -> "15050800.camsv":0 [ENABLED,IMMUTABLE]
        pad6: Source
                [fmt:SGRBG10_1X10/1920x1080 field:none colorspace:srgb quantization:full-range]
        pad7: Source
                [fmt:SGRBG10_1X10/1920x1080 field:none colorspace:srgb quantization:full-range]

- entity 10: 15050000.camsv (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb quantization:full-range]
                <- "15040000.seninf":4 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb quantization:full-range]
                -> "15050000.camsv video stream":0 [ENABLED,IMMUTABLE]

- entity 13: 15050000.camsv video stream (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video0
        pad0: Sink
                <- "15050000.camsv":1 [ENABLED,IMMUTABLE]

- entity 21: 15050800.camsv (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev2
        pad0: Sink
                [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb quantization:full-range]
                <- "15040000.seninf":5 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb quantization:full-range]
                -> "15050800.camsv video stream":0 [ENABLED,IMMUTABLE]

- entity 24: 15050800.camsv video stream (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
        pad0: Sink
                <- "15050800.camsv":1 [ENABLED,IMMUTABLE]

- entity 32: ar0430 2-0036 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
        pad0: Source
                [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb
                 crop.bounds:(0,0)/2316x1746
                 crop:(12,231)/2304x1296]
                -> "15040000.seninf":0 [ENABLED,IMMUTABLE]

- entity 36: ar0430 3-0036 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev4
        pad0: Source
                [fmt:SGRBG10_1X10/2304x1296 field:none colorspace:srgb
                 crop.bounds:(0,0)/2316x1746
                 crop:(12,231)/2304x1296]
                -> "15040000.seninf":1 [ENABLED,IMMUTABLE]

You can also dump the media graph for a better view by media-ctl --print-dot. The graph shows the connection between each camera subdevice. As the figure shows below, there are two AR0430 sensors connected to seninf which receives data from the two sensors and sends it to two camsv nodes.

The media graph of camera subsystem

Confirm Streaming

After setting the camera pipeline using media-ctl, you can stream RAW frames in SGRBG10 packed format. The packed SGRBG10 format for ISP3.0’s P1 is called MBgA (V4L2_PIX_FMT_MTISP_SGRBG10). Replace <N> by the device node for camsv video stream.

v4l2-ctl -d<N> --set-fmt-video=width=2304,height=1296,pixelformat=MBgA --stream-mmap --stream-count=10 --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture Multiplanar:
        Width/Height      : 2304/1296
        Pixel Format      : 'MBgA' (10-bit Bayer GRBG MTISP Packed)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : sRGB
        Transfer Function : Default
        YCbCr/HSV Encoding: Default
        Quantization      : Full Range
        Plane 0           :
           Bytes per Line : 2880
           Size Image     : 3732480
                VIDIOC_REQBUFS returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      0 bytesused: 3732480 ts: 413.092186 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      1 bytesused: 3732480 ts: 413.125523 delta: 33.337 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      2 bytesused: 3732480 ts: 413.158861 delta: 33.338 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      3 bytesused: 3732480 ts: 413.192198 delta: 33.337 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      4 bytesused: 3732480 ts: 413.225535 delta: 33.337 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      5 bytesused: 3732480 ts: 413.258872 delta: 33.337 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      6 bytesused: 3732480 ts: 413.292210 delta: 33.338 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      7 bytesused: 3732480 ts: 413.325548 delta: 33.338 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      8 bytesused: 3732480 ts: 413.358885 delta: 33.337 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      9 bytesused: 3732480 ts: 413.392222 delta: 33.337 ms fps: 30.00 (ts-monotonic, ts-src-eof)