Genio 350-EVK

Camera Package and Specification

The default camera package in the box is the combination of Onsemi AR0430 camera sensor and Onsemi AP1302 ISP. For more details about camera and ISP, please refer to:

In each EVK box, it contains:

Items

Qty

AP1302 ISP board

2

AR0430 IAS sensor module

2

miniSAS cable

2

Connect The Camera to The EVK

There are two MIPI-CSI ports on the EVK, which means you can connect up to two cameras.

../../../../_images/sw_rity_app-dev_camera_mipicsi-port.png

CAM+ISP mode

The hardware connection will be AR0430 ---> AP1302 ---> MIPI-CSI On EVK. The camera daughter board can be configured in CAM+ISP mode and CAM mode. On CAM+ISP mode, the images from the AR0430 camera module are passed to ISP before being received by the EVK. On CAM mode, the images from the AR0430 camera module are transferred to the EVK directly. By default, AIoT Yocto only supports CAM+ISP mode.

First, configure camera DTB in CAM+ISP mode. You will need 6 jumpers configured as below figure.

../../../../_images/sw_rity_app-dev_camera_cam-isp-mode.png

CAM+ISP mode

Second, connect AR0430 to the CAM+ISP slot.

../../../../_images/sw_rity_app-dev_camera_ar0430.png

Connect AR0430

Third, connect the camera DTB with a miniSAS cable.

../../../../_images/sw_rity_app-dev_camera_miniSAS.png

Connect miniSAS cable

After finishing the above steps, the hardware connection is complete.

Warning

Please make sure all the above settings, including the jumpers, camera slot, and miniSAS cable, are correct. Otherwise, the I2C connection between the sensor, ISP, and SoC will fail.

Select Camera Device Tree Blob Overlay

The camera is inactive by default. The kernel has to load a specific device tree blob overlay to enable the camera. Please refer to bl33u-boot for more details.

aiot-flash --list-dtbo
List of available DTBO:
    - camera-ap1302-ar0430-dual.dtbo
    - camera-ap1302-ar0430-single-csi0.dtbo
    - camera-ap1302-ar0430-single-csi1.dtbo
aiot-flash --load-dtbo camera-ap1302-ar0430-dual.dtbo # Dual camera on MIPI-CSI0 and MIPICSI1
aiot-flash --load-dtbo camera-ap1302-ar0430-single-csi0.dtbo # Single camera on MIPI-CSI0
aiot-flash --load-dtbo camera-ap1302-ar0430-single-csi1.dtbo # Single camera on MIPI-CSI1

Warning

You have to select the correct dtbo according to the camera connection you applied. For example, if you connect only one camera to MIPI-CSI0 slot, please use camera-ap1302-ar0430-single-csi0.dtbo. Otherwise, you may encounter a camera probing error.

Set Camera Properties Through media-ctl

Before using the camera, you should set the resolution and the pixel format through media-ctl. All of the example in this section uses the dual camera. Onsemi AP1302 ISP with Onsemi AR0430 sensor captures the image with 2316 * 1746, UYUV format. Therefore, you have to configure the format for MIPI-CSI0 and MIPI-CSI1 cameras.

The media device is the key to setting up the camera. You should find the media device with the module name, mtk-camsys-3.0. You can operate this media device to set the format, the resolution, and the enablement. The index of the media device may differ depending on the probing order of drivers and the number of devices. For example, the media device is /dev/media1.

ls -l /sys/bus/media/devices/ | grep seninf
lrwxrwxrwx 1 root root 0 Sep 20  2020 media1 -> ../../../devices/platform/soc/15040000.seninf/media1
cat /sys/bus/media/devices/media1/model
mtk-camsys-3.0
ls -l /dev/media1
crw-rw---- 1 root video 250, 1 Jan  1  2010 /dev/media1

To set the format of the MIPI-CSI0 camera:

media-ctl -d /dev/media1 -V "'ap1302.2-003d':2 [fmt:UYVY8_1X16/2316x1746]"
media-ctl -d /dev/media1 -V "'15040000.seninf':4 [fmt:UYVY8_1X16/2316x1746]"
media-ctl -d /dev/media1 -V "'15050000.camsv':1 [fmt:UYVY8_1X16/2316x1746]"

To set the format of the MIPI-CSI1 camera:

media-ctl -d /dev/media1 -V "'ap1302.3-003d':2 [fmt:UYVY8_1X16/2316x1746]"
media-ctl -d /dev/media1 -V "'15040000.seninf':5 [fmt:UYVY8_1X16/2316x1746]"
media-ctl -d /dev/media1 -V "'15050800.camsv':1 [fmt:UYVY8_1X16/2316x1746]"

Important

Before launching the camera, you have to use media-ctl to set the format and the resolution of the camera pipeline. Otherwise, the camera pipeline may fail due to the unknown format.

You can use media-ctl tool to dump the current state of the device topology. The name and the number in the above commands correspond to the entity and the pad. For more details, you can use media-ctl --help.

media-ctl -d /dev/media1 -p
Media controller API version 5.10.104

Media device information
------------------------
driver          mtk-seninf
model           mtk-camsys-3.0
serial
bus info        platform:15040000.seninf
hw revision     0x0
driver version  5.10.104

Device topology
- entity 1: 15040000.seninf (8 pads, 4 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:UYVY8_1X16/2316x1746 field:none colorspace:srgb]
                <- "ap1302.2-003d":2 [ENABLED,IMMUTABLE]
        pad1: Sink
                [fmt:UYVY8_1X16/2316x1746 field:none colorspace:srgb]
                <- "ap1302.3-003d":2 [ENABLED,IMMUTABLE]
        pad4: Source
                [fmt:UYVY8_1X16/2316x1746 field:none]
                -> "15050000.camsv":0 [ENABLED,IMMUTABLE]
        pad5: Source
                [fmt:UYVY8_1X16/2316x1746 field:none]
                -> "15050800.camsv":0 [ENABLED,IMMUTABLE]

- entity 10: 15050000.camsv (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:UYVY8_1X16/2316x1746 field:none colorspace:srgb]
                <- "15040000.seninf":4 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:UYVY8_1X16/2316x1746 field:none colorspace:srgb]
                -> "15050000.camsv video stream":0 [ENABLED,IMMUTABLE]

- entity 13: 15050000.camsv video stream (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video3
        pad0: Sink
                <- "15050000.camsv":1 [ENABLED,IMMUTABLE]

- entity 21: 15050800.camsv (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev2
        pad0: Sink
                [fmt:UYVY8_1X16/2316x1746 field:none colorspace:srgb]
                <- "15040000.seninf":5 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:UYVY8_1X16/2316x1746 field:none colorspace:srgb]
                -> "15050800.camsv video stream":0 [ENABLED,IMMUTABLE]

- entity 24: 15050800.camsv video stream (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video4
        pad0: Sink
                <- "15050800.camsv":1 [ENABLED,IMMUTABLE]

- entity 32: ap1302.2-003d (3 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev4
        pad0: Sink
                [fmt:SGRBG12_1X12/2316x1746 field:none colorspace:srgb
                 crop.bounds:(0,0)/2316x1746
                 crop:(0,0)/2316x1746]
                <- "ar0430 0":0 [ENABLED,IMMUTABLE]
        pad2: Source
                [fmt:UYVY8_1X16/2316x1746 field:none colorspace:srgb
                 crop.bounds:(0,0)/2316x1746
                 crop:(0,0)/2316x1746]
                -> "15040000.seninf":0 [ENABLED,IMMUTABLE]

- entity 36: ar0430 0 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
        pad0: Source
                [fmt:SGRBG12_1X12/2316x1746 field:none colorspace:srgb]
                -> "ap1302.2-003d":0 [ENABLED,IMMUTABLE]

- entity 42: ap1302.3-003d (3 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev6
        pad0: Sink
                [fmt:SGRBG12_1X12/2316x1746 field:none colorspace:srgb
                 crop.bounds:(0,0)/2316x1746
                 crop:(0,0)/2316x1746]
                <- "ar0430 0":0 [ENABLED,IMMUTABLE]
        pad2: Source
                [fmt:UYVY8_1X16/2316x1746 field:none colorspace:srgb
                 crop.bounds:(0,0)/2316x1746
                 crop:(0,0)/2316x1746]
                -> "15040000.seninf":1 [ENABLED,IMMUTABLE]

- entity 46: ar0430 0 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev5
        pad0: Source
                [fmt:SGRBG12_1X12/2316x1746 field:none colorspace:srgb]
                -> "ap1302.3-003d":0 [ENABLED,IMMUTABLE]

You can also dump the media graph for a better view by media-ctl --print-dot. The graph shows the connection between each camera subdevice. As the figure shows below, there are two AR0430 sensors connected to two AP1302 ISPs. seninf receives the data from two ISPs and sends it to two camsv respectively.

../../../../_images/sw_rity_app-dev_camera_media-device.svg

The media graph of camera subsystem

Multi-Camera Support

If you connect two camera daughter boards and one USB camera to the i350-EVK, you can launch three cameras simultaneously. After setting the camera format and finding out the video device node, use the following command to launch cameras.

gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=2316,height=1746,format=UYVY ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=400,height=300 ! fpsdisplaysink video-sink=waylandsink sync=false &
gst-launch-1.0 v4l2src device=/dev/video4 ! video/x-raw,width=2316,height=1746,format=UYVY ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=400,height=300 ! fpsdisplaysink video-sink=waylandsink sync=false &
gst-launch-1.0 v4l2src device=/dev/video5 io-mode=mmap ! v4l2convert ! video/x-raw,width=400,height=300 ! fpsdisplaysink video-sink=waylandsink sync=false &

You will see there are three cameras shown on Weston display.

../../../../_images/sw_rity_app-dev_camera_multi-camera.gif

Show multi-camera through GStreamer

Known Issues

  • The camera will freeze if the sensor is under a low ambient light during the boot process. Please make sure the sensor is under adequate ambient light during the boot process.

Troubleshooting

Low Framerate

If you found that some frames are dropped and the framerate is very low when using GStreamer to open the camera. You can use the GStreamer element fpsdisplaysink to calculate the framerate. For more details, please refer to fpsdisplaysink.

Here are the examples:

  • To calculate the framerate from v4l2src:

gst-launch-1.0 -v v4l2src device=/dev/video3 ! video/x-raw,width=2316,height=1746,format=UYVY ! fpsdisplaysink video-sink=fakesink sync=false
  • To calculate the framerate from v4l2convert:

gst-launch-1.0 -v v4l2src device=/dev/video3 ! video/x-raw,width=2316,height=1746,format=UYVY ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=400,height=300 ! fpsdisplaysink video-sink=fakesink sync=false
  • To calculate the framerate from waylandsink, and show it on the screen:

gst-launch-1.0 -v v4l2src device=/dev/video3 ! video/x-raw,width=2316,height=1746,format=UYVY ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=400,height=300 ! fpsdisplaysink video-sink=waylandsink sync=false

Some possible reasons cause low framerate:

  1. videoconvert is too slow to transform images on time.

Solution: Replace videoconvert with v4l2convert, which uses hardware converter MDP.

  1. waylandsink tries to synchronize the timestamp with the clock.

Solution: Add sync=false property to waylandsink.

  1. v4l2convert is slow when using mmap memory method.

Solution: Add output-io-mode=dmabuf-import property to v4l2convert.

Wrong Image Texture

If you found that the images from the pipeline are wrong, e.g. noise point, weird color, etc. You can dump the images from each pipeline element and check whether the output is correct or not. The GStreamer element filesink writes incoming data to a file in the local file system. For more details, please refer to filesink.

Here are the examples:

  • To dump the images from v4l2src:

gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=2316,height=1746,format=UYVY ! filesink location=test.bin
  • To dump the images from v4l2convert:

gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=2316,height=1746,format=UYVY ! v4l2convert output-io-mode=dmabuf-import ! video/x-raw,width=400,height=300 ! filesink location=test.bin

By dumping the image, you can find out which element acts incorrectly.

GStreamer Error

For other GStreamer pipeline issues, you can refer to GStreamer Debugging Tools. The page contains some useful debug methods provided by the GStreamer framework.

Camera Probe Failure

If you find that, both video and media devices are created, but using the command media-ctl to set properties fails.

ls -l /sys/bus/media/devices/ | grep seninf
lrwxrwxrwx 1 root root 0 Sep 20 10:43 media1 -> ../../../devices/platform/soc/15040000.seninf/media1
ls -l /sys/class/video4linux/ | grep seninf
lrwxrwxrwx 1 root root 0 Sep 20 10:43 video3 -> ../../devices/platform/soc/15040000.seninf/video4linux/video3
lrwxrwxrwx 1 root root 0 Sep 20 10:43 video4 -> ../../devices/platform/soc/15040000.seninf/video4linux/video4
media-ctl -d /dev/media1 -V "'ap1302.2-003d':2 [fmt:UYVY8_1X16/2316x1746]"
Unable to setup formats: No such file or directory (2)

Moreover, media-ctl -d /dev/media1 -p doesn’t show any format information. For example:

media-ctl -d /dev/media1 -p
Media controller API version 5.10.104

Media device information
------------------------
driver          mtk-seninf
model           mtk-camsys-3.0
serial
bus info        platform:15040000.seninf
hw revision     0x0
driver version  5.10.104

Device topology
- entity 1: 15040000.seninf (8 pads, 3 links)
            type V4L2 subdev subtype Unknown flags 0
        pad0: Sink
                <- "ap1302.2-003d":2 [ENABLED,IMMUTABLE]
        pad4: Source
                -> "15050000.camsv":0 [ENABLED,IMMUTABLE]
        pad5: Source
                -> "15050800.camsv":0 [ENABLED,IMMUTABLE]
...

This is due to the misconfiguration of hardware settings and device tree blob overlay. Please refer to the sections Connect The Camera to The EVK and Select Camera Device Tree Blob Overlay. Make sure you have the correct settings.