Genio 1200-demo

Introduction

The following figure shows the archtecture of Camera ISP on Genio 1200-demo.

../../../../_images/sw_yocto_app-dev_camera_arch.png

Camera ISP Architecture

The Camera ISP feature includes 4 major parts

  • Sensor Driver

  • Camera ISP OOT Driver

  • Camera ISP Middleware

  • V4L2 Adapter

Sensor Driver is the in-tree SONY IMX214 kernel driver with MediaTek extension patches.

Camera ISP OOT Driver is for MediaTek proprietary camera ISP hardware. They are maintained in the out-of-tree repository mtk-camisp-driver because they won’t be pushed to the mainline kernel.

Camera ISP Middleware is responsible for controlling the whole ISP devices, managing buffers and streams, and exporting Mediatek proprietary user interface. The middleware is released in binary mtk-camisp-prebuilts, and can only be accessed under an NDA contract.

V4L2 Adapter wraps Mediatek proprietary user interface into the V4L2 interface, which can be used by OSS tools, such as v4l2-utils, GStreamer, OpenCV, etc. The adapter exposes 3 V4L2 device nodes (Preview, Video, and Capture) for each hardware sensor.

The end users can operate the camera through the V4L2 devices exposed by the adapter. For the usage, please check the following sections.

Camera Sensor

The default camera sensor for Genio 1200-demo is SONY IMX214 CMOS image sensor. The camera socket on Genio 1200-demo is for SONY IMX258, so there’s an FPC converting IMX214 to IMX258.

../../../../_images/sw_yocto_app-dev_camera_imx214.png

SONY IMX214 and FPC (IMX214 to IMX258)

Connect The Camera To The EVK

There are 3 MIPI-CSI ports on Genio 1200-demo, which are CAM1, CAM2, and CAM3. In AIoT Yocto v22.2 release, it only supports CAM1. Please connect the camera sensor to CAM1 slot.

../../../../_images/sw_yocto_app-dev_camera_connect-imx214.png

Connect SONY IMX214 to Genio 1200-demo

Select Camera Device Tree Blob Overlay

The camera is inactive by default. The kernel has to load a specific device tree blob overlay to enable the camera. Please refer to bl33u-boot for more details. The dtbo for the camera ISP feature is camera-imx214-csi0.dtbo.

aiot-flash --list-dtbo
List of available DTBO:
        - camera-imx214-csi0.dtbo
aiot-flash --load-dtbo camera-imx214-csi0.dtbo

V4L2 Device Node

Camera ISP is a complicated feature that has a large number of device nodes. The camera ISP driver creates 125 video devices in total.

  • MTK-ISP-DIP-V4L2 has 68 devices

  • mtk-cam has 54 devices

  • mtk-v4l2-camera has 3 devices

MTK-ISP-DIP-V4L2 and mtk-cam are used by the camera ISP middleware. The user can just focus on mtk-v4l2-camera which is for streaming.

v4l2-ctl --list-devices
...
MTK-ISP-DIP-V4L2 (platform:15000000.imgsys_fw):
        (68 video devices)
        /dev/media0

mtk-cam (platform:16000000.camisp):
        (54 video devices)
        /dev/media2

mtk-v4l2-camera (platform:mtkcam0):
        /dev/video74
        /dev/video75
        /dev/video76

Launch Camera

The V4L2 nodes with the name mtk-v4l2-camera can be used for streaming. There are 3 V4L2 devices representing Preview, Video, and Capture streams. The user can find the devices by the following command:

v4l2-ctl --list-device
mtk-v4l2-camera (platform:mtkcam0):
        /dev/video74
        /dev/video75
        /dev/video76
cat /sys/class/video4linux/video{74,75,76}/name
mtk-v4l2-camera@0-Preview
mtk-v4l2-camera@0-Video
mtk-v4l2-camera@0-Capture

Important

The above v4l2 device node number is for example. The node number may change every time the system boots.

To quickly fetch the correct node and assign the environment variable:

declare -a video=(`v4l2-ctl --list-devices | grep mtk-v4l2-camera -A 3 | grep video | tr -d "\n"`)
printf "Preview Node\t= ${video[0]}\nVideo Node\t= ${video[1]}\nCapture Node\t= ${video[2]}\n"
Preview Node    = /dev/video74
Video Node      = /dev/video75
Capture Node    = /dev/video76

Preview and Video stream can deliver raw format image, and Capture stream only has JPEG output. The Capture stream is used to fetch the image that the user sees on the Preview or Video stream. The 3A configuration of the Capture stream relies on the Preview or Video stream. Therefore, before starting the Capture stream, make sure one of the Preview and Video streams is running. The following table shows the supported formats and resolutions for each stream.

../../../../_images/sw_yocto_app-dev_camera_format-resolution.png

Support formats and resolutions

To launch the camera, the user can use v4l2-ctl, gst-launch-1.0, or other V4L2 applications.

  • Use v4l2-ctl

    v4l2-ctl -d ${video[0]} --set-fmt-video=width=1920,height=1080,pixelformat=YUYV --stream-mmap=6 --stream-count=1 --stream-to=/tmp/preview.yuv --verbose

  • Use gst-launch-1.0

    gst-launch-1.0 v4l2src device=${video[0]} ! video/x-raw,width=640,height=480,format=YUY2 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${video[1]} ! video/x-raw,width=640,height=480,format=NV12 ! waylandsink sync=false &
gst-launch-1.0 v4l2src device=${video[2]} ! image/jpeg,width=640,height=480,format=JPEG ! jpegdec ! waylandsink sync=false &
    ../../../../_images/sw_yocto_app-dev_camera_show-on-weston.png

    Show Preview/Video/Capture streams on Weston

Moreover, the user can show 480P Preview content on Weston, record 10 seconds 720P Video to mp4 file, and Capture 1080P JPEG images by the following commands:

# Show 480P Preview on Weston
gst-launch-1.0 v4l2src device=${video[0]} ! video/x-raw,width=640,height=480,format=YUY2 ! waylandsink sync=false &
# Record 10 seconds 720P Video
gst-launch-1.0 v4l2src device=${video[1]} num-buffers=300 ! video/x-raw,width=1280,height=720,format=NV12 ! v4l2h264enc ! h264parse ! queue ! mp4mux ! filesink location=video.mp4 &
# Capture 1080P JPEG image
v4l2-ctl -d ${video[2]} --set-fmt-video=width=1920,height=1080,pixelformat=JPEG --stream-mmap=2 --stream-count=1 --stream-to=capture.jpg --verbose

Note

To use Mediatek video codec hardware, the user needs to load video.dtbo. For more details, please refer to Genio 1200-demo Video Codec.

Set Camera ISP Controls

For each stream, there are several controls can be set. The following table shows the available controls. There are 12 V4L2 standard controls and 1 vendor-specific control. The user can use the control to set 3A (Auto Focus, Auto Exposure, and Auto White Balance) and ISP pipeline feature (Brightness, Contrast, Saturation, and Sharpness).

../../../../_images/sw_yocto_app-dev_camera_controls.png

Available camera controls

The user can also use v4l2-ctl to list all controls by the following command:

v4l2-ctl --list-ctrls -d ${video[0]}

User Controls

                     brightness 0x00980900 (int)    : min=0 max=100 step=1 default=0 value=0 flags=slider
                       contrast 0x00980901 (int)    : min=0 max=10 step=1 default=0 value=0 flags=slider
                     saturation 0x00980902 (int)    : min=0 max=10 step=1 default=0 value=0 flags=slider
        white_balance_automatic 0x0098090c (bool)   : default=1 value=1
                       exposure 0x00980911 (int)    : min=-40 max=40 step=1 default=0 value=0
           power_line_frequency 0x00980918 (menu)   : min=0 max=3 default=3 value=3 (Auto)
      white_balance_temperature 0x0098091a (int)    : min=2700 max=6500 step=1 default=6500 value=6500
                      sharpness 0x0098091b (int)    : min=0 max=10 step=1 default=0 value=0 flags=slider
                            iso 0x009819a9 (int)    : min=100 max=6400 step=100 default=100 value=100

Camera Controls

                  auto_exposure 0x009a0901 (menu)   : min=0 max=1 default=0 value=0 (Auto Mode)
         exposure_time_absolute 0x009a0902 (int)    : min=100000 max=100000000 step=1 default=33000000 value=33000000
                 focus_absolute 0x009a090a (int)    : min=0 max=255 step=1 default=0 value=0
     focus_automatic_continuous 0x009a090c (bool)   : default=0 value=0

Note

Preview stream and Video stream share the same control setting, so the user modifies one and affects another. However, the control setting of Capture is separated, it doesn’t affect others.

Auto Focus

Note

SONY IMX214 sensor doesn’t support AF, so this is not yet available.

Auto Exposure

Auto Exposure can be set to AUTO mode and MANUAL mode.

In AUTO mode, the exposure is calculated automatically. The user can set exposure compensation to adjust the exposure level.

v4l2-ctl -d ${video[0]} -c auto_exposure=0                    # AUTO
v4l2-ctl -d ${video[0]} -c exposure=-40                       # MIN
v4l2-ctl -d ${video[0]} -c exposure=0                         # DEFAULT
v4l2-ctl -d ${video[0]} -c exposure=40                        # MAX

In MANUAL mode, the exposure is decided manually by the user. The user can set exposure time and ISO to adjust the exposure level.

v4l2-ctl -d ${video[0]} -c auto_exposure=1                    # MANUAL
v4l2-ctl -d ${video[0]} -c exposure_time_absolute=100000      # MIN
v4l2-ctl -d ${video[0]} -c exposure_time_absolute=100000000   # MAX
v4l2-ctl -d ${video[0]} -c iso=100                            # MIN
v4l2-ctl -d ${video[0]} -c iso=6400                           # MAX

Auto White Balance

Auto White Balance can be set to AUTO mode or MANUAL mode.

In AUTO mode, the white balance value is calculated automatically.

v4l2-ctl -d ${video[0]} -c white_balance_automatic=1          # AUTO

In MANUAL mode, the white balance value is decided manually by the user. The user can set the white balance temperature value.

v4l2-ctl -d ${video[0]} -c white_balance_automatic=0          # MANUAL
v4l2-ctl -d ${video[0]} -c white_balance_temperature=2800     # INCANDESCENT
v4l2-ctl -d ${video[0]} -c white_balance_temperature=3300     # WARM_FLUORESCENT
v4l2-ctl -d ${video[0]} -c white_balance_temperature=3800     # TWILIGHT
v4l2-ctl -d ${video[0]} -c white_balance_temperature=4400     # FLUORESCENT
v4l2-ctl -d ${video[0]} -c white_balance_temperature=5200     # DAYLIGHT
v4l2-ctl -d ${video[0]} -c white_balance_temperature=5800     # CLOUDY_DAYLIGHT
v4l2-ctl -d ${video[0]} -c white_balance_temperature=6300     # SHADE

Note

Although the temperature value is consecutive, the white balance change is discrete. Because the value is mapped to a white balance mode, such as INCANDESCENT, DAYLIGHT, etc.

ISP Pipeline Feature

There are 4 ISP pipeline feature controls (brightness, contrast, saturation, and sharpness). The user can set them and observe the image quality change.

v4l2-ctl -d ${video[0]} -c brightness=0                       # MIN
v4l2-ctl -d ${video[0]} -c brightness=100                     # MAX
v4l2-ctl -d ${video[0]} -c contrast=0                         # MIN
v4l2-ctl -d ${video[0]} -c contrast=10                        # MAX
v4l2-ctl -d ${video[0]} -c saturation=0                       # MIN
v4l2-ctl -d ${video[0]} -c saturation=10                      # MAX
v4l2-ctl -d ${video[0]} -c sharpness=0                        # MIN
v4l2-ctl -d ${video[0]} -c sharpness=10                       # MAX

Known Issues

Camera ISP Known Issues

ID

Description

AUTO00201126

[G12000-demo] ISP Capture: Random crash after open&close the preview/video/capture streams in stress environment

AUTO00205100

[G12000-demo] ISP Capture: In-consistency exposure result in continues stress capture environment

AUTO00205170

[G12000-demo] ISP Capture: mosaic or broken line on the captured JPEG image (randomly)