Camera Integration (V4L2 Sensor)

In this section, we will demonstrate how to integrate a new sensor into Genio 510/700-EVK.

1. Sensor Driver Enablement

The sensor driver should follow the Video 4 Linux 2 framework and function as a V4L2 sub-device. The implementation of the driver should include the necessary sub-device callbacks and the media entity initialization For more details, please refer to V4L2 sub-devices.

For the in-tree sensor driver, the kernel driver configuration can be set in meta-mediatek-bsp/recipes-kernel/linux/linux-mtk/mt8390-evk.cfg.

Set the config to y (built-in) or m (module)
CONFIG_VIDEO_AP1302=m

2. Device Tree Configuration

The device tree configuration is the most important part of the porting progress because it defines the sensor properties and the connection between the sensor and the SoC. Adding the device tree node for the sensor as an out-of-tree DTBO is advised because the SoC’s camera-related dependencies are maintained as out-of-tree modules. If you add the sensor-related device tree node directly into the in-tree Linux kernel, you may encounter errors indicating that the dependencies cannot be found. To begin, copying the configuration from meta-mediatek-bsp/recipes-kernel/dtbo/genio-700-evk/camera-ar0830-ap1302-csi0-std.dts is a good starting point. The device tree configuration heavily relies on the design of the camera daughterboard and the related circuit, so the user has to modify the dtbo according to the hardware design. For example, Genio 510/700-EVK has pre-defined the pin usages for power, clock, reset, control, and data transmission. For the pin definition of camera socket Genio 510/700-EVK, please refer to Camera Pin Definition.

Fragment 4 defines the pins for the power-related control. The setting includes the electrical characteristics, the GPIO pin used, etc.

fragment@4 {
    target-path = "/";
    __overlay__ {
        cam0_pdn: cam0-pdn-regulator {
            compatible = "regulator-fixed";
            regulator-name = "cam0_pdn";
            regulator-min-microvolt = <1800000>;
            regulator-max-microvolt = <1800000>;
            gpio = <&pio 18 0>;
            startup-delay-us = <18000>;
            enable-active-high;
        };

        cam0_dvdd_en: cam0-dvdd-en-regulator {
            compatible = "regulator-fixed";
            regulator-name = "cam0_dvdd_en";
            regulator-min-microvolt = <1800000>;
            regulator-max-microvolt = <1800000>;
            gpio = <&pio 79 0>;
            startup-delay-us = <18000>;
            enable-active-high;
            regulator-always-on;
            pinctrl-names = "default";
            pinctrl-0 = <&cam0_dvdd_en_pins>;
        };

        cam0_avdd_en: cam0-avdd-en-regulator {
            compatible = "regulator-fixed";
            regulator-name = "cam0_avdd_en";
            regulator-min-microvolt = <1800000>;
            regulator-max-microvolt = <1800000>;
            gpio = <&pio 80 0>;
            startup-delay-us = <18000>;
            enable-active-high;
            regulator-always-on;
            pinctrl-names = "default";
            pinctrl-0 = <&cam0_avdd_en_pins>;
        };

        cam1_avdd_en: cam1-avdd-en-regulator {
            compatible = "regulator-fixed";
            regulator-name = "cam1_avdd_en";
            regulator-min-microvolt = <1800000>;
            regulator-max-microvolt = <1800000>;
            gpio = <&pio 81 0>;
            startup-delay-us = <18000>;
            enable-active-high;
            regulator-always-on;
            pinctrl-names = "default";
            pinctrl-0 = <&cam1_avdd_en_pins>;
        };

        cam1_dvdd_en: cam1-dvdd-en-regulator {
            compatible = "regulator-fixed";
            regulator-name = "cam1_dvdd_en";
            regulator-min-microvolt = <1800000>;
            regulator-max-microvolt = <1800000>;
            gpio = <&pio 82 0>;
            startup-delay-us = <18000>;
            enable-active-high;
            regulator-always-on;
            pinctrl-names = "default";
            pinctrl-0 = <&cam1_dvdd_en_pins>;
        };

        cam_vcam_3v3_en: cam-vcam-3v3-en-regulator {
            compatible = "regulator-fixed";
            regulator-name = "cam_vcam_3v3_en";
            regulator-min-microvolt = <3300000>;
            regulator-max-microvolt = <3300000>;
            gpio = <&pio 91 GPIO_ACTIVE_HIGH>;
            enable-active-high;
            regulator-always-on;
            pinctrl-names = "default";
            pinctrl-0 = <&cam_vcam_3v3_en_pins>;
        };
    };
};

Fragment 6 defines the pin modes selected.

fragment@6 {
    target = <&pio>;
    __overlay__ {
        cam0_pins_default: cam0_pins_default {
            pins_cmd_dat {
                pinmux = <PINMUX_GPIO19__FUNC_B_GPIO19>,
                     <PINMUX_GPIO18__FUNC_B_GPIO18>,
                     <PINMUX_GPIO22__FUNC_O_CMMCLK0>;
                drive-strength = <MTK_DRIVE_8mA>;
            };
        };

        cam0_dvdd_en_pins: cam0-dvdd-en-pins {
            pins_cmd_dat {
                pinmux = <PINMUX_GPIO79__FUNC_B_GPIO79>;
            };
        };

        cam0_avdd_en_pins: cam0-avdd-en-pins {
            pins_cmd_dat {
                pinmux = <PINMUX_GPIO80__FUNC_B_GPIO80>;
            };
        };

        cam1_avdd_en_pins: cam1-avdd-en-pins {
            pins_cmd_dat {
                pinmux = <PINMUX_GPIO81__FUNC_B_GPIO81>;
            };
        };

        cam1_dvdd_en_pins: cam1-dvdd-en-pins {
            pins_cmd_dat {
                pinmux = <PINMUX_GPIO82__FUNC_B_GPIO82>;
            };
        };

        cam_vcam_3v3_en_pins: cam-vcam-3v3-en-pins {
            pins_cmd_dat {
                pinmux = <PINMUX_GPIO91__FUNC_B_GPIO91>;
            };
        };
    };
};

Fragment 7 is the definition of the sensor on a specific I2C bus and constitutes the main sensor configuration. The properties may vary depending on the sensor model and the design of the camera daughterboard. Please refer to the sensor’s DT binding document and the board layout for more details.

The key points of the configuration include:

  1. The CSI0 socket utilizes I2C bus 5, so the sensor node should be a sub-node of i2c5.

  2. The clock-frequency of i2c5 is set to 400KHz, which should also be supported by the sensor.

  3. The node ap1302@3c contains all the sensor-related properties.

    • compatible corresponds to a compatible string in the sensor driver.

    • reg represents the device address on the I2C bus.

    • pinctrl-0 indicates the function of pins used by the sensor. (e.g. Reset pin is GPIO).

    • clocks specifies the clock source used by the sensor.

    • assigned-clock-parents sets the clock rate for the sensor. The clock source is 192MHz and is divided by 4, 8, 16, or 32 as required. In this case, Onsemi AP1302 requires a 48MHz clock, so CLK_TOP_UNIVPLL_192M_D4 is given.

    • reset-gpios indicates the reset pin for the Onsemi AP1302 sensor. This property may vary depending on the driver implementation. Not all sensors require a reset pin.

    • power-supply, orientation, rotation, and sensors are sensor-specific properties as well as reset-gpios.

    • port is a crucial element indicating the connection between the sensor and the SoC.

      • remote-endpoint indicates the connected SENINF port.

      • data-lanes indicates the used MIPI PHY data lanes

  4. The node seninf_csi_port_0 in seninf_top represents the SENINF port used to receive the data.

    • compatible corresponds to the compatible string in the SENINF driver.

    • csi-port indicates the CSI port number to which the sensor is connected.

    • settle_delay_ck, dphy_settle_delay_dt, and hs_trail_parameter configures the MIPI CSI-2 timing parameter. For more details, please refer to 5. (Optional) MIPI CSI-2 Timing.

    • port is a crucial element indicating the connection between the sensor and the SoC.

      • remote-endpoint indicates the connected sensor port.

      • data-lanes indicates the used MIPI PHY data lanes

fragment@7 {
    target = <&i2c5>;
    __overlay__ {
        clock-frequency = <400000>;

        ap1302@3c {
            compatible = "onnn,ap1302";

            reg = <0x3c>;
            status = "okay";

            pinctrl-names = "default";
            pinctrl-0 = <&cam0_pins_default>;

            clocks = <&topckgen CLK_TOP_CAMTG>;
            assigned-clocks = <&topckgen CLK_TOP_CAMTG>;
            assigned-clock-parents = <&topckgen CLK_TOP_UNIVPLL_192M_D4>;

            reset-gpios = <&pio 19 GPIO_ACTIVE_LOW>;

            power-supply = <&cam0_pdn>;

            orientation = <0>;
            rotation = <0>;

            sensors {
                #address-cells = <1>;
                #size-cells = <0>;

                onnn,model = "onnn,ar0830";

                sensor@0 {
                    reg = <0>;
                };
            };

            port {
                sensor0_out: endpoint {
                    remote-endpoint = <&seninf_csi_port_0_in>;
                    data-lanes = <1 2 3 4>;
                    link-frequencies = /bits/ 64 <480000000>;
                };
            };
        };
    };
};

fragment@8 {
    target = <&seninf_top>;
    __overlay__ {
        seninf_csi_port_0: seninf_csi_port_0 {
            compatible = "mediatek,seninf";
            csi-port = "0";
            hs_trail_parameter = <0x08>;

            port {
                seninf_csi_port_0_in: endpoint {
                    remote-endpoint = <&sensor0_out>;
                    data-lanes = <1 2 3 4>;
                };
            };
        };
    };
};

3. Configure Media Topology

Once the driver and the device tree configurations have been correctly set, you can proceed to launch the camera. After flashing the image with the sensor DTBO, you can verify the correctness of the device topology by using the media-ctl tool. This tool provides information about the media device, entities, connections, and controls, allowing you to check whether the camera configuration is properly recognized and established.

media-ctl -d /dev/media0 --print-topology     # Print the device topology
media-ctl -d /dev/media0 --print-dot          # Print the device topology as a dot graph
../../../../_images/sw_yocto_app-dev_camera_g700_media-topology_single-ap1302.svg

Media Topology Graph of Single Onsemi AP1302 ISP and SoC

The provided diagram illustrates the entire camera system’s topology, including both the external sensor component and the System on a Chip (SoC). If all the needed sensors and SoC drivers are successfully probed, the topology should resemble the diagram. The topology also reflects the data flow originating from the sensor and ending at the video device.

A link with a dashed style signifies that the link is disabled and the two entities are not connected. If a link is enabled, it will be displayed in bold style, indicating an active connection.

media-ctl -d /dev/media0 -l "'seninf-0':1 -> 'mtk-cam camsv-0':0 [5]"     # Enable the link between seninf-0 pad 1 and mtk-cam camsv-0 pad 0
media-ctl -d /dev/media0 -l "'ap1302.5-003c':2 -> 'seninf-0':0 [1]"       # Enable the link between ap1302.5-003c pad 2 and seninf-0 pad 0

After enabling all the required links, the next step is to configure the format and the size for each pad. The format should be set on the source pad, and the sink pad will automatically adopt the same setting.

In the following example, the output from the Onsemi AP1302 sensor is in the UYVY format with a size of 1920x1080 pixels. Both the SENINF and the CAMSV should be set to the same format and size as the sensor’s.

media-ctl -d /dev/media0 -V "'ap1302.5-003c':2 [fmt:UYVY8_1X16/1920x1080 field:none]"     # Set UYVY/1920x1080 to ap1302.5-003c pad 2
media-ctl -d /dev/media0 -V "'seninf-0':1 [fmt:UYVY8_1X16/1920x1080 field:none]"          # Set UYVY/1920x1080 to seninf-0 pad 1
media-ctl -d /dev/media0 -V "'mtk-cam camsv-0':1 [fmt:UYVY8_1X16/1920x1080 field:none]"   # Set UYVY/1920x1080 to mtk-cam camsv-0 pad 1

4. Launch Camera And Check Correctness

To check whether the camera is working correctly, you can open the video device node and examine its behavior. Here’s a sample bash command that uses the v4l2-ctl tool to open the video device node and retrieve frames from the camera.

declare -a video=(`for i in {0..7}; do media-ctl -d /dev/media0 --entity "mtk-cam camsv-$i main-stream"; done | tr "\n" " "`)
echo ${video[*]}
/dev/video53 /dev/video54 /dev/video55 /dev/video56 /dev/video57 /dev/video58 /dev/video59 /dev/video60
v4l2-ctl -d ${video[0]} --set-fmt-video=width=1920,height=1080,pixelformat=UYVY --stream-mmap --verbose

The Sensor Interface (SENINF) is responsible for receiving the sensor data, decoding the MIPI CSI-2 packets, and dispatching the data to the CAMSV. Some image stream issues (e.g. frame dropping, jittering, noise, abnormal color patterns, etc.) could be indicative of underlying issues within the SENINF. Even if the image stream looks normal, there could exist a potential signal instability. Therefore, it’s necessary to check the SENINF status when the camera pipeline is on.

There is a debugfs node for users to inspect the status of the SENINF. If the SENINF receives the sensor data correctly, the status should be like the below sample.

cat /sys/devices/platform/soc/16010000.seninf_top/status

[seninf-0] port 0 intf 0 test 0 cphy 0 lanes 4      // PHY Port Information
source ap1302.5-003c flags 0x1                      // Sensor Source
csi2 irq_stat 0x00000324                            // IRQ Status
...
[0] vc 0x0 dt 0x1e mux 0 cam 2                      // Virtual Channel and Data Type
        ...
        width 3840 height 1080                      // Width and Height
        ...

The IRQ status is a crucial indicator when verifying the SENINF’s status. An IRQ status with the value 0x324 indicates that the SENINF has received the Start/End frame, validated the CRC (Cyclic Redundancy Check), and detected no ECC (Error Correction Code) errors. In Addition, it’s important to check that the width and the height displayed match the target sensor size. It means that the SENINF has correctly decoded the image data and size.

The width parameter represents the horizontal size of each line in the image, which is also known as the H size. The H size is the value obtained by multiplying the image width and the Bpp (Bytes per pixel). In the given example, the width 3840 is 1920 (image width) x 2 (Bytes per pixel required for the YUV422 format).

If the status node doesn’t show any information, it indicates that the camera pipeline is inactive. If the status node shows incorrect values, one potential cause is the MIPI timing mismatch. In this situation, please refer to 5. (Optional) MIPI CSI-2 Timing for detailed instructions and steps to set the MIPI timing parameters.

5. (Optional) MIPI CSI-2 Timing

MIPI CSI-2 is a high-speed protocol primarily used for real-time imaging and video applications. The MIPI D-PHY physical standard includes several specific timing parameters that govern the behavior of transmission to ensure reliability and correctness. For the detailed timing parameters, please refer to the MIPI CSI-2 specification.

The user can set two MIPI CSI-2 timing parameters, THS-TRAIL and THS-SETTLE, in the device tree to make the transmission stable. The value of the parameters is determined by the MIPI CSI-2 clock rate.

MIPI CSI-2 Timing Spec

Timing Parameter

Spec

Calculation

Property Name

THS-TRAIL

DPHY: max (8 x UI, 60 ns + 4 x UI ) < THS-TRAIL

THS-TRAIL = clock_period x (hs_trail_parameter + 1)

hs_trail_parameter

THS-SETTLE

DPHY: 85 ns + 6 x UI < THS-SETTLE < 145 ns + 10 x UI

THS-SETTLE = clock_period x (dphy_settle_delay_dt + 1)

dphy_settle_delay_dt

For example, the MIPI CSI-2 clock rate is 356MHz, so the corresponding timing parameters should satisfy the below spec.

  • clock_freq = 356 MHz

  • clock_period = 1 / clock_freq = 2.8 ns

  • UI = clock_period / 2 = 1.4 ns

  • THS-TRAIL

    • THS-TRAIL > max (8 x UI, 60 ns + 4 x UI )

    • THS-TRAIL > 65.6 ns

    • 2.8 ns x (hs_trail_parameter + 1) > 65.6 ns

    • hs_trail_parameter > 22.4

  • THS-SETTLE

    • 85 ns + 6 x UI < THS-SETTLE < 145 ns + 10 x UI

    • 93.4 ns < THS-SETTLE < 159 ns

    • 93.4 ns < 2.8 ns x (dphy_settle_delay_dt + 1) < 159 ns

    • 32.4 < dphy_settle_delay_dt < 55.8

After the calculation, fill in the dtbo properties with the values.

fragment@8 {
    target = <&seninf_top>;
    __overlay__ {
        seninf_csi_port_0: seninf_csi_port_0 {
            compatible = "mediatek,seninf";
            csi-port = "0";
            hs_trail_parameter = <0x25>;
            dphy_settle_delay_dt = <0x27>;

            port {
                ...
            };
        };
    };
};

In addition, it’s necessary to consider the timing of MIPI CSI Tx (i.e. the sensor). Sometimes the SoC cannot be compatible with the sensor even if the SoC timing parameters satisfy the MIPI specification.