.. spelling:word-list:: dphy dt hs .. include:: /keyword.rst ================================ Camera Integration (V4L2 Sensor) ================================ .. contents:: Sections :local: :depth: 1 .. toctree:: :hidden: In this section, we will demonstrate how to integrate a new sensor into |G1200-EVK|. 1. Sensor Driver Enablement ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The sensor driver should follow the Video 4 Linux 2 framework and function as a V4L2 sub-device. The implementation of the driver should include the necessary sub-device callbacks and the media entity initialization For more details, please refer to `V4L2 sub-devices `_. For the in-tree sensor driver, the kernel driver configuration can be set in ``meta-mediatek-bsp/recipes-kernel/linux/linux-mtk/mt8395-evk.cfg``. .. prompt:: bash # auto # Set the config to y (built-in) or m (module) CONFIG_VIDEO_AP1302=m 2. Device Tree Configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The device tree configuration is the most important part of the porting progress because it defines the sensor properties and the connection between the sensor and the SoC. Adding the device tree node for the sensor as an out-of-tree DTBO is advised because the SoC's camera-related dependencies are maintained as out-of-tree modules. If you add the sensor-related device tree node directly into the in-tree Linux kernel, you may encounter errors indicating that the dependencies cannot be found. To begin, copying the configuration from ``meta-mediatek-bsp/recipes-kernel/dtbo/genio-1200-evk/camera-ar0830-ap1302-csi0-std.dts`` is a good starting point. The device tree configuration heavily relies on the design of the camera daughterboard and the related circuit, so the user has to modify the `dtbo` file according to the hardware design. For example, |G1200-EVK| has pre-defined the pin usages for power, clock, reset, control, and data transmission. For the pin definition of camera socket |G1200-EVK|, please refer to :ref:`Camera Pin Definition `. Fragment 4 defines the pins for the power-related control. The setting includes the electrical characteristics, the GPIO pin used, etc. .. prompt:: fragment@4 { target-path = "/"; __overlay__ { cam0_pdn: cam0-pdn-regulator { compatible = "regulator-fixed"; regulator-name = "cam0_pdn"; regulator-min-microvolt = <1800000>;` regulator-max-microvolt = <1800000>; gpio = <&pio 54 0>; startup-delay-us = <18000>; enable-active-high; }; cam0_avdd_en: cam0-avdd-en-regulator { compatible = "regulator-fixed"; regulator-name = "cam0_avdd_en"; regulator-min-microvolt = <1800000>; regulator-max-microvolt = <1800000>; gpio = <&pio 56 0>; startup-delay-us = <18000>; enable-active-high; regulator-always-on; pinctrl-names = "default"; pinctrl-0 = <&cam0_avdd_en_pins>; }; cam1_dvdd_en: cam1-dvdd-en-regulator { compatible = "regulator-fixed"; regulator-name = "cam1_dvdd_en"; regulator-min-microvolt = <1800000>; regulator-max-microvolt = <1800000>; gpio = <&pio 58 0>; startup-delay-us = <18000>; enable-active-high; regulator-always-on; pinctrl-names = "default"; pinctrl-0 = <&cam1_dvdd_en_pins>; }; cam1_avdd_en: cam1-avdd-en-regulator { compatible = "regulator-fixed"; regulator-name = "cam1_avdd_en"; regulator-min-microvolt = <1800000>; regulator-max-microvolt = <1800000>; gpio = <&pio 59 0>; startup-delay-us = <18000>; enable-active-high; regulator-always-on; pinctrl-names = "default"; pinctrl-0 = <&cam1_avdd_en_pins>; }; cam0_dvdd_en: cam0-dvdd-en-regulator { compatible = "regulator-fixed"; regulator-name = "cam0_dvdd_en"; regulator-min-microvolt = <1800000>; regulator-max-microvolt = <1800000>; gpio = <&pio 60 0>; startup-delay-us = <18000>; enable-active-high; regulator-always-on; pinctrl-names = "default"; pinctrl-0 = <&cam0_dvdd_en_pins>; }; cam_vcam_3v3_en: cam-vcam-3v3-en-regulator { compatible = "regulator-fixed"; regulator-name = "cam_vcam_3v3_en"; regulator-min-microvolt = <1800000>; regulator-max-microvolt = <1800000>; vin-supply = <&mt6359_vcamio_ldo_reg>; enable-active-high; regulator-always-on; }; }; }; Fragment 5 defines the pin modes selected. .. prompt:: fragment@5 { target = <&pio>; __overlay__ { cam0_pins_default: cam0_pins_default { pins_cmd_dat { pinmux = , , ; drive-strength = ; }; }; cam0_avdd_en_pins: cam0-avdd-en-pins { pins_cmd_dat { pinmux = ; }; }; cam1_dvdd_en_pins: cam1-dvdd-en-pins { pins_cmd_dat { pinmux = ; }; }; cam1_avdd_en_pins: cam1-avdd-en-pins { pins_cmd_dat { pinmux = ; }; }; cam0_dvdd_en_pins: cam0-dvdd-en-pins { pins_cmd_dat { pinmux = ; }; }; }; }; Fragment 6 is the definition of the sensor on a specific I2C bus and constitutes the main sensor configuration. The properties may vary depending on the sensor model and the design of the camera daughterboard. Please refer to the sensor's DT binding document and the board layout for more details. The key points of the configuration include: #. The CSI0 socket utilizes I2C bus 0, so the sensor node should be a sub-node of ``i2c0``. #. The ``clock-frequency`` of i2c0 is set to 400KHz, which should also be supported by the sensor. #. The node ``ap1302@3c`` contains all the sensor-related properties. * ``compatible`` corresponds to a compatible string in the sensor driver. * ``reg`` represents the device address on the I2C bus. * ``pinctrl-0`` indicates the function of pins used by the sensor. (e.g. Reset pin is GPIO). * ``clocks`` specifies the clock source used by the sensor. * ``assigned-clock-parents`` sets the clock rate for the sensor. The clock source is 192MHz and is divided by 4, 8, 16, or 32 as required. In this case, Onsemi AP1302 requires a 48MHz clock, so CLK_TOP_UNIVPLL_192M_D4 is given. * ``reset-gpios`` indicates the reset pin for the Onsemi AP1302 sensor. This property may vary depending on the driver implementation. Not all sensors require a reset pin. * ``power-supply``, ``orientation``, ``rotation``, and ``sensors`` are sensor-specific properties as well as ``reset-gpios``. * ``port`` is a crucial element indicating the connection between the sensor and the SoC. * ``remote-endpoint`` indicates the connected SENINF port. * ``data-lanes`` indicates the used MIPI PHY data lanes #. The node ``seninf_csi_port_0`` in ``seninf_top`` represents the SENINF port used to receive the data. * ``compatible`` corresponds to the compatible string in the SENINF driver. * ``csi-port`` indicates the CSI port number to which the sensor is connected. * ``settle_delay_ck``, ``dphy_settle_delay_dt``, and ``hs_trail_parameter`` configures the MIPI CSI-2 timing parameter. For more details, please refer to `5. (Optional) MIPI CSI-2 Timing`_. * ``port`` is a crucial element indicating the connection between the sensor and the SoC. * ``remote-endpoint`` indicates the connected sensor port. * ``data-lanes`` indicates the used MIPI PHY data lanes .. prompt:: fragment@6 { target = <&i2c0>; __overlay__ { clock-frequency = <400000>; ap1302@3c { compatible = "onnn,ap1302"; reg = <0x3c>; status = "okay"; pinctrl-names = "default"; pinctrl-0 = <&cam0_pins_default>; clocks = <&topckgen CLK_TOP_CAMTG>; assigned-clocks = <&topckgen CLK_TOP_CAMTG>; assigned-clock-parents = <&topckgen CLK_TOP_UNIVPLL_192M_D4>; reset-gpios = <&pio 55 GPIO_ACTIVE_LOW>; power-supply = <&cam0_pdn>; orientation = <0>; rotation = <0>; sensors { #address-cells = <1>; #size-cells = <0>; onnn,model = "onnn,ar0830"; sensor@0 { reg = <0>; }; }; port { sensor0_out: endpoint { remote-endpoint = <&seninf_csi_port_0_in>; data-lanes = <1 2 3 4>; link-frequencies = /bits/ 64 <480000000>; }; }; }; }; }; fragment@7 { target = <&seninf_top>; __overlay__ { seninf_csi_port_0: seninf_csi_port_0 { compatible = "mediatek,seninf"; csi-port = "0"; hs_trail_parameter = <0x08>; port { seninf_csi_port_0_in: endpoint { remote-endpoint = <&sensor0_out>; data-lanes = <1 2 3 4>; }; }; }; }; }; The fragments 8, 9, 10, and 11 are to disable the unnecessary modules in ``isp70.dtsi``. .. prompt:: fragment@8 { target = <&imgsys_fw>; __overlay__ { status = "disabled"; }; }; fragment@9 { target = <&hcp>; __overlay__ { status = "disabled"; }; }; fragment@10 { target-path = "/soc/imgsys_larb"; __overlay__ { status = "disabled"; }; }; fragment@11 { target = <&ipesys_me>; __overlay__ { status = "disabled"; }; }; 3. Configure Media Topology ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Once the driver and the device tree configurations have been correctly set, you can proceed to launch the camera. After flashing the image with the sensor DTBO, you can verify the correctness of the device topology by using the ``media-ctl`` tool. This tool provides information about the media device, entities, connections, and controls, allowing you to check whether the camera configuration is properly recognized and established. .. prompt:: bash # audio media-ctl -d /dev/media0 --print-topology # Print the device topology media-ctl -d /dev/media0 --print-dot # Print the device topology as a dot graph .. figure:: /_asset/sw_yocto_app-dev_camera_g1200_media-topology_single-ap1302.svg :align: center :scale: 50% Media Topology Graph of Single Onsemi AP1302 ISP and SoC The provided diagram illustrates the entire camera system's topology, including both the external sensor component and the System on a Chip (SoC). If all the needed sensors and SoC drivers are successfully probed, the topology should resemble the diagram. The topology also reflects the data flow originating from the sensor and ending at the video device. A link with a dashed style signifies that the link is disabled and the two entities are not connected. If a link is enabled, it will be displayed in bold style, indicating an active connection. .. prompt:: bash # auto # media-ctl -d /dev/media0 -l "'seninf-0':1 -> 'mtk-cam camsv-0':0 [5]" # Enable the link between seninf-0 pad 1 and mtk-cam camsv-0 pad 0 # media-ctl -d /dev/media0 -l "'ap1302.0-003c':2 -> 'seninf-0':0 [1]" # Enable the link between ap1302.0-003c pad 2 and seninf-0 pad 0 After enabling all the required links, the next step is to configure the format and the size for each pad. The format should be set on the source pad, and the sink pad will automatically adopt the same setting. In the following example, the output from the Onsemi AP1302 sensor is in the UYVY format with a size of 1920x1080 pixels. Both the SENINF and the CAMSV should be set to the same format and size as the sensor's. .. prompt:: bash # auto # media-ctl -d /dev/media0 -V "'ap1302.0-003c':2 [fmt:UYVY8_1X16/1920x1080 field:none]" # Set UYVY/1920x1080 to ap1302.0-003c pad 2 # media-ctl -d /dev/media0 -V "'seninf-0':1 [fmt:UYVY8_1X16/1920x1080 field:none]" # Set UYVY/1920x1080 to seninf-0 pad 1 # media-ctl -d /dev/media0 -V "'mtk-cam camsv-0':1 [fmt:UYVY8_1X16/1920x1080 field:none]" # Set UYVY/1920x1080 to mtk-cam camsv-0 pad 1 4. Launch Camera And Check Correctness ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To check whether the camera is working correctly, you can open the video device node and examine its behavior. Here's a sample bash command that uses the ``v4l2-ctl`` tool to open the video device node and retrieve frames from the camera. .. prompt:: bash # auto # declare -a video=(`for i in {0..5}; do media-ctl -d /dev/media0 --entity "mtk-cam camsv-$i main-stream"; done | tr "\n" " "`) # echo ${video[*]} /dev/video53 /dev/video54 /dev/video55 /dev/video56 /dev/video57 /dev/video58 # v4l2-ctl -d ${video[0]} --set-fmt-video=width=1920,height=1080,pixelformat=UYVY --stream-mmap --verbose The Sensor Interface (SENINF) is responsible for receiving the sensor data, decoding the MIPI CSI-2 packets, and dispatching the data to the CAMSV. Some image stream issues (e.g. frame dropping, jittering, noise, abnormal color patterns, etc.) could be indicative of underlying issues within the SENINF. Even if the image stream looks normal, there could exist a potential signal instability. Therefore, it's necessary to check the SENINF status when the camera pipeline is on. There is a `debugfs` node for users to inspect the status of the SENINF. If the SENINF receives the sensor data correctly, the status should be like the below sample. .. prompt:: bash # auto # cat /sys/devices/platform/soc/16007000.seninf_top/status [seninf-0] port 0 intf 0 test 0 cphy 0 lanes 4 // PHY Port Information source ap1302.0-003c flags 0x1 // Sensor Source csi2 irq_stat 0x00000324 // IRQ Status ... [0] vc 0x0 dt 0x1e mux 0 cam 2 // Virtual Channel and Data Type ... width 3840 height 1080 // Width and Height ... The IRQ status is a crucial indicator when verifying the SENINF's status. An IRQ status with the value 0x324 indicates that the SENINF has received the Start/End frame, validated the CRC (Cyclic Redundancy Check), and detected no ECC (Error Correction Code) errors. In Addition, it's important to check that the width and the height displayed match the target sensor size. It means that the SENINF has correctly decoded the image data and size. The width parameter represents the horizontal size of each line in the image, which is also known as the H size. The H size is the value obtained by multiplying the image width and the Bpp (Bytes per pixel). In the given example, the width 3840 is 1920 (image width) x 2 (Bytes per pixel required for the YUV422 format). If the status node doesn't show any information, it indicates that the camera pipeline is inactive. If the status node shows incorrect values, one potential cause is the MIPI timing mismatch. In this situation, please refer to `5. (Optional) MIPI CSI-2 Timing`_ for detailed instructions and steps to set the MIPI timing parameters. 5. (Optional) MIPI CSI-2 Timing ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ MIPI CSI-2 is a high-speed protocol primarily used for real-time imaging and video applications. The MIPI D-PHY physical standard includes several specific timing parameters that govern the behavior of transmission to ensure reliability and correctness. For the detailed timing parameters, please refer to the MIPI CSI-2 specification. The user can set two MIPI CSI-2 timing parameters, THS-TRAIL and THS-SETTLE, in the device tree to make the transmission stable. The value of the parameters is determined by the MIPI CSI-2 clock rate. .. table:: MIPI CSI-2 Timing Spec ================ =========================================================== ====================================================== ==================== Timing Parameter Spec Calculation Property Name ================ =========================================================== ====================================================== ==================== THS-TRAIL DPHY: max (8 x UI, 60 ns + 4 x UI ) < THS-TRAIL THS-TRAIL = clock_period x (hs_trail_parameter + 1) hs_trail_parameter THS-SETTLE DPHY: 85 ns + 6 x UI < THS-SETTLE < 145 ns + 10 x UI THS-SETTLE = clock_period x (dphy_settle_delay_dt + 1) dphy_settle_delay_dt ================ =========================================================== ====================================================== ==================== For example, the MIPI CSI-2 clock rate is 356MHz, so the corresponding timing parameters should satisfy the below spec. - clock_freq = 356 MHz - clock_period = 1 / clock_freq = 2.8 ns - UI = clock_period / 2 = 1.4 ns - THS-TRAIL - THS-TRAIL > max (8 x UI, 60 ns + 4 x UI ) - THS-TRAIL > 65.6 ns - 2.8 ns x (hs_trail_parameter + 1) > 65.6 ns - hs_trail_parameter > 22.4 - THS-SETTLE - 85 ns + 6 x UI < THS-SETTLE < 145 ns + 10 x UI - 93.4 ns < THS-SETTLE < 159 ns - 93.4 ns < 2.8 ns x (dphy_settle_delay_dt + 1) < 159 ns - 32.4 < dphy_settle_delay_dt < 55.8 After the calculation, fill in the properties in the DT overlay with the values: .. code:: fragment@7 { target = <&seninf_top>; __overlay__ { seninf_csi_port_0: seninf_csi_port_0 { compatible = "mediatek,seninf"; csi-port = "0"; hs_trail_parameter = <0x25>; dphy_settle_delay_dt = <0x27>; port { ... }; }; }; }; In addition, it's necessary to consider the timing of MIPI CSI Tx (i.e. the sensor). Sometimes the SoC cannot be compatible with the sensor even if the SoC timing parameters satisfy the MIPI specification.