Camera Integration (V4L2 Sensor)
In this section, we will demonstrate how to integrate a new sensor into Genio 510/700-EVK.
1. Sensor Driver Enablement
The sensor driver should follow the Video 4 Linux 2 framework and function as a V4L2 sub-device. The implementation of the driver should include the necessary sub-device callbacks and the media entity initialization For more details, please refer to V4L2 sub-devices.
For the in-tree sensor driver, the kernel driver configuration can be set in meta-mediatek-bsp/recipes-kernel/linux/linux-mtk/mt8390-evk.cfg.
Set the config to y (built-in) or m (module)
CONFIG_VIDEO_AP1302=m
2. Device Tree Configuration
The device tree configuration is the most important part of the porting progress because it defines the sensor properties and the connection between the sensor and the SoC.
Adding the device tree node for the sensor as an out-of-tree DTBO is advised because the SoC’s camera-related dependencies are maintained as out-of-tree modules.
If you add the sensor-related device tree node directly into the in-tree Linux kernel, you may encounter errors indicating that the dependencies cannot be found.
To begin, copying the configuration from meta-mediatek-bsp/recipes-kernel/dtbo/genio-700-evk/camera-ar0830-ap1302-csi0-std.dts is a good starting point.
The device tree configuration heavily relies on the design of the camera daughterboard and the related circuit, so the user has to modify the dtbo file according to the hardware design.
For example, Genio 510/700-EVK has pre-defined the pin usages for power, clock, reset, control, and data transmission.
For the pin definition of camera socket Genio 510/700-EVK, please refer to Camera Pin Definition.
Fragment 4 defines the pins for the power-related control. The setting includes the electrical characteristics, the GPIO pin used, etc.
fragment@4 {
target-path = "/";
__overlay__ {
cam0_pdn: cam0-pdn-regulator {
compatible = "regulator-fixed";
regulator-name = "cam0_pdn";
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
gpio = <&pio 18 0>;
startup-delay-us = <18000>;
enable-active-high;
};
cam0_dvdd_en: cam0-dvdd-en-regulator {
compatible = "regulator-fixed";
regulator-name = "cam0_dvdd_en";
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
gpio = <&pio 79 0>;
startup-delay-us = <18000>;
enable-active-high;
regulator-always-on;
pinctrl-names = "default";
pinctrl-0 = <&cam0_dvdd_en_pins>;
};
cam0_avdd_en: cam0-avdd-en-regulator {
compatible = "regulator-fixed";
regulator-name = "cam0_avdd_en";
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
gpio = <&pio 80 0>;
startup-delay-us = <18000>;
enable-active-high;
regulator-always-on;
pinctrl-names = "default";
pinctrl-0 = <&cam0_avdd_en_pins>;
};
cam1_avdd_en: cam1-avdd-en-regulator {
compatible = "regulator-fixed";
regulator-name = "cam1_avdd_en";
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
gpio = <&pio 81 0>;
startup-delay-us = <18000>;
enable-active-high;
regulator-always-on;
pinctrl-names = "default";
pinctrl-0 = <&cam1_avdd_en_pins>;
};
cam1_dvdd_en: cam1-dvdd-en-regulator {
compatible = "regulator-fixed";
regulator-name = "cam1_dvdd_en";
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
gpio = <&pio 82 0>;
startup-delay-us = <18000>;
enable-active-high;
regulator-always-on;
pinctrl-names = "default";
pinctrl-0 = <&cam1_dvdd_en_pins>;
};
cam_vcam_3v3_en: cam-vcam-3v3-en-regulator {
compatible = "regulator-fixed";
regulator-name = "cam_vcam_3v3_en";
regulator-min-microvolt = <3300000>;
regulator-max-microvolt = <3300000>;
gpio = <&pio 91 GPIO_ACTIVE_HIGH>;
enable-active-high;
regulator-always-on;
pinctrl-names = "default";
pinctrl-0 = <&cam_vcam_3v3_en_pins>;
};
};
};
Fragment 6 defines the pin modes selected.
fragment@6 {
target = <&pio>;
__overlay__ {
cam0_pins_default: cam0_pins_default {
pins_cmd_dat {
pinmux = <PINMUX_GPIO19__FUNC_B_GPIO19>,
<PINMUX_GPIO18__FUNC_B_GPIO18>,
<PINMUX_GPIO22__FUNC_O_CMMCLK0>;
drive-strength = <MTK_DRIVE_8mA>;
};
};
cam0_dvdd_en_pins: cam0-dvdd-en-pins {
pins_cmd_dat {
pinmux = <PINMUX_GPIO79__FUNC_B_GPIO79>;
};
};
cam0_avdd_en_pins: cam0-avdd-en-pins {
pins_cmd_dat {
pinmux = <PINMUX_GPIO80__FUNC_B_GPIO80>;
};
};
cam1_avdd_en_pins: cam1-avdd-en-pins {
pins_cmd_dat {
pinmux = <PINMUX_GPIO81__FUNC_B_GPIO81>;
};
};
cam1_dvdd_en_pins: cam1-dvdd-en-pins {
pins_cmd_dat {
pinmux = <PINMUX_GPIO82__FUNC_B_GPIO82>;
};
};
cam_vcam_3v3_en_pins: cam-vcam-3v3-en-pins {
pins_cmd_dat {
pinmux = <PINMUX_GPIO91__FUNC_B_GPIO91>;
};
};
};
};
Fragment 7 is the definition of the sensor on a specific I2C bus and constitutes the main sensor configuration. The properties may vary depending on the sensor model and the design of the camera daughterboard. Please refer to the sensor’s DT binding document and the board layout for more details.
The key points of the configuration include:
The CSI0 socket utilizes I2C bus 5, so the sensor node should be a sub-node of
i2c5.The
clock-frequencyof i2c5 is set to 400KHz, which should also be supported by the sensor.The node
ap1302@3ccontains all the sensor-related properties.compatiblecorresponds to a compatible string in the sensor driver.regrepresents the device address on the I2C bus.pinctrl-0indicates the function of pins used by the sensor. (e.g. Reset pin is GPIO).clocksspecifies the clock source used by the sensor.assigned-clock-parentssets the clock rate for the sensor. The clock source is 192MHz and is divided by 4, 8, 16, or 32 as required. In this case, Onsemi AP1302 requires a 48MHz clock, so CLK_TOP_UNIVPLL_192M_D4 is given.reset-gpiosindicates the reset pin for the Onsemi AP1302 sensor. This property may vary depending on the driver implementation. Not all sensors require a reset pin.power-supply,orientation,rotation, andsensorsare sensor-specific properties as well asreset-gpios.portis a crucial element indicating the connection between the sensor and the SoC.remote-endpointindicates the connected SENINF port.data-lanesindicates the used MIPI PHY data lanes
The node
seninf_csi_port_0inseninf_toprepresents the SENINF port used to receive the data.compatiblecorresponds to the compatible string in the SENINF driver.csi-portindicates the CSI port number to which the sensor is connected.settle_delay_ck,dphy_settle_delay_dt, andhs_trail_parameterconfigures the MIPI CSI-2 timing parameter. For more details, please refer to 5. (Optional) MIPI CSI-2 Timing.portis a crucial element indicating the connection between the sensor and the SoC.remote-endpointindicates the connected sensor port.data-lanesindicates the used MIPI PHY data lanes
fragment@7 {
target = <&i2c5>;
__overlay__ {
clock-frequency = <400000>;
ap1302@3c {
compatible = "onnn,ap1302";
reg = <0x3c>;
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <&cam0_pins_default>;
clocks = <&topckgen CLK_TOP_CAMTG>;
assigned-clocks = <&topckgen CLK_TOP_CAMTG>;
assigned-clock-parents = <&topckgen CLK_TOP_UNIVPLL_192M_D4>;
reset-gpios = <&pio 19 GPIO_ACTIVE_LOW>;
power-supply = <&cam0_pdn>;
orientation = <0>;
rotation = <0>;
sensors {
#address-cells = <1>;
#size-cells = <0>;
onnn,model = "onnn,ar0830";
sensor@0 {
reg = <0>;
};
};
port {
sensor0_out: endpoint {
remote-endpoint = <&seninf_csi_port_0_in>;
data-lanes = <1 2 3 4>;
link-frequencies = /bits/ 64 <480000000>;
};
};
};
};
};
fragment@8 {
target = <&seninf_top>;
__overlay__ {
seninf_csi_port_0: seninf_csi_port_0 {
compatible = "mediatek,seninf";
csi-port = "0";
hs_trail_parameter = <0x08>;
port {
seninf_csi_port_0_in: endpoint {
remote-endpoint = <&sensor0_out>;
data-lanes = <1 2 3 4>;
};
};
};
};
};
3. Configure Media Topology
Once the driver and the device tree configurations have been correctly set, you can proceed to launch the camera.
After flashing the image with the sensor DTBO, you can verify the correctness of the device topology by using the media-ctl tool.
This tool provides information about the media device, entities, connections, and controls, allowing you to check whether the camera configuration is properly recognized and established.
media-ctl -d /dev/media0 --print-topology # Print the device topology
media-ctl -d /dev/media0 --print-dot # Print the device topology as a dot graph
Media Topology Graph of Single Onsemi AP1302 ISP and SoC
The provided diagram illustrates the entire camera system’s topology, including both the external sensor component and the System on a Chip (SoC). If all the needed sensors and SoC drivers are successfully probed, the topology should resemble the diagram. The topology also reflects the data flow originating from the sensor and ending at the video device.
A link with a dashed style signifies that the link is disabled and the two entities are not connected. If a link is enabled, it will be displayed in bold style, indicating an active connection.
media-ctl -d /dev/media0 -l "'seninf-0':1 -> 'mtk-cam camsv-0':0 [5]" # Enable the link between seninf-0 pad 1 and mtk-cam camsv-0 pad 0
media-ctl -d /dev/media0 -l "'ap1302.5-003c':2 -> 'seninf-0':0 [1]" # Enable the link between ap1302.5-003c pad 2 and seninf-0 pad 0
After enabling all the required links, the next step is to configure the format and the size for each pad. The format should be set on the source pad, and the sink pad will automatically adopt the same setting.
In the following example, the output from the Onsemi AP1302 sensor is in the UYVY format with a size of 1920x1080 pixels. Both the SENINF and the CAMSV should be set to the same format and size as the sensor’s.
media-ctl -d /dev/media0 -V "'ap1302.5-003c':2 [fmt:UYVY8_1X16/1920x1080 field:none]" # Set UYVY/1920x1080 to ap1302.5-003c pad 2
media-ctl -d /dev/media0 -V "'seninf-0':1 [fmt:UYVY8_1X16/1920x1080 field:none]" # Set UYVY/1920x1080 to seninf-0 pad 1
media-ctl -d /dev/media0 -V "'mtk-cam camsv-0':1 [fmt:UYVY8_1X16/1920x1080 field:none]" # Set UYVY/1920x1080 to mtk-cam camsv-0 pad 1
4. Launch Camera And Check Correctness
To check whether the camera is working correctly, you can open the video device node and examine its behavior.
Here’s a sample bash command that uses the v4l2-ctl tool to open the video device node and retrieve frames from the camera.
declare -a video=(`for i in {0..7}; do media-ctl -d /dev/media0 --entity "mtk-cam camsv-$i main-stream"; done | tr "\n" " "`)
echo ${video[*]}
/dev/video53 /dev/video54 /dev/video55 /dev/video56 /dev/video57 /dev/video58 /dev/video59 /dev/video60
v4l2-ctl -d ${video[0]} --set-fmt-video=width=1920,height=1080,pixelformat=UYVY --stream-mmap --verbose
The Sensor Interface (SENINF) is responsible for receiving the sensor data, decoding the MIPI CSI-2 packets, and dispatching the data to the CAMSV. Some image stream issues (e.g. frame dropping, jittering, noise, abnormal color patterns, etc.) could be indicative of underlying issues within the SENINF. Even if the image stream looks normal, there could exist a potential signal instability. Therefore, it’s necessary to check the SENINF status when the camera pipeline is on.
There is a debugfs node for users to inspect the status of the SENINF. If the SENINF receives the sensor data correctly, the status should be like the below sample.
cat /sys/devices/platform/soc/16010000.seninf_top/status
[seninf-0] port 0 intf 0 test 0 cphy 0 lanes 4 // PHY Port Information
source ap1302.5-003c flags 0x1 // Sensor Source
csi2 irq_stat 0x00000324 // IRQ Status
...
[0] vc 0x0 dt 0x1e mux 0 cam 2 // Virtual Channel and Data Type
...
width 3840 height 1080 // Width and Height
...
The IRQ status is a crucial indicator when verifying the SENINF’s status. An IRQ status with the value 0x324 indicates that the SENINF has received the Start/End frame, validated the CRC (Cyclic Redundancy Check), and detected no ECC (Error Correction Code) errors. In Addition, it’s important to check that the width and the height displayed match the target sensor size. It means that the SENINF has correctly decoded the image data and size.
The width parameter represents the horizontal size of each line in the image, which is also known as the H size. The H size is the value obtained by multiplying the image width and the Bpp (Bytes per pixel). In the given example, the width 3840 is 1920 (image width) x 2 (Bytes per pixel required for the YUV422 format).
If the status node doesn’t show any information, it indicates that the camera pipeline is inactive. If the status node shows incorrect values, one potential cause is the MIPI timing mismatch. In this situation, please refer to 5. (Optional) MIPI CSI-2 Timing for detailed instructions and steps to set the MIPI timing parameters.
5. (Optional) MIPI CSI-2 Timing
MIPI CSI-2 is a high-speed protocol primarily used for real-time imaging and video applications. The MIPI D-PHY physical standard includes several specific timing parameters that govern the behavior of transmission to ensure reliability and correctness. For the detailed timing parameters, please refer to the MIPI CSI-2 specification.
The user can set two MIPI CSI-2 timing parameters, THS-TRAIL and THS-SETTLE, in the device tree to make the transmission stable. The value of the parameters is determined by the MIPI CSI-2 clock rate.
Timing Parameter |
Spec |
Calculation |
Property Name |
|---|---|---|---|
THS-TRAIL |
DPHY: max (8 x UI, 60 ns + 4 x UI ) < THS-TRAIL |
THS-TRAIL = clock_period x (hs_trail_parameter + 1) |
hs_trail_parameter |
THS-SETTLE |
DPHY: 85 ns + 6 x UI < THS-SETTLE < 145 ns + 10 x UI |
THS-SETTLE = clock_period x (dphy_settle_delay_dt + 1) |
dphy_settle_delay_dt |
For example, the MIPI CSI-2 clock rate is 356MHz, so the corresponding timing parameters should satisfy the below spec.
clock_freq = 356 MHz
clock_period = 1 / clock_freq = 2.8 ns
UI = clock_period / 2 = 1.4 ns
THS-TRAIL
THS-TRAIL > max (8 x UI, 60 ns + 4 x UI )
THS-TRAIL > 65.6 ns
2.8 ns x (hs_trail_parameter + 1) > 65.6 ns
hs_trail_parameter > 22.4
THS-SETTLE
85 ns + 6 x UI < THS-SETTLE < 145 ns + 10 x UI
93.4 ns < THS-SETTLE < 159 ns
93.4 ns < 2.8 ns x (dphy_settle_delay_dt + 1) < 159 ns
32.4 < dphy_settle_delay_dt < 55.8
After the calculation, fill in the properties in the DT overlay with the values:
fragment@8 {
target = <&seninf_top>;
__overlay__ {
seninf_csi_port_0: seninf_csi_port_0 {
compatible = "mediatek,seninf";
csi-port = "0";
hs_trail_parameter = <0x25>;
dphy_settle_delay_dt = <0x27>;
port {
...
};
};
};
};
In addition, it’s necessary to consider the timing of MIPI CSI Tx (i.e. the sensor). Sometimes the SoC cannot be compatible with the sensor even if the SoC timing parameters satisfy the MIPI specification.