Image Codec

Note

The JPEG hardware codec is NOT available on the Genio 350 platform.

Note

Command operations and test results presented in this chapter are based on the IoT Yocto v23.0 and Genio 700-EVK board.

JPEG Codec Overview

On IoT Yocto, JPEG codec hardware provides the V4L2 interface to userspace programs. GStreamer is integrated to provide wrapper plugins over the V4L2 interface and to assist in setting up image processing pipelines.

The GStreamer framework provides software-based or V4L2 hardware-accelerated video processing. To see the list of V4L2 JPEG codecs available on GStreamer, use the following command:

gst-inspect-1.0 | grep v4l2jpeg
    video4linux2:  v4l2jpegdec: V4L2 JPEG Decoder
    video4linux2:  v4l2jpegenc: V4L2 JPEG Encoder

Example: JPEG Image Encoding Using GStreamer

The following examples use GStreamer v4l2jpegenc plug-in for hardware-accelerated JPEG image encoding.

gst-launch-1.0 -v videotestsrc num-buffers=1 ! queue ! video/x-raw,framrate=30/1,width=160,height=128,format=NV12 ! v4l2jpegenc ! queue ! jpegparse ! filesink location=out-160x128-NV12.jpg

Example: JPEG Image Playback Using GStreamer

The following examples use GStreamer v4l2jpegdec plug-in for hardware-accelerated JPEG image decoding.

gst-launch-1.0 -v filesrc location=out-160x128-NV12.jpg ! jpegparse ! v4l2jpegdec ! waylandsink

JPEG Codec Devices and V4L2 Interface

The hardware JPEG image decoder and encoder support V4L2 API in IoT Yocto. To check V4L2 devices in the console, run the following commands:

ls -l /sys/class/video4linux/
    lrwxrwxrwx 1 root root 0 Apr 28  2022 video0 -> ../../devices/platform/soc/1a030000.jpgenc/video4linux/video0
    lrwxrwxrwx 1 root root 0 Apr 28  2022 video1 -> ../../devices/platform/soc/1a040000.jpgdec/video4linux/video1
    lrwxrwxrwx 1 root root 0 Dec 15 18:47 video2 -> ../../devices/platform/soc/14001000.mdp3-rdma0/video4linux/video2
    lrwxrwxrwx 1 root root 0 Dec 15 18:47 video3 -> ../../devices/platform/soc/18000000.vdec/video4linux/video3
    lrwxrwxrwx 1 root root 0 Dec 15 18:47 video4 -> ../../devices/platform/soc/1a020000.venc/video4linux/video4

Another utility to enumerate the v4l2 devices is v4l2-sysfs-path:

v4l2-sysfs-path
Video device: video2
Video device: video3
Video device: video4
Video device: video0
Video device: video1
Alsa playback device(s): hw:0,0 hw:0,1 hw:0,2 hw:0,3 hw:0,4 hw:0,5 hw:0,6

You can also use v4l2-dbg -D -d <device#> to query information about each V4L2 video device, for example:

v4l2-dbg -D -d 0
Driver info:
        Driver name   : mtk-jpeg-enc
        Card type     : mtk-jpeg-enc
        Bus info      : platform:1a030000.jpgenc
        Driver version: 5.15.37
        Capabilities  : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
v4l2-dbg -D -d 1
Driver info:
        Driver name   : mtk-jpeg-dec
        Card type     : mtk-jpeg-dec
        Bus info      : platform:1a040000.jpgdec
        Driver version: 5.15.37
        Capabilities  : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities

As shown in the example above, there are 2 device nodes related to JPEG image codec:

  1. JPEG Encoder (/dev/video0 and /sys/devices/platform/soc/1a030000.jpgenc/video4linux/video0)

  2. JPEG Decoder (/dev/video1 and /sys/devices/platform/soc/1a040000.jpgdec/video4linux/video1)

All two devices are M2M (memory-to-memory) devices.

The userspace clients should access these devices through the V4L2 userspace API. IoT Yocto integrates the GStreamer framework, which provides V4L2 plugins for evaluation and application development.

Note

The JPEG decoder device cannot decode into NV12 or other formats directly. It can only decode the bitstream into YUV420M or YUV422M format. Please refer to the sections below to convert the proprietary format to what you required.

Output Format of JPEG Image Decoder

One thing worth noticing is that the output buffer format of the video decoder device is YM12(YUV420M) or YM16(YUV422M) format and depends on the bitstream. This can be observed with the following commands:

v4l2-ctl --list-formats -d 1
ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture Multiplanar

    [0]: 'YM12' (Planar YUV 4:2:0 (N-C))
    [1]: 'YM16' (Planar YUV 4:2:2 (N-C))

To see other information such as accepted bitstream format, please add --all parameter:

v4l2-ctl --all -d 1
Driver Info:
        Driver name      : mtk-jpeg-dec
        Card type        : mtk-jpeg-dec
        Bus info         : platform:1a040000.jpgdec
        Driver version   : 5.15.37
        Capabilities     : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
Priority: 2
Format Video Capture Multiplanar:
        Width/Height      : 32/32
        Pixel Format      : 'YM12' (Planar YUV 4:2:0 (N-C))
        Field             : None
        Number of planes  : 3
        Flags             :
        Colorspace        : sRGB
        Transfer Function : sRGB
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Full Range
        Plane 0           :
        Bytes per Line : 32
        Size Image     : 1024
        Plane 1           :
        Bytes per Line : 16
        Size Image     : 256
        Plane 2           :
        Bytes per Line : 16
        Size Image     : 256
Format Video Output Multiplanar:
        Width/Height      : 32/32
        Pixel Format      : 'JPEG' (JFIF JPEG)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : sRGB
        Transfer Function : sRGB
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Full Range
        Plane 0           :
        Bytes per Line : 0
        Size Image     : 1048576
Selection Video Capture: compose, Left 0, Top 0, Width 32, Height 32, Flags:
Selection Video Capture: compose_default, Left 0, Top 0, Width 32, Height 32, Flags:
Selection Video Capture: compose_bounds, Left 0, Top 0, Width 32, Height 32, Flags:
Selection Video Capture: compose_padded, Left 0, Top 0, Width 32, Height 32, Flags:
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'YM12' (Planar YUV 4:2:0 (N-C))
        [1]: 'YM16' (Planar YUV 4:2:2 (N-C))

Note

Please note that the term Format Video Capture means the format of a capture device, which produces buffers. On the contrary, the term Format Video Output means the format of a video output device, which takes buffers as inputs.

Therefore, for a M2M device like the decoder,

  • The Video Output format is the input buffer format of the decoder device.

  • The Video Capture format is the output buffer format of the decoder device.

Output Format Conversion

If the decoder output format does not meet the requirement of usage, A GStreamer plug-in, v4l2convert, can be added to the pipeline as a HW format converter.

The following example uses v4l2jpegdec and v4l2convert for decoding a 160x128 JPEG image with resizing to 640x480.

gst-launch-1.0 -v filesrc location=out-160x128-NV12.jpg ! jpegparse ! v4l2jpegdec ! v4l2convert output-io-mode=dmabuf ! video/x-raw,width=640,height=480,format=NV12 ! waylandsink

For the detail of v4l2convert, please refer to the section: MDP and Format Conversion.

Input Format of JPEG Image Encoder

That the input buffer format of the JPEG encoder device is limited. This can be observed with the following commands:

v4l2-ctl --list-formats-out -d 0
ioctl: VIDIOC_ENUM_FMT
    Type: Video Output Multiplanar

    [0]: 'NM12' (Y/CbCr 4:2:0 (N-C))
    [1]: 'NM21' (Y/CrCb 4:2:0 (N-C))
    [2]: 'YUYV' (YUYV 4:2:2)
    [3]: 'YVYU' (YVYU 4:2:2)

To see other information such as accepted bitstream format, please add --all parameter:

v4l2-ctl --all -d 0v4l2-ctl --all -d 0
Driver Info:
        Driver name      : mtk-jpeg-enc
        Card type        : mtk-jpeg-enc
        Bus info         : platform:1a030000.jpgenc
        Driver version   : 5.15.37
        Capabilities     : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
Priority: 2
Format Video Capture Multiplanar:
        Width/Height      : 32/32
        Pixel Format      : 'JPEG' (JFIF JPEG)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : sRGB
        Transfer Function : sRGB
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Full Range
        Plane 0           :
        Bytes per Line : 0
        Size Image     : 1048576
Format Video Output Multiplanar:
        Width/Height      : 32/32
        Pixel Format      : 'YUYV' (YUYV 4:2:2)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : sRGB
        Transfer Function : sRGB
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Full Range
        Plane 0           :
        Bytes per Line : 64
        Size Image     : 2048
Selection Video Output: crop, Left 0, Top 0, Width 0, Height 0, Flags:
Selection Video Output: crop_default, Left 0, Top 0, Width 32, Height 32, Flags:
Selection Video Output: crop_bounds, Left 0, Top 0, Width 32, Height 32, Flags:

JPEG Compression Controls

            restart_interval 0x009d0902 (int)    : min=0 max=100 step=1 default=0 value=0
            compression_quality 0x009d0903 (int)    : min=48 max=100 step=1 default=90 value=90
                active_markers 0x009d0904 (bitmask): max=0x00000002 default=0x00000000 value=0

Note

Please note that the term Format Video Capture means the format of a capture device, which produces buffers. On the contrary, the term Format Video Output means the format of a video output device, which takes buffers as inputs.

Therefore, for a M2M device like the encoder,

  • The Video Output format is the input buffer format of the encoder device.

  • The Video Capture format is the output buffer format of the encoder device.

Motion JPEG Video

Motion JPEG (M-JPEG or MJPEG) is a video compression format in which each video frame of a digital video sequence is compressed separately as a JPEG image. The JPEG encoder device can be used for MJPEG video encoding.

Example: Encode MJPEG Video Using GStreamer

The following examples use GStreamer v4l2jpegenc plug-in for hardware-accelerated MJPEG video encoding.

gst-launch-1.0 -v videotestsrc num-buffers=300 ! queue ! video/x-raw,framrate=30/1,width=3840,height=2160,format=NV12 ! v4l2jpegenc ! queue ! jpegparse ! qtmux ! filesink location=out-3840x2160-NV12.mov

Also, the JPEG decoder device is ready for the MJPEG video playback.

Example: Decode MJPEG Video Using GStreamer

The following examples use GStreamer v4l2jpegdec plug-in for hardware-accelerated MJPEG video playback.

gst-launch-1.0 -v filesrc location=out-3840x2160-NV12.mov ! parsebin ! v4l2jpegdec ! waylandsink fullscreen=true

JPEG Hardware Codec Specification

Decoder Specification

Codec types of mtk-jpeg-dec

Codec

Input Format

Output Format

Record Spec

Max Resoultion

Min Resolution

JPEG

JPEG

YUV420M, YUV422M

4K30fps

65535x65535

32x32

Encoder Specification

Codec types of mtk-jpeg-enc

Codec

Input Format

Output Format

Record Spec

Max Resoultion

Min Resolution

JPEG

NV12, NV21, YUYV, YVYU

JPEG

4K30fps

65535x65535

32x32