Image Codec
Note
The JPEG hardware codec is NOT available on the Genio 350 platform.
Note
Cmd operations and test results presented in this chapter are based on the IoT Yocto v23.0 and Genio 700-EVK board.
JPEG Codec Overview
On IoT Yocto, JPEG codec hardware provides the V4L2 interface to userspace programs. GStreamer is integrated to provide wrapper plugins over the V4L2 interface and to assist in setting up image processing pipelines.
The GStreamer framework provides software-based or V4L2 hardware-accelerated video processing. To see the list of V4L2 jpeg codecs available on GStreamer, use the following command:
gst-inspect-1.0 | grep v4l2jpeg
video4linux2: v4l2jpegdec: V4L2 JPEG Decoder
video4linux2: v4l2jpegenc: V4L2 JPEG Encoder
Example: JPEG Image Encoding Using GStreamer
The following examples use GStreamer v4l2jpegenc
plug-in for hardware-accelerated JPEG image encoding.
gst-launch-1.0 -v videotestsrc num-buffers=1 ! queue ! video/x-raw,framrate=30/1,width=160,height=128,format=NV12 ! v4l2jpegenc ! queue ! jpegparse ! filesink location=out-160x128-NV12.jpg
Example: JPEG Image Playback Using GStreamer
The following examples use GStreamer v4l2jpegdec
plug-in for hardware-accelerated JPEG image decoding.
gst-launch-1.0 -v filesrc location=out-160x128-NV12.jpg ! jpegparse ! v4l2jpegdec ! waylandsink
JPEG Codec Devices and V4L2 Interface
The hardware JPEG image decoder and encoder support V4L2 API in IoT Yocto. To check V4L2 devices in the console, run the following commands:
ls -l /sys/class/video4linux/
lrwxrwxrwx 1 root root 0 Apr 28 2022 video0 -> ../../devices/platform/soc/1a030000.jpgenc/video4linux/video0
lrwxrwxrwx 1 root root 0 Apr 28 2022 video1 -> ../../devices/platform/soc/1a040000.jpgdec/video4linux/video1
lrwxrwxrwx 1 root root 0 Dec 15 18:47 video2 -> ../../devices/platform/soc/14001000.mdp3-rdma0/video4linux/video2
lrwxrwxrwx 1 root root 0 Dec 15 18:47 video3 -> ../../devices/platform/soc/18000000.vdec/video4linux/video3
lrwxrwxrwx 1 root root 0 Dec 15 18:47 video4 -> ../../devices/platform/soc/1a020000.venc/video4linux/video4
Another utility to enumerate the v4l2 devices is v4l2-sysfs-path
:
v4l2-sysfs-path
Video device: video2
Video device: video3
Video device: video4
Video device: video0
Video device: video1
Alsa playback device(s): hw:0,0 hw:0,1 hw:0,2 hw:0,3 hw:0,4 hw:0,5 hw:0,6
You can also use v4l2-dbg -D -d <device#>
to query information about each V4L2 video device, for example:
v4l2-dbg -D -d 0
Driver info:
Driver name : mtk-jpeg-enc
Card type : mtk-jpeg-enc
Bus info : platform:1a030000.jpgenc
Driver version: 5.15.37
Capabilities : 0x84204000
Video Memory-to-Memory Multiplanar
Streaming
Extended Pix Format
Device Capabilities
v4l2-dbg -D -d 1
Driver info:
Driver name : mtk-jpeg-dec
Card type : mtk-jpeg-dec
Bus info : platform:1a040000.jpgdec
Driver version: 5.15.37
Capabilities : 0x84204000
Video Memory-to-Memory Multiplanar
Streaming
Extended Pix Format
Device Capabilities
As shown in the example above, there are 2 device nodes related to JPEG image codec:
JPEG Encoder (
/dev/video0
and/sys/devices/platform/soc/1a030000.jpgenc/video4linux/video0
)JPEG Decoder (
/dev/video1
and/sys/devices/platform/soc/1a040000.jpgdec/video4linux/video1
)
All two devices are M2M (memory-to-memory) devices.
The userspace clients should access these devices through the V4L2 userspace API. IoT Yocto integrates the GStreamer framework, which provides V4L2 plugins for evaluation and application development.
Note
The JPEG decoder device cannot decode into NV12 or other formats directly. It can only decode the bitstream into YUV420M or YUV422M format. Please refer to the sections below to convert the proprietary format to what you required.
Output Format of JPEG Image Decoder
One thing worth noticing is that the output buffer format of the video decoder device is YM12(YUV420M) or YM16(YUV422M) format and depends on the bitstream. This can be observed with the following commands:
v4l2-ctl --list-formats -d 1
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture Multiplanar
[0]: 'YM12' (Planar YUV 4:2:0 (N-C))
[1]: 'YM16' (Planar YUV 4:2:2 (N-C))
To see other information such as accepted bitstream format, please add --all
parameter:
v4l2-ctl --all -d 1
Driver Info:
Driver name : mtk-jpeg-dec
Card type : mtk-jpeg-dec
Bus info : platform:1a040000.jpgdec
Driver version : 5.15.37
Capabilities : 0x84204000
Video Memory-to-Memory Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04204000
Video Memory-to-Memory Multiplanar
Streaming
Extended Pix Format
Priority: 2
Format Video Capture Multiplanar:
Width/Height : 32/32
Pixel Format : 'YM12' (Planar YUV 4:2:0 (N-C))
Field : None
Number of planes : 3
Flags :
Colorspace : sRGB
Transfer Function : sRGB
YCbCr/HSV Encoding: ITU-R 601
Quantization : Full Range
Plane 0 :
Bytes per Line : 32
Size Image : 1024
Plane 1 :
Bytes per Line : 16
Size Image : 256
Plane 2 :
Bytes per Line : 16
Size Image : 256
Format Video Output Multiplanar:
Width/Height : 32/32
Pixel Format : 'JPEG' (JFIF JPEG)
Field : None
Number of planes : 1
Flags :
Colorspace : sRGB
Transfer Function : sRGB
YCbCr/HSV Encoding: ITU-R 601
Quantization : Full Range
Plane 0 :
Bytes per Line : 0
Size Image : 1048576
Selection Video Capture: compose, Left 0, Top 0, Width 32, Height 32, Flags:
Selection Video Capture: compose_default, Left 0, Top 0, Width 32, Height 32, Flags:
Selection Video Capture: compose_bounds, Left 0, Top 0, Width 32, Height 32, Flags:
Selection Video Capture: compose_padded, Left 0, Top 0, Width 32, Height 32, Flags:
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture Multiplanar
[0]: 'YM12' (Planar YUV 4:2:0 (N-C))
[1]: 'YM16' (Planar YUV 4:2:2 (N-C))
Note
Please note that the term Format Video Capture
means the format of a capture device, which produces buffers.
On the contrary, the term Format Video Output
means the format of a video output device, which takes buffers as inputs.
Therefore, for a M2M device like the decoder,
The Video Output format is the input buffer format of the decoder device.
The Video Capture format is the output buffer format of the decoder device.
Output Format Conversion
If the decoder output format does not meet the requirement of usage, A Gstreamer plug-in, v4l2convert
, can be added to the pipeline as a HW format converter.
The following example uses v4l2jpegdec
and v4l2convert
for decoding a 160x128 JPEG image with resizing to 640x480.
gst-launch-1.0 -v filesrc location=out-160x128-NV12.jpg ! jpegparse ! v4l2jpegdec ! v4l2convert output-io-mode=dmabuf ! video/x-raw,width=640,height=480,format=NV12 ! waylandsink
For the detail of v4l2convert
, please refer to the section: MDP and Format Conversion.
Input Format of JPEG Image Encoder
That the input buffer format of the JPEG encoder device is limited. This can be observed with the following commands:
v4l2-ctl --list-formats-out -d 0
ioctl: VIDIOC_ENUM_FMT
Type: Video Output Multiplanar
[0]: 'NM12' (Y/CbCr 4:2:0 (N-C))
[1]: 'NM21' (Y/CrCb 4:2:0 (N-C))
[2]: 'YUYV' (YUYV 4:2:2)
[3]: 'YVYU' (YVYU 4:2:2)
To see other information such as accepted bitstream format, please add --all
parameter:
v4l2-ctl --all -d 0v4l2-ctl --all -d 0
Driver Info:
Driver name : mtk-jpeg-enc
Card type : mtk-jpeg-enc
Bus info : platform:1a030000.jpgenc
Driver version : 5.15.37
Capabilities : 0x84204000
Video Memory-to-Memory Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04204000
Video Memory-to-Memory Multiplanar
Streaming
Extended Pix Format
Priority: 2
Format Video Capture Multiplanar:
Width/Height : 32/32
Pixel Format : 'JPEG' (JFIF JPEG)
Field : None
Number of planes : 1
Flags :
Colorspace : sRGB
Transfer Function : sRGB
YCbCr/HSV Encoding: ITU-R 601
Quantization : Full Range
Plane 0 :
Bytes per Line : 0
Size Image : 1048576
Format Video Output Multiplanar:
Width/Height : 32/32
Pixel Format : 'YUYV' (YUYV 4:2:2)
Field : None
Number of planes : 1
Flags :
Colorspace : sRGB
Transfer Function : sRGB
YCbCr/HSV Encoding: ITU-R 601
Quantization : Full Range
Plane 0 :
Bytes per Line : 64
Size Image : 2048
Selection Video Output: crop, Left 0, Top 0, Width 0, Height 0, Flags:
Selection Video Output: crop_default, Left 0, Top 0, Width 32, Height 32, Flags:
Selection Video Output: crop_bounds, Left 0, Top 0, Width 32, Height 32, Flags:
JPEG Compression Controls
restart_interval 0x009d0902 (int) : min=0 max=100 step=1 default=0 value=0
compression_quality 0x009d0903 (int) : min=48 max=100 step=1 default=90 value=90
active_markers 0x009d0904 (bitmask): max=0x00000002 default=0x00000000 value=0
Note
Please note that the term Format Video Capture
means the format of a capture device, which produces buffers.
On the contrary, the term Format Video Output
means the format of a video output device, which takes buffers as inputs.
Therefore, for a M2M device like the encoder,
The Video Output format is the input buffer format of the encoder device.
The Video Capture format is the output buffer format of the encoder device.
Motion JPEG Video
Motion JPEG (M-JPEG or MJPEG) is a video compression format in which each video frame of a digital video sequence is compressed separately as a JPEG image. The JPEG encoder device can be used for MJPEG video encoding.
Example: Encode MJPEG Video Using GStreamer
The following examples use GStreamer v4l2jpegenc
plug-in for hardware-accelerated MJPEG video encoding.
gst-launch-1.0 -v videotestsrc num-buffers=300 ! queue ! video/x-raw,framrate=30/1,width=3840,height=2160,format=NV12 ! v4l2jpegenc ! queue ! jpegparse ! qtmux ! filesink location=out-3840x2160-NV12.mov
Also, the JPEG decoder device is ready for the MJPEG video playback.
Example: Decode MJPEG Video Using GStreamer
The following examples use GStreamer v4l2jpegdec
plug-in for hardware-accelerated MJPEG video playback.
gst-launch-1.0 -v filesrc location=out-3840x2160-NV12.mov ! parsebin ! v4l2jpegdec ! waylandsink fullscreen=true
JPEG Hardware Codec Specification
Decoder Specification
Codec |
Input Format |
Output Format |
Record Spec |
Max Resoultion |
Min Resolution |
---|---|---|---|---|---|
JPEG |
JPEG |
YUV420M, YUV422M |
4K30fps |
65535x65535 |
32x32 |
Encoder Specification
Codec |
Input Format |
Output Format |
Record Spec |
Max Resoultion |
Min Resolution |
---|---|---|---|---|---|
JPEG |
NV12, NV21, YUYV, YVYU |
JPEG |
4K30fps |
65535x65535 |
32x32 |