.. include:: /keyword.rst .. _image-common: =========== Image Codec =========== .. contents:: Sections :local: :depth: 3 .. note:: The JPEG hardware codec is **NOT** available on the **Genio 350** platform. .. note:: Command operations and test results presented in this chapter are based on the `IoT Yocto v23.0` and |G700-EVK| board. JPEG Codec Overview =================== On |IOT-YOCTO|, JPEG codec hardware provides the `V4L2`_ interface to userspace programs. `GStreamer`_ is integrated to provide wrapper plugins over the V4L2 interface and to assist in setting up image processing pipelines. .. _GStreamer: https://gstreamer.freedesktop.org/documentation/tutorials/basic/gstreamer-tools.html?gi-language=c The GStreamer framework provides software-based or V4L2 hardware-accelerated video processing. To see the list of V4L2 JPEG codecs available on GStreamer, use the following command: .. prompt:: bash # auto # gst-inspect-1.0 | grep v4l2jpeg video4linux2: v4l2jpegdec: V4L2 JPEG Decoder video4linux2: v4l2jpegenc: V4L2 JPEG Encoder Example: JPEG Image Encoding Using GStreamer -------------------------------------------- The following examples use GStreamer ``v4l2jpegenc`` plug-in for hardware-accelerated JPEG image encoding. .. prompt:: bash # auto # gst-launch-1.0 -v videotestsrc num-buffers=1 ! queue ! video/x-raw,framrate=30/1,width=160,height=128,format=NV12 ! v4l2jpegenc ! queue ! jpegparse ! filesink location=out-160x128-NV12.jpg Example: JPEG Image Playback Using GStreamer -------------------------------------------- The following examples use GStreamer ``v4l2jpegdec`` plug-in for hardware-accelerated JPEG image decoding. .. prompt:: bash # auto # gst-launch-1.0 -v filesrc location=out-160x128-NV12.jpg ! jpegparse ! v4l2jpegdec ! waylandsink JPEG Codec Devices and V4L2 Interface ====================================== The hardware JPEG image decoder and encoder support `V4L2`_ API in |IOT-YOCTO|. To check V4L2 devices in the console, run the following commands: .. _V4L2: https://www.kernel.org/doc/html/v5.15/userspace-api/media/v4l/v4l2.html#v4l2spec .. prompt:: bash # auto # ls -l /sys/class/video4linux/ lrwxrwxrwx 1 root root 0 Apr 28 2022 video0 -> ../../devices/platform/soc/1a030000.jpgenc/video4linux/video0 lrwxrwxrwx 1 root root 0 Apr 28 2022 video1 -> ../../devices/platform/soc/1a040000.jpgdec/video4linux/video1 lrwxrwxrwx 1 root root 0 Dec 15 18:47 video2 -> ../../devices/platform/soc/14001000.mdp3-rdma0/video4linux/video2 lrwxrwxrwx 1 root root 0 Dec 15 18:47 video3 -> ../../devices/platform/soc/18000000.vdec/video4linux/video3 lrwxrwxrwx 1 root root 0 Dec 15 18:47 video4 -> ../../devices/platform/soc/1a020000.venc/video4linux/video4 Another utility to enumerate the v4l2 devices is ``v4l2-sysfs-path``: .. prompt:: bash # auto # v4l2-sysfs-path Video device: video2 Video device: video3 Video device: video4 Video device: video0 Video device: video1 Alsa playback device(s): hw:0,0 hw:0,1 hw:0,2 hw:0,3 hw:0,4 hw:0,5 hw:0,6 You can also use ``v4l2-dbg -D -d `` to query information about each V4L2 video device, for example: .. prompt:: bash # auto # v4l2-dbg -D -d 0 Driver info: Driver name : mtk-jpeg-enc Card type : mtk-jpeg-enc Bus info : platform:1a030000.jpgenc Driver version: 5.15.37 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities # v4l2-dbg -D -d 1 Driver info: Driver name : mtk-jpeg-dec Card type : mtk-jpeg-dec Bus info : platform:1a040000.jpgdec Driver version: 5.15.37 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities As shown in the example above, there are 2 device nodes related to JPEG image codec: 1. JPEG Encoder (``/dev/video0`` and ``/sys/devices/platform/soc/1a030000.jpgenc/video4linux/video0``) 2. JPEG Decoder (``/dev/video1`` and ``/sys/devices/platform/soc/1a040000.jpgdec/video4linux/video1``) All two devices are `M2M (memory-to-memory)`_ devices. The userspace clients should access these devices through the V4L2 userspace API. |IOT-YOCTO| integrates the GStreamer framework, which provides `V4L2 plugins`_ for evaluation and application development. .. _M2M (memory-to-memory): https://www.kernel.org/doc/html/v5.10/userspace-api/media/v4l/dev-mem2mem.html .. _V4L2 plugins: https://github.com/GStreamer/gst-plugins-good/tree/master/sys/v4l2 .. _decoder-format: .. note:: The JPEG decoder device cannot decode into NV12 or other formats directly. It can only decode the bitstream into YUV420M or YUV422M format. Please refer to the sections below to convert the proprietary format to what you required. Output Format of JPEG Image Decoder ----------------------------------- One thing worth noticing is that the output buffer format of the video decoder device is YM12(YUV420M) or YM16(YUV422M) format and depends on the bitstream. This can be observed with the following commands: .. prompt:: bash # auto # v4l2-ctl --list-formats -d 1 ioctl: VIDIOC_ENUM_FMT Type: Video Capture Multiplanar [0]: 'YM12' (Planar YUV 4:2:0 (N-C)) [1]: 'YM16' (Planar YUV 4:2:2 (N-C)) To see other information such as accepted bitstream format, please add ``--all`` parameter: .. prompt:: bash # auto # v4l2-ctl --all -d 1 Driver Info: Driver name : mtk-jpeg-dec Card type : mtk-jpeg-dec Bus info : platform:1a040000.jpgdec Driver version : 5.15.37 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities Device Caps : 0x04204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Priority: 2 Format Video Capture Multiplanar: Width/Height : 32/32 Pixel Format : 'YM12' (Planar YUV 4:2:0 (N-C)) Field : None Number of planes : 3 Flags : Colorspace : sRGB Transfer Function : sRGB YCbCr/HSV Encoding: ITU-R 601 Quantization : Full Range Plane 0 : Bytes per Line : 32 Size Image : 1024 Plane 1 : Bytes per Line : 16 Size Image : 256 Plane 2 : Bytes per Line : 16 Size Image : 256 Format Video Output Multiplanar: Width/Height : 32/32 Pixel Format : 'JPEG' (JFIF JPEG) Field : None Number of planes : 1 Flags : Colorspace : sRGB Transfer Function : sRGB YCbCr/HSV Encoding: ITU-R 601 Quantization : Full Range Plane 0 : Bytes per Line : 0 Size Image : 1048576 Selection Video Capture: compose, Left 0, Top 0, Width 32, Height 32, Flags: Selection Video Capture: compose_default, Left 0, Top 0, Width 32, Height 32, Flags: Selection Video Capture: compose_bounds, Left 0, Top 0, Width 32, Height 32, Flags: Selection Video Capture: compose_padded, Left 0, Top 0, Width 32, Height 32, Flags: ioctl: VIDIOC_ENUM_FMT Type: Video Capture Multiplanar [0]: 'YM12' (Planar YUV 4:2:0 (N-C)) [1]: 'YM16' (Planar YUV 4:2:2 (N-C)) .. note:: Please note that the term ``Format Video Capture`` means the format of a **capture device**, which produces buffers. On the contrary, the term ``Format Video Output`` means the format of a **video output device**, which takes buffers as inputs. Therefore, for a M2M device like the decoder, * The **Video Output** format is the input buffer format of the decoder device. * The **Video Capture** format is the output buffer format of the decoder device. Output Format Conversion ~~~~~~~~~~~~~~~~~~~~~~~~ If the decoder output format does not meet the requirement of usage, A GStreamer plug-in, ``v4l2convert``, can be added to the pipeline as a HW format converter. The following example uses ``v4l2jpegdec`` and ``v4l2convert`` for decoding a 160x128 JPEG image with resizing to 640x480. .. prompt:: bash # auto # gst-launch-1.0 -v filesrc location=out-160x128-NV12.jpg ! jpegparse ! v4l2jpegdec ! v4l2convert output-io-mode=dmabuf ! video/x-raw,width=640,height=480,format=NV12 ! waylandsink For the detail of ``v4l2convert``, please refer to the section: :ref:`MDP and Format Conversion `. Input Format of JPEG Image Encoder ---------------------------------- That the input buffer format of the JPEG encoder device is limited. This can be observed with the following commands: .. prompt:: bash # auto # v4l2-ctl --list-formats-out -d 0 ioctl: VIDIOC_ENUM_FMT Type: Video Output Multiplanar [0]: 'NM12' (Y/CbCr 4:2:0 (N-C)) [1]: 'NM21' (Y/CrCb 4:2:0 (N-C)) [2]: 'YUYV' (YUYV 4:2:2) [3]: 'YVYU' (YVYU 4:2:2) To see other information such as accepted bitstream format, please add ``--all`` parameter: .. prompt:: bash # auto # v4l2-ctl --all -d 0v4l2-ctl --all -d 0 Driver Info: Driver name : mtk-jpeg-enc Card type : mtk-jpeg-enc Bus info : platform:1a030000.jpgenc Driver version : 5.15.37 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities Device Caps : 0x04204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Priority: 2 Format Video Capture Multiplanar: Width/Height : 32/32 Pixel Format : 'JPEG' (JFIF JPEG) Field : None Number of planes : 1 Flags : Colorspace : sRGB Transfer Function : sRGB YCbCr/HSV Encoding: ITU-R 601 Quantization : Full Range Plane 0 : Bytes per Line : 0 Size Image : 1048576 Format Video Output Multiplanar: Width/Height : 32/32 Pixel Format : 'YUYV' (YUYV 4:2:2) Field : None Number of planes : 1 Flags : Colorspace : sRGB Transfer Function : sRGB YCbCr/HSV Encoding: ITU-R 601 Quantization : Full Range Plane 0 : Bytes per Line : 64 Size Image : 2048 Selection Video Output: crop, Left 0, Top 0, Width 0, Height 0, Flags: Selection Video Output: crop_default, Left 0, Top 0, Width 32, Height 32, Flags: Selection Video Output: crop_bounds, Left 0, Top 0, Width 32, Height 32, Flags: JPEG Compression Controls restart_interval 0x009d0902 (int) : min=0 max=100 step=1 default=0 value=0 compression_quality 0x009d0903 (int) : min=48 max=100 step=1 default=90 value=90 active_markers 0x009d0904 (bitmask): max=0x00000002 default=0x00000000 value=0 .. note:: Please note that the term ``Format Video Capture`` means the format of a **capture device**, which produces buffers. On the contrary, the term ``Format Video Output`` means the format of a **video output device**, which takes buffers as inputs. Therefore, for a M2M device like the encoder, * The **Video Output** format is the input buffer format of the encoder device. * The **Video Capture** format is the output buffer format of the encoder device. Motion JPEG Video ================= Motion JPEG (M-JPEG or MJPEG) is a video compression format in which each video frame of a digital video sequence is compressed separately as a JPEG image. The JPEG encoder device can be used for MJPEG video encoding. Example: Encode MJPEG Video Using GStreamer ------------------------------------------- The following examples use GStreamer ``v4l2jpegenc`` plug-in for hardware-accelerated MJPEG video encoding. .. prompt:: bash # auto # gst-launch-1.0 -v videotestsrc num-buffers=300 ! queue ! video/x-raw,framrate=30/1,width=3840,height=2160,format=NV12 ! v4l2jpegenc ! queue ! jpegparse ! qtmux ! filesink location=out-3840x2160-NV12.mov Also, the JPEG decoder device is ready for the MJPEG video playback. Example: Decode MJPEG Video Using GStreamer ------------------------------------------- The following examples use GStreamer ``v4l2jpegdec`` plug-in for hardware-accelerated MJPEG video playback. .. prompt:: bash # auto # gst-launch-1.0 -v filesrc location=out-3840x2160-NV12.mov ! parsebin ! v4l2jpegdec ! waylandsink fullscreen=true JPEG Hardware Codec Specification ================================= Decoder Specification --------------------- .. csv-table:: Codec types of `mtk-jpeg-dec` :class: longtable :file: /_asset/tables/image_dec_spec.csv :width: 80% :header-rows: 1 Encoder Specification --------------------- .. csv-table:: Codec types of `mtk-jpeg-enc` :class: longtable :file: /_asset/tables/image_enc_spec.csv :width: 80% :header-rows: 1