.. include:: /keyword.rst

==================
Kernel Development
==================

The Linux kernel is a system software that performs hardware initialization and manages resources
of a system. It hides implementation details of underlying hardware and provides an unified
interface to applications built upon it. Yocto Project provides a simple way to allow users to build
kernel image and customize it to fit application needs.

The following sections will explain how to build, modify, re-build and load the modified kernel in |IOT-YOCTO|.

.. contents:: Sections
    :local:
    :depth: 1

Build and load kernel
=====================

To build kernel image in Yocto Project, perform the following steps:

    1. Set up a build environment by following instructions in :doc:`environment setup </sw/yocto/get-started>`.

    2. Build reference board kernel for the associated SoC:

       .. code-block::

            MACHINE=<machine> bitbake linux-mtk

       The available values for <machine> can be:

        * ``genio-350-evk`` for Genio 350 EVK.
        * ``genio-510-evk`` for Genio 510 EVK.
        * ``genio-700-evk`` for Genio 700 EVK.
        * ``genio-1200-evk`` for Genio 1200 EVK.

       The resulting kernel binary is located in ``<build-dir>/tmp/work/<machine>-poky-linux/linux-mtk/<version>``.
       
       The corresponding kernel images suitable for flashing process is located in  ``<build-dir>/tmp/deploy/images/<machine>``.

    3. Kernel patches and custom kernel configuration can be changed by modifying ``linux-mtk_<version>.bb`` and related board config (``*.cfg``).
       
       These files (``linux-mtk_<version>.bb`` and ``*.cfg``) can be found in ``<source-dir>/meta-mediatek-bsp/recipes-kernel/linux/``.

    4. Once new kernel image is built you can use :doc:`Genio Tools</sw/yocto/get-started/flash>` to update the kernel partition with:

       .. prompt:: bash

           genio-flash kernel

For building Linux outside of Yocto source tree, please refer to `BSP development guide <https://mediatek.gitlab.io/aiot/rity/meta-mediatek-bsp/development.html#building-linux-outside-of-yocto>`_.

.. _modify-kernel-config:

Modify Kernel Configuration
===========================

The kernel configuration is stored as Yocto kernel `config fragments <https://www.yoctoproject.org/docs/latest/kernel-dev/kernel-dev.html#idm45835993764704>`_ in the path like:

``src/meta-mediatek-bsp/recipes-kernel/linux/linux-mtk/<config file>``

Where the ``<config file>`` is the configuration fragment associated with SoC and boards:

 * SoC specific config fragment for MT8365 is ``mt8365.cfg``.
 * Board specific config fragment for Genio-350-EVK is ``mt8365-evk.cfg``.

For example, to enable the `USB FunctionFS <https://www.kernel.org/doc/Documentation/usb/functionfs.txt>`_ for MT8365, you can add the following lines to ``mt8365.cfg``:

.. code:: kconfig
    
    # USB
    CONFIG_USB_FUNCTIONFS=m
    CONFIG_USB_FUNCTIONFS_GENERIC=y
    
and rebuild the ``linux-mtk`` recipe.

Sometimes a board config fragment might has a different name from the name used for building image. To locate which fragment is used by which board (the value of ``MACHINE`` used in ``bitbake`` command),
you can search configuration files under ``src/meta-mediatek-bsp/conf/machine/`` for machine names.

For example, the command ``MACHINE=genio-1200-evk bitbake linux-mtk`` reads configurations from ``src/meta-mediatek-bsp/conf/machine/genio-1200-evk.conf``, which is an alias to ``mt8395-evk.conf``. And in ``mt8395-evk.conf``, there is a machine override setting like this:

  .. code:: basemake

      MACHINEOVERRIDES =. "mt8395-evk:genio-1200-evk:"

These overridden machine names are used by ``recipes-kernel/linux/linux-mtk-common.inc`` to determine which kernel configuration fragment to be loaded:

  .. code:: basemake

      SRC_URI:append:mt8395-evk = " file://mt8395-evk.cfg "

As a result, the configuration fragment ``mt8395-evk.cfg`` will be included to the KCONFIG when ``MACHIINE`` is set to ``genio-1200-evk``.


Modify kernel Source
====================

To modify kernel source, perform the following:

    1. Run ``devtool modify linux-mtk`` to get kernel source
    2. Kernel source is located under ``<build-dir>/workspace/sources/linux-mtk``
    3. Modify kernel source
    4. Build modified kernel by ``bitbake linux-mtk`` or ``devtools build linux-mtk``
    5. Flash the fitImage by running :doc:`Genio Tools</sw/yocto/get-started/flash>` command ``genio-flash kernel``
    6. Commit your changes and format them as patches with ``git``:

       .. prompt:: bash
           
           git commit -s
           git format-patch <previous-commit>..<current-commit>

       * For details on git commands, please refer to `git documentation <https://git-scm.com/docs>`_.
    
    7. You can apply the generated git patch by putting those patches to ``src/meta-mediatek-bsp/recipes-kernel/linux/linux-mtk/``.

    8. Add the file name of the patch to the ``SRC_URI`` in corresponding ``.bb`` recipe file. For example, if you are working on ``linux-mtk_5.10.bb``, the ``SRC_URI`` becomes:
       
       .. code:: make

            SRC_URI += "${@bb.utils.contains('DISTRO_FEATURES', 'optee', 'file://optee.cfg', '', d)} \
                        ${@bb.utils.contains('MACHINE_FEATURES', 'vesper-hat', 'file://vesper.cfg', '', d)}file://usbnet.cfg \
                        file://pm.cfg \
                        ${@bb.utils.contains('DISTRO_FEATURES', 'nfs', 'file://nfs.cfg', '', d)} \
                        ${@bb.utils.contains("DISTRO_FEATURES", "bringup", "${BRINGUP_CFG}", "", d)} \
                        ${@bb.utils.contains("DISTRO_FEATURES", "demo", "${DEMO_CFG}", "", d)} \
                        file://mt8365-evk.cfg \
                        file://mt8365.cfg \
                        file://0001-your-new-patch.patch \
                        "
       
       by appending the line ``file://0001-your-new-patch.patch \``.

    .. note::

        The patches added to the ``linux-mtk`` recipe will be applied when ``bitbake linux-mtk`` is run but will not be applied after ``devtool modify linux-mtk``.

        Hence, when working with the ``devtool`` environment, you will need to manually add patches. For example:
        
        .. code-block:: bash
            
            cd workspace/sources/linux-mtk && patch -p1 < <path to 0001-your-new-patch.patch>

        This is because the `bbappend` files (located in workspace/appends) generated by ``devtool`` contains ``do_patch[no_exec] = 1``.

Step 6. to 8. can also be combined together using the following ``bitbake`` command:

.. prompt:: bash

    devtool update-recipe linux-mtk


Flash kernel image
==================

After building `linux-mtk`, the built image can be found under the path ``<build-dir>/tmp/deploy/images/<machine>``.
Now we can flash the fitImage with :doc:`Genio Tools</sw/yocto/get-started/flash>`:

.. prompt:: bash
    
    genio-flash kernel

Kernel Debugging Tools
======================

Ramoops oops/panic logger
-------------------------

Ramoops can be used to examine the kernel logs and trace buffers when OS freezing, fatal kernel panic or OOPS happens.
For more information on Ramoops, please refer to `Ramoops oops/panic logger <https://www.kernel.org/doc/html/latest/admin-guide/ramoops.html>`_.

To enable Ramoops, perform the following steps:

    1. Add **PSTORE** configurations to kernel config file, as instructed in :ref:`modify-kernel-config`.

       .. code-block:: kconfig

          CONFIG_PSTORE_CONSOLE=y
          CONFIG_PSTORE_FTRACE=y
          CONFIG_PSTORE_PMSG=y
          CONFIG_PSTORE_RAM=m
          CONFIG_PSTORE=y

    2. Create a new config file ``ramoops.cfg`` and put it under the folder ``src/meta-rity/meta-rity-demo/recipes-kernel/linux/files``,
       and modify ``linux-mtk_%.bbappend`` to include this config file:

       .. code-block:: kconfig

          DEMO_CFG = " \
          file://usb-audio.cfg \
          file://exfat.cfg \
          file://ramoops.cfg \

    3. Reserve a memory block for ``ramoops`` in your board device tree:

       .. code-block::
   
           reserved-memory {
                   /* other reserved regions */
   
                   ramoops@8f000000 {
                           compatible = "ramoops";
                           reg = <0 0x8f000000 0 0x100000>;
                           record-size = <0x4000>;
                           console-size = <0x4000>;
                   };
           };
   
       .. note::
   
           When reserving the memory region, make sure the region you reserved 
           does not overlap with other existing reserved regions, 
           such as CMA (for GPU) and OPTEE regions.


    4. Rebuild and flash the kernel image.

In the shell of target board, load ``ramoops`` module and perform testing:

.. code-block:: bash

    $ modprobe ramoops
    $ <perform your test>
    $ reboot

After rebooting, read the kernel log from the previous boot:

.. code-block:: bash

    $ modprobe ramoops
    $ cat /sys/fs/pstore/console-ramoops-0
    $ cat /sys/fs/pstore/dmesg-ramoops-0

Register Access Tools
======================

`Devmem2`
---------

Using the ``Devmem`` tool, which allows the reading and writing of peripheral registers.

To use ``Devmem``, enter the following command:

.. prompt:: bash
    
   devmem2 address [type [data]] 

Where:
    * [address] Memory address to act upon.
    * [type]    Access operation type: `[b]yte`, `[h]alfword`, `[w]ord`.
    * [data]    Data to be written.

.. _initramfs-support:

Initramfs support
=================

|IOT-YOCTO| supports `initramfs <https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt>`_
which allows user to run small programs to perform early setup before handing over control
to the real init. The upstream Yocto provides a simple framework for generating initramfs.
|IOT-YOCTO| adopts the framework, and extends it to provide a reference setup, which
can be a starting point for further customization.

To enable initramfs support, please modify ``local.conf`` under your build folder and add
following lines:

.. code-block:: text

    INITRAMFS_IMAGE = "rity-image-initramfs"
    INITRAMFS_IMAGE_BUNDLE = "1"

Here we use the image recipe ``rity-image-initramfs`` for generating initramfs. And we specify
``INITRAMFS_IMAGE_BUNDLE`` to embed initramfs in resulting kernel image. After saving changes,
rebuild image.

During booting, the kernel automatically extracts embedded initramfs to memory and starts
running ``init`` in initramfs.

Customizing initramfs
---------------------

To customize initramfs, it's recommended to either extend ``rity-image-initramfs`` or create a new recipe,
and add one or more initramfs modules based on your needs. There are already several module recipes under
``poky`` and ``meta-rity`` for reference.

The following subsections will demonstrate how to add a new script to initramfs.

Creating a new initramfs module
*******************************

To add a new script, firstly we need to create a new recipe called ``initramfs-module-<module>_1.0.bb``,
with settings for installing a script to the
image. Note the script should be installed under ``${D}/init.d``, with file name in a
format such as ``50-<module>``. The number prefix indicates the execution order when multiple scripts
to be run in initramfs. The script with lower number will be run before the one with higher number.

For example, if we want to add a new module called *hello*, we can create a new recipe like following:

.. code-block:: text

    # initramfs-module-hello_1.0.bb
    SUMMARY = "initramfs-framework module for demo"
    LICENSE = "MIT"
    LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
    RDEPENDS:${PN} = "initramfs-framework-base"

    PR = "r0"

    inherit allarch

    SRC_URI = "file://hello-script"

    S = "${WORKDIR}"

    do_install() {
        install -d ${D}/init.d
        install -m 0755 ${WORKDIR}/hello-script ${D}/init.d/50-hello
    }

    FILES:${PN} = "/init.d/50-hello"

And an example script ``hello-script`` can be as simple as following:

.. code-block:: text

    #!/bin/sh

    hello_enabled() {
        return 0
    }

    hello_run() {
        msg "Hello world"
    }

Note for a valid script to be run by initramfs framework, two shell functions must be provided:

* *<module>_enabled*: the function controls if the module is enabled and should be run.
  The return value *0* means enabling the module, while *1* for disabling.
* *<module>_run*: the function contains code to be run by the framework.

The module name prefix in shell function names must match with the file name of script installed in
initramfs (*hello* in the file name ``50-hello`` in this case).


Extending initramfs recipe
**************************

After creating a new initramfs module, we need to add it to initramfs image so that it can be
run in booting process. Although you can create your own initramfs recipe, in this case we
extend ``rity-image-initramfs`` to add a module to the image:

.. code-block:: text

    # rity-image-initramfs.bbappend
    INITRAMFS_SCRIPTS:append = " \
            initramfs-module-hello \
    "

Put the `bbappend` file under an appropriate folder in your layer, and rebuild the image.

.. _module-partition-support:

Module partition support
========================

After making changes to recipe, kernel sources or patches, to verify changes on target board
without building complete image, the commonly used approach is setting up TFTP and NFS
servers, putting built kernel image and modules under server folders, loading kernel from
TFTP, and mounting NFS as rootfs on target board. While it is useful, it requires setting up
servers and associated network, which might not be feasible under certain circumstances (e.g.
corporate intranet). |IOT-YOCTO| provides module partition support as an alternative way for
kernel development.

The idea of module partition is to add a new partition in storage, only for storing kernel
modules. During booting process, with help of initramfs, a small script is run to mount the
partition and setup ``/lib/modules`` by using `overlayfs <https://docs.kernel.org/filesystems/overlayfs.html>`_,
so that new modules override old version when loading drivers at remaining booting stages.

With this mechanism, after changing kernel source, we only need to rebuild kernel & module
partition image, and flash them to target board. Since there is no need to generate new rootfs,
compared to building complete image, this approach can save up to 50% of building time.

.. warning::

    The module partition should only be used for development purpose. Due to sophisticated
    versioning issue caused by updating kernel and modules, we don't recommend enabling the
    support on production system.

Requirement
-----------

The version of `genio-tools` needs to be *1.3.3* or above to support flashing module partition.

Enabling module partition
-------------------------

To enable module partition support, please modify ``local.conf`` under your build folder and
add following lines:

.. code-block:: text

    INITRAMFS_IMAGE = "rity-image-initramfs"
    INITRAMFS_IMAGE_BUNDLE = "1"
    DISTRO_FEATURES:append = " modimg"

After saving changes, rebuild the complete image (*rity-demo-image*) and flash it to your
target board.

.. note::

    A complete image rebuilding is required for the first time. And the built image needs
    to be flashed to the target board, so that module partition is created in on-board storage.

Usage
-----

Now we can make changes to the kernel recipe, and this time we only rebuild kernel:

.. code-block:: bash

    bitbake linux-mtk

After building is finished, copy following built artifacts (under
``build/tmp/deploy/images/<machine>``) to the folder you used for flashing images:

* ``fitImage`` (kernel image)
* ``modules-<machine>.modimg.ext4`` (module disk image)

And then flash kernel & module disk images to target board by running:

.. code-block:: bash

    genio-flash kernel modules

.. note::

    Out-of-tree (OOT) drivers are drivers maintained outside of kernel source. Since
    the recipe ``linux-mtk`` only builds kernel and in-tree modules, OOT
    drivers will never get updated unless building manually. Hence OOT drivers won't
    benefit from module partition support.

After login on target board, running ``df`` shows different output from
the system without module partition:

.. code-block:: text

    target# df
    Filesystem     1K-blocks   Used Available Use% Mounted on
    devtmpfs         3924312      0   3924312   0% /dev
    /dev/mmcblk0p4   1494788 768412    630904  55% /
    /dev/mmcblk0p3     18327  15589      1268  93% /var/modules
    overlay            18327  15589      1268  93% /lib/modules
    tmpfs            4025752      0   4025752   0% /dev/shm
    tmpfs            1610304   9664   1600640   1% /run
    tmpfs               4096      0      4096   0% /sys/fs/cgroup
    tmpfs            4025756      0   4025756   0% /tmp
    tmpfs            4025752    192   4025560   1% /var/volatile
    tmpfs             805148      0    805148   0% /run/user/0

Note that an overlay filesystem is mounted on the folder ``/lib/modules``, and
the module partition is mounted on ``/var/modules``. The directory trees
of original ``/lib/modules`` and ``/var/modules`` are *merged* into a single unified
tree. In this tree, if a file exists under ``/var/modules``, it will override the
counterpart under ``/lib/modules``. However, if a file exists under ``/lib/modules``
but not in ``/var/modules``, it will not be overridden.

Any change made under ``/lib/modules`` will be written to ``/var/modules``,
and the change will be persistent even after rebooting.