Kernel Development

The Linux kernel is a system software that performs hardware initialization and manages resources of a system. It hides implementation details of underlying hardware and provides an unified interface to applications built upon it. Yocto Project provides a simple way to allow users to build kernel image and customize it to fit application needs.

The following sections will explain how to build, modify, re-build and load the modified kernel in IoT Yocto.

Build and load kernel 

To build kernel image in Yocto Project, perform the following steps:

Set up a build environment by following instructions in environment setup.
Build reference board kernel for the associated SoC:
MACHINE=<machine> bitbake linux-mtk
The available values for <machine> can be:

genio-350-evk for Genio 350 EVK.

genio-510-evk for Genio 510 EVK.

genio-700-evk for Genio 700 EVK.

genio-1200-evk for Genio 1200 EVK.

The resulting kernel binary is located in <build-dir>/tmp/work/<machine>-poky-linux/linux-mtk/<version>.

The corresponding kernel images suitable for flashing process is located in <build-dir>/tmp/deploy/images/<machine>.
Kernel patches and custom kernel configuration can be changed by modifying linux-mtk_<version>.bb and related board config (*.cfg).

These files (linux-mtk_<version>.bb and *.cfg) can be found in <source-dir>/meta-mediatek-bsp/recipes-kernel/linux/.
Once new kernel image is built you can use Genio Tools to update the kernel partition with:
genio-flash kernel

For building Linux outside of Yocto source tree, please refer to BSP development guide.

Modify Kernel Configuration 

The kernel configuration is stored as Yocto kernel config fragments in the path like:

src/meta-mediatek-bsp/recipes-kernel/linux/linux-mtk/<config file>

Where the <config file> is the configuration fragment associated with SoC and boards:

SoC specific config fragment for MT8365 is mt8365.cfg.

Board specific config fragment for Genio-350-EVK is mt8365-evk.cfg.

For example, to enable the USB FunctionFS for MT8365, you can add the following lines to mt8365.cfg:

# USB
CONFIG_USB_FUNCTIONFS=m
CONFIG_USB_FUNCTIONFS_GENERIC=y

and rebuild the linux-mtk recipe.

Sometimes a board config fragment might has a different name from the name used for building image. To locate which fragment is used by which board (the value of MACHINE used in bitbake command), you can search configuration files under src/meta-mediatek-bsp/conf/machine/ for machine names.

For example, the command MACHINE=genio-1200-evk bitbake linux-mtk reads configurations from src/meta-mediatek-bsp/conf/machine/genio-1200-evk.conf, which is an alias to mt8395-evk.conf. And in mt8395-evk.conf, there is a machine override setting like this:

MACHINEOVERRIDES =. "mt8395-evk:genio-1200-evk:"

These overridden machine names are used by recipes-kernel/linux/linux-mtk-common.inc to determine which kernel configuration fragment to be loaded:

SRC_URI:append:mt8395-evk = " file://mt8395-evk.cfg "

As a result, the configuration fragment mt8395-evk.cfg will be included to the KCONFIG when MACHIINE is set to genio-1200-evk.

Modify kernel Source 

To modify kernel source, perform the following:

Run devtool modify linux-mtk to get kernel source

Kernel source is located under <build-dir>/workspace/sources/linux-mtk

Modify kernel source

Build modified kernel by bitbake linux-mtk or devtools build linux-mtk

Flash the fitImage by running Genio Tools command genio-flash kernel
Commit your changes and format them as patches with git:
git commit -s
git format-patch <previous-commit>..<current-commit>
For details on git commands, please refer to git documentation.
You can apply the generated git patch by putting those patches to src/meta-mediatek-bsp/recipes-kernel/linux/linux-mtk/.
Add the file name of the patch to the SRC_URI in corresponding .bb recipe file. For example, if you are working on linux-mtk_5.10.bb, the SRC_URI becomes:
SRC_URI += "${@bb.utils.contains('DISTRO_FEATURES', 'optee', 'file://optee.cfg', '', d)} \
            ${@bb.utils.contains('MACHINE_FEATURES', 'vesper-hat', 'file://vesper.cfg', '', d)}file://usbnet.cfg \
            file://pm.cfg \
            ${@bb.utils.contains('DISTRO_FEATURES', 'nfs', 'file://nfs.cfg', '', d)} \
            ${@bb.utils.contains("DISTRO_FEATURES", "bringup", "${BRINGUP_CFG}", "", d)} \
            ${@bb.utils.contains("DISTRO_FEATURES", "demo", "${DEMO_CFG}", "", d)} \
            file://mt8365-evk.cfg \
            file://mt8365.cfg \
            file://0001-your-new-patch.patch \
            "
by appending the line file://0001-your-new-patch.patch \.
Note

The patches added to the linux-mtk recipe will be applied when bitbake linux-mtk is run but will not be applied after devtool modify linux-mtk.

Hence, when working with the devtool environment, you will need to manually add patches. For example:
cd workspace/sources/linux-mtk && patch -p1 < <path to 0001-your-new-patch.patch>
This is because the bbappend files (located in workspace/appends) generated by devtool contains do_patch[no_exec] = 1.

Step 6. to 8. can also be combined together using the following bitbake command:

devtool update-recipe linux-mtk

Flash kernel image 

After building linux-mtk, the built image can be found under the path <build-dir>/tmp/deploy/images/<machine>. Now we can flash the fitImage with Genio Tools:

genio-flash kernel

Kernel Debugging Tools 

Ramoops oops/panic logger

Ramoops can be used to examine the kernel logs and trace buffers when OS freezing, fatal kernel panic or OOPS happens. For more information on Ramoops, please refer to Ramoops oops/panic logger.

To enable Ramoops, perform the following steps:

Add PSTORE configurations to kernel config file, as instructed in Modify Kernel Configuration.
CONFIG_PSTORE_CONSOLE=y
CONFIG_PSTORE_FTRACE=y
CONFIG_PSTORE_PMSG=y
CONFIG_PSTORE_RAM=m
CONFIG_PSTORE=y
Create a new config file ramoops.cfg and put it under the folder src/meta-rity/meta-rity-demo/recipes-kernel/linux/files, and modify linux-mtk_%.bbappend to include this config file:
DEMO_CFG = " \
file://usb-audio.cfg \
file://exfat.cfg \
file://ramoops.cfg \
Reserve a memory block for ramoops in your board device tree:
reserved-memory {
        /* other reserved regions */

        ramoops@8f000000 {
                compatible = "ramoops";
                reg = <0 0x8f000000 0 0x100000>;
                record-size = <0x4000>;
                console-size = <0x4000>;
        };
};
Note

When reserving the memory region, make sure the region you reserved does not overlap with other existing reserved regions, such as CMA (for GPU) and OPTEE regions.
Rebuild and flash the kernel image.

In the shell of target board, load ramoops module and perform testing:

$ modprobe ramoops
$ <perform your test>
$ reboot

After rebooting, read the kernel log from the previous boot:

$ modprobe ramoops
$ cat /sys/fs/pstore/console-ramoops-0
$ cat /sys/fs/pstore/dmesg-ramoops-0

Register Access Tools 

Devmem2

Using the Devmem tool, which allows the reading and writing of peripheral registers.

To use Devmem, enter the following command:

devmem2 address [type [data]]

Where:

[address] Memory address to act upon.
[type] Access operation type: [b]yte, [h]alfword, [w]ord.
[data] Data to be written.

Initramfs support 

IoT Yocto supports initramfs which allows user to run small programs to perform early setup before handing over control to the real init. The upstream Yocto provides a simple framework for generating initramfs. IoT Yocto adopts the framework, and extends it to provide a reference setup, which can be a starting point for further customization.

To enable initramfs support, please modify local.conf under your build folder and add following lines:

INITRAMFS_IMAGE = "rity-image-initramfs"
INITRAMFS_IMAGE_BUNDLE = "1"

Here we use the image recipe rity-image-initramfs for generating initramfs. And we specify INITRAMFS_IMAGE_BUNDLE to embed initramfs in resulting kernel image. After saving changes, rebuild image.

During booting, the kernel automatically extracts embedded initramfs to memory and starts running init in initramfs.

Customizing initramfs

To customize initramfs, it’s recommended to either extend rity-image-initramfs or create a new recipe, and add one or more initramfs modules based on your needs. There are already several module recipes under poky and meta-rity for reference.

The following subsections will demonstrate how to add a new script to initramfs.

Creating a new initramfs module

To add a new script, firstly we need to create a new recipe called initramfs-module-<module>_1.0.bb, with settings for installing a script to the image. Note the script should be installed under ${D}/init.d, with file name in a format such as 50-<module>. The number prefix indicates the execution order when multiple scripts to be run in initramfs. The script with lower number will be run before the one with higher number.

For example, if we want to add a new module called hello, we can create a new recipe like following:

# initramfs-module-hello_1.0.bb
SUMMARY = "initramfs-framework module for demo"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
RDEPENDS:${PN} = "initramfs-framework-base"

PR = "r0"

inherit allarch

SRC_URI = "file://hello-script"

S = "${WORKDIR}"

do_install() {
    install -d ${D}/init.d
    install -m 0755 ${WORKDIR}/hello-script ${D}/init.d/50-hello
}

FILES:${PN} = "/init.d/50-hello"

And an example script hello-script can be as simple as following:

#!/bin/sh

hello_enabled() {
    return 0
}

hello_run() {
    msg "Hello world"
}

Note for a valid script to be run by initramfs framework, two shell functions must be provided:

<module>_enabled: the function controls if the module is enabled and should be run. The return value 0 means enabling the module, while 1 for disabling.
<module>_run: the function contains code to be run by the framework.

The module name prefix in shell function names must match with the file name of script installed in initramfs (hello in the file name 50-hello in this case).

Extending initramfs recipe

After creating a new initramfs module, we need to add it to initramfs image so that it can be run in booting process. Although you can create your own initramfs recipe, in this case we extend rity-image-initramfs to add a module to the image:

# rity-image-initramfs.bbappend
INITRAMFS_SCRIPTS:append = " \
        initramfs-module-hello \
"

Put the bbappend file under an appropriate folder in your layer, and rebuild the image.

Module partition support 

After making changes to recipe, kernel sources or patches, to verify changes on target board without building complete image, the commonly used approach is setting up TFTP and NFS servers, putting built kernel image and modules under server folders, loading kernel from TFTP, and mounting NFS as rootfs on target board. While it is useful, it requires setting up servers and associated network, which might not be feasible under certain circumstances (e.g. corporate intranet). IoT Yocto provides module partition support as an alternative way for kernel development.

The idea of module partition is to add a new partition in storage, only for storing kernel modules. During booting process, with help of initramfs, a small script is run to mount the partition and setup /lib/modules by using overlayfs, so that new modules override old version when loading drivers at remaining booting stages.

With this mechanism, after changing kernel source, we only need to rebuild kernel & module partition image, and flash them to target board. Since there is no need to generate new rootfs, compared to building complete image, this approach can save up to 50% of building time.

Warning

The module partition should only be used for development purpose. Due to sophisticated versioning issue caused by updating kernel and modules, we don’t recommend enabling the support on production system.

Requirement

The version of genio-tools needs to be 1.3.3 or above to support flashing module partition.

Enabling module partition

To enable module partition support, please modify local.conf under your build folder and add following lines:

INITRAMFS_IMAGE = "rity-image-initramfs"
INITRAMFS_IMAGE_BUNDLE = "1"
DISTRO_FEATURES:append = " modimg"

After saving changes, rebuild the complete image (rity-demo-image) and flash it to your target board.

Note

A complete image rebuilding is required for the first time. And the built image needs to be flashed to the target board, so that module partition is created in on-board storage.

Usage

Now we can make changes to the kernel recipe, and this time we only rebuild kernel:

bitbake linux-mtk

After building is finished, copy following built artifacts (under build/tmp/deploy/images/<machine>) to the folder you used for flashing images:

fitImage (kernel image)
modules-<machine>.modimg.ext4 (module disk image)

And then flash kernel & module disk images to target board by running:

genio-flash kernel modules

Note

Out-of-tree (OOT) drivers are drivers maintained outside of kernel source. Since the recipe linux-mtk only builds kernel and in-tree modules, OOT drivers will never get updated unless building manually. Hence OOT drivers won’t benefit from module partition support.

After login on target board, running df shows different output from the system without module partition:

target# df
Filesystem     1K-blocks   Used Available Use% Mounted on
devtmpfs         3924312      0   3924312   0% /dev
/dev/mmcblk0p4   1494788 768412    630904  55% /
/dev/mmcblk0p3     18327  15589      1268  93% /var/modules
overlay            18327  15589      1268  93% /lib/modules
tmpfs            4025752      0   4025752   0% /dev/shm
tmpfs            1610304   9664   1600640   1% /run
tmpfs               4096      0      4096   0% /sys/fs/cgroup
tmpfs            4025756      0   4025756   0% /tmp
tmpfs            4025752    192   4025560   1% /var/volatile
tmpfs             805148      0    805148   0% /run/user/0

Note that an overlay filesystem is mounted on the folder /lib/modules, and the module partition is mounted on /var/modules. The directory trees of original /lib/modules and /var/modules are merged into a single unified tree. In this tree, if a file exists under /var/modules, it will override the counterpart under /lib/modules. However, if a file exists under /lib/modules but not in /var/modules, it will not be overridden.

Any change made under /lib/modules will be written to /var/modules, and the change will be persistent even after rebooting.