.. spelling:word-list:: fio iperf glmark neuronrt ng .. include:: /keyword.rst ================= Stress Test Suite ================= The stress test is used for verifying long-term stability and robustness of a system. By exercising different subsystems in parallel under different conditions, we can explore the behavior of a system under heavy loading, and design applications according to those characteristics. MediaTek provides a stress test suite to allow customers for evaluating MediaTek AIoT platforms. The suite is maintained as a Yocto layer which consists of: * Recipes for installing needed tools not included by default |IOT-YOCTO| configuration. * Python3 scripts for launching tools to stress the system. Currently the stress test suite supports following boards: * :doc:`/hw/i350-evk`. * :doc:`/hw/g510-evk`. * :doc:`/hw/g700-evk`. * :doc:`/hw/g1200-evk`. The following sections will describe how to build and run the stress suite. .. contents:: Sections :local: :depth: 1 .. _stress_tools: Tools ===== The stress test suite uses following tools for stressing system: .. list-table:: :widths: 10 10 20 :header-rows: 1 * - Category - Tool - Project URL * - CPU/Memory/IO - stress-ng - https://github.com/ColinIanKing/stress-ng * - Disk IO - fio - https://github.com/axboe/fio * - Networking - iperf3 - https://iperf.fr/ * - GPU - glmark2 - https://github.com/glmark2/glmark2 * - MDLA/VPU - neuronrt - N/A (MediaTek proprietary tool) .. note:: Only specific platforms support MDLA (MediaTek Deep Learning Accelerator). For more information, please refer to :ref:`Hardware Devices ` Testing setup ============= The testing setup is depicted in following diagram (Genio 1200 EVK board in this case): .. figure:: /_asset/stress_test_topology.jpg :align: center :width: 400px * The testing board is connected to a log server via GiB Ethernet connection. The switch used in the test should also support GiB Ethernet. * The board has following peripherals connected to it: * A monitor connected via HDMI * 2 USB disks * 1 MicroSD card * The log server also acts as a `iperf3` server for sending/receiving network traffic. .. note:: The testing setup is based on Genio 1200 EVK board. It might vary depending on different development boards. Building stress test suite ========================== .. note:: Before downloading the Yocto layer, make sure you already have a populated Yocto building environment. Please refer to :doc:`/sw/yocto/get-started/build-code` for more information. Please run following command to download the Yocto layer: .. prompt:: bash git clone https://gitlab.com/mediatek/aiot/rity/meta-stress-test.git $PROJ_ROOT/src/meta-stress-test cd $PROJ_ROOT/src/meta-stress-test git checkout origin/kirkstone .. note:: The branch (*kirkstone*) used in this example might not be the same for your Yocto building environment. Please use correct branch for checking out. And then setup Yocto building environment as described in :doc:`/sw/yocto/get-started/build-code`: .. prompt:: bash cd $PROJ_ROOT export TEMPLATECONF=${PWD}/src/meta-rity/meta/conf/ source src/poky/oe-init-build-env export BUILD_DIR=`pwd` And run following command to add the layer we just downloaded to the environment: .. prompt:: bash bitbake-layers add-layer ../src/meta-stress-test Finally rebuild the image: .. prompt:: bash DISTRO=rity-demo MACHINE=genio-1200-evk bitbake rity-demo-image Please refer to :doc:`/sw/yocto/get-started/flash` to flash the target board with the built image. Host system setup ================= To receive logs sent from target board, the host system needs to have NFS server installed. To install a NFS service, run following command (take Ubuntu as an example): .. prompt:: auto $ sudo apt install nfs-kernel-server # Edit /etc/exports to add filesystems to be exported. $ sudo vi /etc/exports $ sudo systemctl restart nfs-kernel-server .. note:: For details on NFS configuration, please visit `Ubuntu document `_. The host system also needs to have `iperf3` server running: .. prompt:: auto $ sudo apt install iperf3 $ iperf3 -s .. note:: On Ubuntu system, the `iperf3` server needs to be run manually, thus it's recommended to open another terminal to do the task. Running stress test suite ========================= Before running the stress suite, please connect following peripherals to the board: * USB disks * HDMI monitor * MicroSD card * Ethernet .. note:: Which peripherals are needs depends on different board configurations. Please consult related documents for more information. Before running the test, please edit the script ``start.py``: .. prompt:: auto # cd ~/stress_scripts # vi start.py to update global variables according to your testing setup. Here are several settings worth being mentioned: * ``NFS_SERVER``: This variable specifies the path of NFS server in your environment. * ``IPERF_SERVER``: The IP address of `iperf3` server for receiving/sending network traffic. * ``IPERF_BANDWIDTH``: The default bandwidth used in test is 20MB/s. The value can be increased if the environment can afford larger bandwidth. * ``MEM_SIZE``: The total memory used for memory stress test. Please note that other stress tools also occupy large amount of memory, so you can experiment with different values and find an appropriate one. * ``DISK_RATIO``: The ratio of portion of disk spaces are used for running stress test. The larger the ratio, the longer it might take to populate test files before starting stressing. After finishing editing, in the command prompt of target board, run following commands: .. prompt:: auto # cd ~/stress_scripts # ./start.py The output should look like following: .. prompt:: auto [05-05 07:38:30] Mounting NFS folder... emmc mounted Mounting usb1 (/dev/sda1 => /mnt/disk1) Mounting usb2 (/dev/sdb1 => /mnt/disk2) Mounting sd (/dev/mmcblk1p1 => /mnt/sd) [05-05 07:38:31] Test starting... [05-05 07:38:31] Start collecting data... Removing stress test suite ========================== To remove the suite, please run following commands and rebuild the image: .. prompt:: bash bitbake-layers remove-layer ../src/meta-stress-test DISTRO=rity-demo MACHINE=genio-1200-evk bitbake rity-demo-image How to read logs ================ When exercising a system, multiple stressors are run concurrently, which stress different parts of the system. Each stressor writes logs to NFS server, in the `logs` folder under the root folder of the server. After finishing stressing, the listing of log files may look like this: .. prompt:: cpu_220504093357.log cpu_220504100007.log cpu_220504102727.log .... disk_220504093357.log disk_220505053152.log disk_220505054124.log .... ethernet_220505053152.log ethernet_220505054124.log ethernet_220505054227.log .... gpu_220504101603.log gpu_220504102210.log gpu_220504102816.log .... sar_220505054124.data sar_220505054124.log sar_220505073831.data sar_220505073831.log .... The filename of a log file starts with the category of stressor, followed by a timestamp created when running a stressor, in the format of ``. To interpret log content generated by stressors, please consult documents of corresponding tools listed in :ref:`Tools `. There are special files with filename starting with ``sar_`` (e.g. ``sar_220505054124.data``), which are generated by `sar` tool. `sar` is an utility provided by `sysstat `_ for collecting system statistics. Developers can analyze system performance during stressing by drawing graphs using `sadf` tool (another utility of `sysstat`) with collected data files: .. figure:: /_asset/tools_stress-suite_cpu-graph.png :align: center :width: 700px Please refer to `sysstat` documentation to understand more information.