.. include:: /keyword.rst

===================
Accuracy Evaluation
===================

.. important::

   This evaluation flow is not general to all neural network architectures. Developers must design evaluation tools and workflows tailored to the unique characteristics and requirements of their specific models.

This section provides validation metrics for the original PyTorch model, TFLite models (Quant8 and FP32), and compiled DLA models. Evaluation includes:

* Models converted with open-source tools.
* Models converted using the NeuroPilot Converter tool.
* Hardware-level accuracy on the Genio-700 platform.

.. figure:: /_asset/ai-workflow-overview-step5.png
   :align: center
   :width: 80%

.. raw:: html

   <br><br>

.. note::

   It is recommended to use **Python 3.7** for optimal compatibility with the required libraries and frameworks.

The following examples use the ``yolov5s_int8`` model, which is available on the :doc:`Model ZOO-YOLOv5s </ai_hub/model_zoo/litert_analytical/YOLOv5s>`.


Accuracy Comparison
===================

The following table summarizes the accuracy results for YOLOv5s.

.. csv-table:: YOLOv5s Model Accuracy Comparison
   :class: longtable
   :file: /_asset/tables/accuracy_comparison.csv
   :width: 100%
   :widths: 16 12 12 12 12 12 12 12

.. note::
   * **Source**: TFLite model from an open-source converter.
   * **MTK**: TFLite model from the NeuroPilot Converter Tool.
   * **PC**: Accuracy calculated on a host PC.
   * **Device**: Accuracy calculated on the Genio 700 platform.

End-to-End Conversion Flow and Accuracy Evaluation
==================================================

The following figure provides a detailed view of the end-to-end conversion flow and accuracy evaluation process, highlighting the relationship between host-side preparation and device-side verification.

.. figure:: /_asset/ai-workflow-overview-step5-detail.png
   :alt: Detailed workflow for accuracy evaluation
   :align: center
   :width: 80%

   Detailed end-to-end conversion flow and accuracy evaluation

.. raw:: html

   <br>

This section outlines the systematic steps required to verify YOLOv5s model accuracy throughout the end-to-end deployment workflow.

Source Model Evaluation
-----------------------

Evaluate the original model on a host PC to establish a performance baseline. Ensure all hardware requirements and software dependencies are met before execution.

PyTorch Model Evaluation
^^^^^^^^^^^^^^^^^^^^^^^^

1. Obtain the PyTorch source model:

   .. code-block:: bash

       git clone http://github.com/ultralytics/yolov5
       cd yolov5
       git reset --hard 485da42
       pip install -r requirements.txt

2. Run the evaluation script:

   .. code-block:: bash

       python val.py --weights yolov5s.pt --data coco128.yaml --img 640

   * ``--weights``: Path to the PyTorch weights file.
   * ``--data``: Data configuration file.
   * ``--img``: Input image size.

3. Review the baseline results:

   +----------------------------------------------------+---------+
   | Metric                                             | Value   |
   +====================================================+=========+
   | P (Precision)                                      | 0.709   |
   +----------------------------------------------------+---------+
   | R (Recall)                                         | 0.634   |
   +----------------------------------------------------+---------+
   | mAP\@50 (Mean Average Precision at IoU=0.50)       | 0.713   |
   +----------------------------------------------------+---------+
   | mAP\@50-95 (mAP at IoU=0.50:0.95)                  | 0.475   |
   +----------------------------------------------------+---------+

TFLite Model Evaluation
-----------------------

Evaluate the converted TFLite models on a host PC.

Evaluating Open-Source Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**INT8 Model**

1. Export the model with INT8 quantization:

   .. code-block:: bash

       python export.py --weights yolov5s.pt --include tflite --int8

2. Validate the model:

   .. code-block:: bash

       python val.py --weights yolov5s-int8.tflite --data coco128.yaml --img 640

3. Result:

   +----------------+---------+
   | Metric         | Value   |
   +================+=========+
   | P              | 0.723   |
   +----------------+---------+
   | R              | 0.583   |
   +----------------+---------+
   | mAP\@50        | 0.675   |
   +----------------+---------+
   | mAP\@50-95     | 0.416   |
   +----------------+---------+

**FP32 Model**

1. Download the required conversion scripts and patches:

   .. code-block:: bash

       wget https://mediatek-aiot.s3.ap-southeast-1.amazonaws.com/aiot/download/model-zoo/scripts/model_conversion_YOLOv5s_example_20240916.zip
       unzip -j model_conversion_YOLOv5s_example_20240916.zip

2. Apply the `export_fp32.patch` and export the model:

   .. code-block:: bash

       git apply export_fp32.patch
       python export.py --weights yolov5s.pt --include tflite

3. Validate the model:

   .. code-block:: bash

       python val.py --weights yolov5s-fp32.tflite --data coco128.yaml --img 640

Evaluating NeuroPilot Converter Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**INT8 Model**

Ensure the :doc:`NeuroPilot Converter Tool </ai_hub/ai-workflow/model_converter/neuropilot_converter_tool>` is installed before proceeding.

1. Export the model to TorchScript:

   .. code-block:: bash

       git apply Fix_yolov5_mtk_tflite_issue.patch
       python export.py --weights yolov5s.pt --img-size 640 640 --include torchscript

2. Prepare calibration data and convert to INT8 TFLite:

   .. code-block:: bash

       python prepare_calibration_data.py
       python convert_to_quant_tflite.py

3. Validate the MTK INT8 TFLite model:

   .. code-block:: bash

       python val.py --weights yolov5s_int8_mtk.tflite --data coco128.yaml --img 640

**FP32 Model**

1. Convert the model:

   .. code-block:: bash

       python export.py --weights yolov5s.pt --img-size 640 640 --include torchscript
       python convert_to_tflite.py

2. Validate the MTK FP32 TFLite model:

   .. code-block:: bash

       python val.py --weights yolov5s_mtk.tflite --data coco128.yaml --img 640

DLA Model Evaluation
--------------------

The developer performs this evaluation on a **Genio 700 platform**. Use **NeuroPilot SDK version 6** to ensure full device compatibility.

1. Download and extract the NeuroPilot SDK All-In-One Bundle from the `Official Portal <https://neuropilot.mediatek.com/sphinx/neuropilot-6-basic-customer/html/l1_downloads/downloads_external.html#neuropilot-downloads>`__.

2. Set the environment variables:

   .. code-block:: bash

       export LD_LIBRARY_PATH=/path/to/neuropilot-sdk-basic-<version>/neuron_sdk/host/lib

INT8 DLA Model
^^^^^^^^^^^^^^

1. Compile the INT8 TFLite model to DLA format:

   .. code-block:: bash

       /path/to/ncc-tflite --arch=mdla3.0 yolov5s_int8_mtk.tflite

2. Push the required files to the device:

   .. code-block:: bash

       python prepare_evaluation_dataset.py
       adb shell mkdir -p /tmp/yolov5/device_outputs
       adb push yolov5s_int8_mtk.dla /tmp/yolov5
       adb push evaluation_dataset /tmp/yolov5
       adb push run_device.sh /tmp/yolov5

3. Execute inference on the device and pull results:

   .. code-block:: bash

       adb shell "cd /tmp/yolov5 && sh run_device.sh"
       adb pull /tmp/yolov5/device_outputs .
       python val_int8_inference.py --weights yolov5s_int8_mtk.tflite

FP32 DLA Model
^^^^^^^^^^^^^^

1. Compile the FP32 model with the `--relax-fp32` flag:

   .. code-block:: bash

       /path/to/ncc-tflite --arch=mdla3.0 --relax-fp32 yolov5s_mtk.tflite

2. Push files, execute on-device, and validate:

   .. code-block:: bash

       python prepare_evaluation_dataset_fp32.py
       adb shell mkdir -p /tmp/yolov5/device_outputs_fp32
       adb push yolov5s_mtk.dla /tmp/yolov5
       adb push evaluation_dataset_fp32 /tmp/yolov5
       adb push run_device_for_fp32.sh /tmp/yolov5
       adb shell "cd /tmp/yolov5 && sh run_device_for_fp32.sh"
       adb pull /tmp/yolov5/device_outputs_fp32 .
       python val_fp32_inference.py --weights yolov5s_mtk.tflite

Troubleshooting
===============

PyTorch Version Incompatibility
-------------------------------

**Error:** `RuntimeError: PyTorchConverter only supports 2.0.0 > torch >= 1.3.0.`

**Solution:** Install a compatible PyTorch version:

.. code-block:: bash

    pip3 install torch==1.9.0 torchvision==0.10.0

Missing Shared Library (libtinfo.so.5)
--------------------------------------

**Error:** `ncc-tflite: error while loading shared libraries: libtinfo.so.5`

**Solution:** Install the missing library on the host system:

.. code-block:: bash

    sudo apt-get install libtinfo5