Genio 1200-demo

MT8395 System On Chip

Hardware	MT8395
CPU	4x CA78 2.2GHz 4x CA55 2.0GHz
GPU	ARM G57
AI	APU 3.0 (MDLA 2.0, VPU)

Please refer to the MT8395 (Genio 1200) to find detailed specifications.

APU 3.0

The MediaTek APU 3.0 is a brand-new architecture that houses five AI processors consisting of two big cores, two small cores, and a single tiny core. This highly heterogeneous design is ideally suited to meet the wide variety of smartphone needs that spread across AI-camera, AI-assistant, in-app, or OS-enhancements that make up modern smartphone experiences.

The new APU 3.0 is the fastest ever, offering a huge 4 TOPS - over 2.5X performance uplift over the previous generation.

Overview

On MT8395 P1V6 (Genio 1200) demo board, we provide different software solutions to boost AI computing by GPU and APU (MDLA 2.0, VPU).

GPU Neural Network Acceleration

We provide tensorflow lite with hardware acceleration to develop and deploy a wide range of machine learning. By using TensorFlow Lite Delegates, you can enable hardware acceleration of TFLite models by leveraging on-device accelerators such as the GPU and Digital Signal Processor (DSP)

IoT Yocto already integrated the following two delegates:

• GPU delegate: The GPU delegate uses Open GL ES compute shader on the device to inference TFLite model.

• Arm NN delegate: Arm NN is a set of open-source software that enables machine learning workloads on Arm hardware devices. It provides a bridge between existing neural network frameworks and Cortex-A CPUs, Arm Mali GPUs.

APU Neural Network Acceleration

We introduce the MediaTek-proprietary Machine Learning solution: NeuroPilot on IoT Yocto on MT8395 P1V6 (Genio 1200) demo board.

NeuroPilot is a collection of software tools and APIs which are at the center of MediaTek’s AI ecosystem. With NeuroPilot, users can develop and deploy AI applications on edge devices with extremely high efficiency. This makes a wide variety of AI applications run faster, while also keeping data private.

On MT8395 P1V6 (Genio 1200) demo board, we support Neuron SDK which is one of NeuroPilot software collections. Neuron SDK provides a Neuron compiler (ncc-tflite) to convert TFLite models to MediaTek-proprietary binaries (DLA, Deep Learning Archive) for deployment on MediaTek platforms. The resulting models are highly efficient, with reduced latency and a smaller memory footprint. Neuron SDK also provides Neuron Run-time API which provides a set of APIs that users can invoke from within a C/C++ program to create a run-time environment, parse compiled model file and perform on-device network inference.

Machine learning software stack on MT8395 P1V6 (Genio 1200) demo board

Note

Software information, cmd operations, and test results presented in this chapter are based on the latest version of IoT Yocto (v22.1), MT8395 P1V6 (Genio 1200) demo board.

---

Tensorflow Lite and Delegates

IoT Yocto integrated the tensorflow lite and Arm NN delegate to provide GPU neural network acceleration. The software versions are as follows:

Component	Version	Support Operations
TFLite	2.9.0	TFLite Ops
Arm NN	22.02	Arm NN TFLite Delegate Supported Operators

Supported Operations

Supported Operations

TFLite 2.9.0	Arm NN 22.02
abs	ABS
add	ADD
add_n
arg_max	ARGMAX
arg_min	ARGMIN
assign_variable
average_pool_2d	AVERAGE_POOL_2D
basic_lstm
batch_matmul
batch_to_space_nd	BATCH_TO_SPACE_ND
bidirectional_sequence_lstm
broadcast_args
broadcast_to
bucketize
call_once
cast	CAST
ceil
complex_abs
concatenation	CONCATENATION
conv_2d	CONV_2D
conv_3d	CONV_3D
conv_3d_transpose
cos
cumsum
custom
custom_tf
densify
depth_to_space	DEPTH_TO_SPACE
depthwise_conv_2d	DEPTHWISE_CONV_2D
dequantize	DEQUANTIZE
div	DIV
dynamic_update_slice
elu	ELU
embedding_lookup
equal	EQUAL
exp	EXP
expand_dims
external_const
fake_quant
fill
floor	FLOOR
floor_div
floor_mod
fully_connected	FULLY_CONNECTED
gather	GATHER
gather_nd
gelu
greater	GREATER
greater_equal	GREATER_OR_EQUAL
hard_swish	HARD_SWISH
hashtable
hashtable_find
hashtable_import
hashtable_size
if
imag
l2_normalization	L2_NORMALIZATION
leaky_relu
less	LESS
less_equal	LESS_OR_EQUAL
local_response_normalization	LOCAL_RESPONSE_NORMALIZATION
log
log_softmax	LOG_SOFTMAX
logical_and	LOGICAL_AND
logical_not	LOGICAL_NOT
logical_or	LOGICAL_OR
logistic	LOGISTIC
lstm	LSTM
matrix_diag
matrix_set_diag
max_pool_2d	MAX_POOL_2D
maximum	MAXIMUM
mean	MEAN
minimum	MINIMUM
mirror_pad	MIRROR_PAD
mul	MUL
multinomial
neg	NEG
no_value
non_max_suppression_v4
non_max_suppression_v5
not_equal	NOT_EQUAL
NumericVerify
one_hot
pack	PACK
pad	PAD
padv2
poly_call
pow
prelu	PRELU
pseudo_const
pseudo_qconst
pseudo_sparse_const
pseudo_sparse_qconst
quantize	QUANTIZE
random_standard_normal
random_uniform
range
rank	RANK
read_variable
real
reduce_all
reduce_any
reduce_max	REDUCE_MAX
reduce_min	REDUCE_MIN
reduce_prod
relu	RELU
relu6	RELU6
relu_n1_to_1
reshape	RESHAPE
resize_bilinear	RESIZE_BILINEAR
resize_nearest_neighbor	RESIZE_NEAREST_NEIGHBOR
reverse_sequence
reverse_v2
rfft2d
round
rsqrt	RSQRT
scatter_nd
segment_sum
select
select_v2
shape	SHAPE
sin
slice
softmax	SOFTMAX
space_to_batch_nd	SPACE_TO_BATCH_ND
space_to_depth	SPACE_TO_DEPTH
sparse_to_dense
split	SPLIT
split_v	SPLIT_V
sqrt	SQRT
square
squared_difference
squeeze
strided_slice	STRIDED_SLICE
sub	SUB
sum	SUM
svdf
tanh	TANH
tile
topk_v2
transpose	TRANSPOSE
transpose_conv	TRANSPOSE_CONV
unidirectional_sequence_lstm	UNIDIRECTIONAL_SEQUENCE_LSTM
unidirectional_sequence_rnn
unique
unpack	UNPACK
var_handle
where
while
yield
zeros_like
	L2_POOL_2D

Demo

A python demo application for image recognition is built in the image that can be found in the /usr/share/label_image directory. It is adapted from the upstream label_image.py

cd /usr/share/label_image
ls -l

-rw-r--r-- 1 root root   940650 Mar  9  2018 grace_hopper.bmp
-rw-r--r-- 1 root root    61306 Mar  9  2018 grace_hopper.jpg
-rw-r--r-- 1 root root    10479 Mar  9  2018 imagenet_slim_labels.txt
-rw-r--r-- 1 root root 95746802 Mar  9  2018 inception_v3_2016_08_28_frozen.pb
-rw-r--r-- 1 root root     4388 Mar  9  2018 label_image.py
-rw-r--r-- 1 root root    10484 Mar  9  2018 labels_mobilenet_quant_v1_224.txt
-rw-r--r-- 1 root root  4276352 Mar  9  2018 mobilenet_v1_1.0_224_quant.tflite

Basic commands for running the demo with different delegates are as follows.

• Execute on CPU

cd /usr/share/label_image
python3 label_image.py --label_file labels_mobilenet_quant_v1_224.txt --image grace_hopper.jpg --model_file mobilenet_v1_1.0_224_quant.tflite

• Execute on GPU, with GPU delegate

cd /usr/share/label_image
python3 label_image.py --label_file labels_mobilenet_quant_v1_224.txt --image grace_hopper.jpg --model_file mobilenet_v1_1.0_224_quant.tflite -e /usr/lib64/gpu_external_delegate.so

• Execute on GPU, with Arm NN delegate

cd /usr/share/label_image
python3 label_image.py --label_file labels_mobilenet_quant_v1_224.txt --image grace_hopper.jpg --model_file mobilenet_v1_1.0_224_quant.tflite -e /usr/lib64/libarmnnDelegate.so.25 -o "backends:GpuAcc,CpuAcc"

---

Benchmark Tool

benchmark_model is provided in Tenforflow Performance Measurement for performance evaluation.

Basic commands for running the benchmark tool with CPU and different delegates are as follows.

• Execute on CPU (8 threads):

benchmark_model --graph=/usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite --num_threads=8 --num_runs=10

• Execute on GPU, with GPU delegate:

benchmark_model --graph=/usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite --use_gpu=1 --allow_fp16=0 --gpu_precision_loss_allowed=0 --num_runs=10

• Execute on GPU, with Arm NN delegate:

benchmark_model --graph=/usr/share/label_image/mobilenet_v1_1.0_224_quant.tflite --external_delegate_path=/usr/lib64/libarmnnDelegate.so.25 --external_delegate_options="backends:GpuAcc,CpuAcc" --num_runs=10

---

Neuron SDK

On MT8395 P1V6 (Genio 1200) demo board, IoT Yocto supports Neuron SDK which is one of MediaTek NeuroPilot software collections. Neuron SDK provides Neuron compiler (ncc-tflite) to convert TFLite models to MediaTek-proprietary binaries (DLA, Deep Learning Archive) for deployment on MediaTek platforms. Neuron SDK also provides Neuron Run-time API which provides a set of APIs that users can invoke from within a C/C++ program to create a run-time environment, parse compiled model file and perform on-device network inference. Please refer to Neuron SDK chapter to find all supporting detail.

Supported Operations

Refer to Supported Operations to find all the neural network operations supported by Neuron SDK, and any restrictions placed on their use.

Note

Different compute devices may have restrictions on supported operations. These restrictions are a function of:

1. Op Type

2. Op parameters (e.g. kernel dimensions and modifiers, such as stride)

3. Tensor dimensions (both input and output)

4. Soc Platform

5. Numeric format, both data type, and quantization method

Each device will have its guidelines and restrictions.

Demo

A python demo application for image recognition is built into the image that can be found in the /usr/share/demo_dla directory.

cd /usr/share/demo_dla
ls -l
-rw-r--r-- 1 root root   61306 Mar  9  2018 grace_hopper.jpg
-rw-r--r-- 1 root root   10479 Mar  9  2018 imagenet_slim_labels.txt
-rw-r--r-- 1 root root    1402 Mar  9  2018 label_image.py
-rw-r--r-- 1 root root 4276352 Mar  9  2018 mobilenet_v1_1.0_224_quant.tflite

Use cmd:python3 label_image.py to run the demo. The demo program will convert mobilenet_v1_1.0_224_quant.tflite into DLA, then inference it on APU to classify the image: grace_hopper.jpg. Finally, print out the result of image classification, it should be “military uniform”.

cd /usr/share/demo_dla
python3 label_image.py
/usr/share/demo_dla/mobilenet_v1_1.0_224_quant.dla
/usr/share/demo_dla/grace_hopper.bin
stdout:::  [apusys][info]apusysSession: Seesion(0xaaaadf9ca5e0): thd(runtime_api_sam) version(2) log(0)
The required size of the input buffer is 150528
The required size of the output buffer is 1001
[apusys][info]run: Cmd v2(0xaaaadf9e7b40): run
[apusys][info]run: Cmd v2(0xaaaadf9e7b40): run done(0)
The top index is 653
The image: military uniform

Benchmark Tool

A python application for benchmarking is built in the image that can be found in the /usr/share/benchmark_dla directory.

cd  /usr/share/benchmark_dla
ls -l
-rw-r--r-- 1 root root 26539112 Mar  9  2018 ResNet50V2_224_1.0_quant.tflite
-rw-r--r-- 1 root root     9020 Mar  9  2018 benchmark.py
-rw-r--r-- 1 root root 23942928 Mar  9  2018 inception_v3_quant.tflite
-rw-r--r-- 1 root root  3577760 Mar  9  2018 mobilenet_v2_1.0_224_quant.tflite
-rw-r--r-- 1 root root  6885840 Mar  9  2018 ssd_mobilenet_v1_coco_quantized.tflite

Use cmd:python3 benchmark.py auto to run the benchmark. It will find all TFLite models in /usr/share/benchmark_dla and compile them into DLA, then inference them on APU. Finally, the benchmark result will be saved in /usr/share/benchmark_dla/benchmark.log

cd  /usr/share/benchmark_dla
# run benchmark to evaluate inference time of each model in the current folder
python3 benchmark.py auto

# check inference time of each model
cat benchmark.log
[INFO] mobilenet_v2_1.0_224_quant.tflite, mdla2.0, avg inference time: 3.2
[INFO] mobilenet_v2_1.0_224_quant.tflite, vpu, avg inference time: 14.1
[INFO] ssd_mobilenet_v1_coco_quantized.tflite, mdla2.0, avg inference time: 4.2
[INFO] ssd_mobilenet_v1_coco_quantized.tflite, vpu, avg inference time: 20.1
[INFO] ResNet50V2_224_1.0_quant.tflite, mdla2.0, avg inference time: 8.2
[INFO] ResNet50V2_224_1.0_quant.tflite, vpu, avg inference time: 56.3
[INFO] inception_v3_quant.tflite, mdla2.0, avg inference time: 11.2
[INFO] inception_v3_quant.tflite, vpu, avg inference time: 71.4

---

Benchmark Result

The following table are the benchmark results under performance mode

Average inference time(ms)

Run model (.tflite) 10 times	CPU (Thread:8)	GPU	ARMNN(GpuAcc)	ARMNN(CpuAcc)	APU(MDLA 2.0)	APU(VPU)
inception_v3	97.869	117.606	361.05	361.733
inception_v3_quant	34.627	120.63	113.362	113.358	11.1	71.4
mobilenet_v2_1.0.224	7.259	9.879	21.811	21.764
mobilenet_v2_1.0.224_quant	8.051	10.077	13.288	13.272	3.2	14.1
ResNet50V2_224_1.0	69.717	69.147	232.85	232.685
ResNet50V2_224_1.0_quant	40.337	72.006	91.882	91.718	8.2	56.3
ssd_mobilenet_v1_coco	25.048	31.369	80.44	80.455
ssd_mobilenet_v1_coco_quantized	10.468	37.361	30.73	30.705	4.2	20.1

---

Performance Mode

Force CPU, GPU, APU (MDLA 2.0, VPU) to run at maximum frequency.

• CPU at maximum frequency

  Command to set performance mode for CPU governor

  echo performance > /sys/devices/system/cpu/cpufreq/policy4/scaling_governor
  

• GPU at maximum frequency

  Please refer to Adjust GPU Frequency to fix GPU to run at maximum frequency.

  Or you could just set performance mode for GPU governor, and make the GPU statically to the highest frequency.

  echo performance > /sys/devices/platform/soc/13000000.mali/devfreq/13000000.mali/governor
  

• APU at maximum frequency

  Please refer to QoS Tuning Flow to set qos.boostValue to NEURONRUNTIME_BOOSTVALUE_MAX.