.. include:: /keyword.rst

========================================================
Example: Integrate with Multimedia Frameworks
========================================================

In this section, we provide some examples of integrate :doc:`/sw/yocto/ml-guide/neuron-sdk/neuron_runtime_api` with `GstInference <https://developer.ridgerun.com/wiki/index.php?title=GstInference>`_.
The ``Prediction`` function that is provided below for evaluating frame buffer with neuron runtime is extracted from ``GstInference``. 
Please refer to :doc:`/tools/ai-demo-app` to get familiar with the user interface of ``GstInference``. 
And for the details of Neuron Runtime API, please find :doc:`/sw/yocto/ml-guide/neuron-sdk/neuron_api`.

Create neuron runtime environment at the ``Start`` of the inference engine . 
This function will be called only once after the APP executes.

.. code-block:: cpp

	RuntimeError Engine::Start (const char *in_path)  {
		void *runtime = nullptr;
		int err_code = (*fnNeuronRuntimeV2_create)(dla_model_path, 1, &runtime, /* backlog */2048u);
		if (err_code != NEURONRUNTIME_NO_ERROR) {
			std::cerr << "Failed to create Neuron runtime." << std::endl;
			exit(3);
		}
	}

Now, have a deep dive into the ``Prediction`` phase.

.. literalinclude:: /_asset/code/ml_g1200_neuron_sdk_neuron-api-with-gstinference.cpp
   :language: cpp
   
.. note::
	The version of NeuronAPI now we support on |IOT-YOCTO| is v5.0.1.
	Currently, there is no way for users to insert the meta information of the model when compiling the DLA model.
	Also, model metadata cannot be read out using ``NeuronRuntime_getMetadata`` at runtime. 
	That is why in the above example, the value of ``scale`` and ``zero_point`` 
	in ``ConvertArrayToFixedPoint`` and ``ConvertArrayToFloatingPoint`` are constants.
	Users must have a clear understanding of the model information to have the correct implementation for the inference.

.. note::
	
	Related function for model metadata will be supported in NeuronAPI v5.0.2.

Template for converting float-point to fixed-point.

.. code-block:: cpp
	
	template <typename T1, typename T2>
	static T2 ConvertToFixedPoint(const T1 value, const float scale, const int zero_point) {
		return static_cast<T2>((value / scale) + zero_point);
	}
	
	template <typename T1, typename T2>
	static void ConvertArrayToFixedPoint(const T1 *data, T2 *output_data, const int size, const float scale, const int zero_point) {
	for (int index = 0; index < size; index++) {
		output_data[index] = static_cast<T2>(ConvertToFixedPoint<T1, T2>(data[index], scale, zero_point));
		}
	}

Template for converting fixed-point to float-point.

.. code-block:: cpp

	template <typename T1, typename T2>
	static T1 ConvertToFloatingPoint(const T2 value, const float scale, const int zero_point) {
		return static_cast<T1>((value - zero_point) * scale);
	}
	
	template <typename T1, typename T2>
	static void ConvertArrayToFloatingPoint(const T2 *data, std::vector<T1> &output_data, const int size, const float scale, const int zero_point) {
		for (int index = 0; index < size; index++) {
			output_data[index] = ConvertToFloatingPoint<T1, T2>(data[index], scale, zero_point);
		}
	}