This Lab 2 explains how to get started with MNIST Handwritten Digit application demo on i.MX8 board using eIQ™ ML Software Development Environment.
Get the source code available on code aurora:
MNIST Handwritten Digits
The MNIST is a large database of handwritten digits commonly used for training various image processing systems. This section provides a comparison of Caffe and TensorFlow models for Handwritten Digit Recognition. The data set used for these applications is from Yann Lecun. This is an MNIST data set sample:
Setting Up the Board
Step 1 - Create the following folder and grant it permission as it follows:
root@imx8mmevk:~# mkdir -p /opt/mnist
root@imx8mmevk:~# chmod 777 /opt/mnist
Step 2 - To easily deploy the demos to the board, get the board's IP address using ifconfig command, then set the IMX_INET_ADDR environment variable as follows:
$ export IMX_INET_ADDR=<imx_ip>
Setting Up the Host
Step 1 - Obtain the eIQ toolchain (3.2.9.Generating the Toolchain) from NXP eIQ(TM) Machine Learning Enablement.
Step 2 - Install the toolchain:
$ chmod +x <toolchain>.sh
$ ./<toolchain>.shThis provides all the needed setup for building ARM64 applications on a x86 machine.
Step 3 - Download the application from eIQ Sample Apps.
Step 4 - Get the models and dataset. The following command-lines create the needed folder structure for the demos and retrieves the mnist dataset, and the Caffe and TensorFlow models:
$ mkdir -p bin data model
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/data/t10k-images-idx3-ubyte -P data/
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/data/t10k-labels-idx1-ubyte -P data/
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/model/lenet_iter_9000.caffemodel -P model/
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/model/simple_mnist_tf.pb -P model/
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/model/simple_mnist_tf.prototxt -P model/
$ wget -qN https://github.com/ARM-software/Tool-Solutions/raw/master/ml-tool-examples/mnist-draw/model/optimize... -P model/Step 5 - Compile the source code using the eIQ toolchain:
$ ${CXX} -Wall -Wextra -O3 -std=c++14 caffe_inference.cpp -o caffe_inference -larmnn -larmnnCaffeParser $ ${CXX} -Wall -Wextra -O3 -std=c++14 tensorflow_inference.cpp -o tensorflow_inference -larmnn -larmnnTfParserStep 6 - Deploy the built files to the board:
$ scp -r caffe_inference tensorflow_inference data/ model/ root@${IMX_INET_ADDR}:/opt/mnist
Inference Comparison Applications
Step 1 - At user space, enter the mnist folder which holds the demo files:
root@imx8mmevk:/opt/mnist#This is how the mnist folder structure should look like:
│...
├── caffe_inference
├── tensorflow_inference
├── data
│├── t10k-images-idx3-ubyte
│└── t10k-labels-idx1-ubyte
├── model
│├── lenet_iter_9000.caffemodel
│├── optimized_mnist_tf.pb
│├── simple_mnist_tf.pb
│└── simple_mnist_tf.prototxtStep 2 - Run the applications:
NOTE: For running these applications, please provide the wanted number of predictions, which can vary from 0 to 9999 since the dataset has 10K images.
1 - Handwritten Digit Recognition using Caffe
root@imx8mmevk:/opt/mnist# ./caffe_inference 10
[0] Caffe >> Actual: 7 Predict: 7 Time: 0.0336484s
[1] Caffe >> Actual: 2 Predict: 2 Time: 0.028399s
[2] Caffe >> Actual: 1 Predict: 1 Time: 0.0283713s
[3] Caffe >> Actual: 0 Predict: 0 Time: 0.0284133s
[4] Caffe >> Actual: 4 Predict: 4 Time: 0.0280637s
[5] Caffe >> Actual: 1 Predict: 1 Time: 0.0281574s
[6] Caffe >> Actual: 4 Predict: 4 Time: 0.0285136s
[7] Caffe >> Actual: 9 Predict: 9 Time: 0.0283779s
[8] Caffe >> Actual: 5 Predict: 5 Time: 0.0283902s
[9] Caffe >> Actual: 9 Predict: 9 Time: 0.0283282s
Total Time: 0.296081s
Sucessfull: 10 Failed: 02 - Handwritten Digit Recognition using TensorFlow
root@imx8mmevk:/opt/mnist# ./tensorflow_inference 10
[0] Tensor >> Actual: 7 Predict: 7 Time: 0.00670075s
[1] Tensor >> Actual: 2 Predict: 2 Time: 0.00377025s
[2] Tensor >> Actual: 1 Predict: 1 Time: 0.0036785s
[3] Tensor >> Actual: 0 Predict: 0 Time: 0.0036815s
[4] Tensor >> Actual: 4 Predict: 4 Time: 0.00372875s
[5] Tensor >> Actual: 1 Predict: 1 Time: 0.003669s
[6] Tensor >> Actual: 4 Predict: 4 Time: 0.00367825s
[7] Tensor >> Actual: 9 Predict: 9 Time: 0.0036955s
[8] Tensor >> Actual: 5 Predict: 6 Time: 0.00367488s FAILED
[9] Tensor >> Actual: 9 Predict: 9 Time: 0.0036025s
Total Time: 0.0414569s Sucessfull: 10 Failed: 1NOTE: The argument 10 refers to the number of predictions for each test.
These tests run the inference on the input MNIST dataset images (Actual), showing the inference results (Predict) and how long it took to complete the prediction. The input images for this test are in the binary form and can be found at the t10k-images-idx3-ubyte.gz package from Yann Lecun.
By the output results, it is possible to notice that the Caffe model is slower than TensorFlow, however, it is also more accurate than the latter. Change the argument to compare further results between the two models.
Go to the eIQ Sample Apps - Object Recognition using OpenCV DNN.
