This Lab 2 explains how to get started with MNIST Handwritten Digit application demo on i.MX8 board using eIQ^{™} ML Software Development Environment.
Get the source code available on code aurora:
MNIST Handwritten Digits
The MNIST is a large database of handwritten digits commonly used for training various image processing systems. This section provides a comparison of Caffe and TensorFlow models for Handwritten Digit Recognition. The data set used for these applications is from Yann Lecun. This is an MNIST data set sample:
Setting Up the Board
Step 1 - Create the following folder and grant it permission as it follows:
root@imx8mmevk:~# mkdir -p /opt/mnist
root@imx8mmevk:~# chmod 777 /opt/mnist
Step 2 - To easily deploy the demos to the board, get the board's IP address using ifconfig command, then set the IMX_INET_ADDR environment variable as follows:
$ export IMX_INET_ADDR=<imx_ip>
Setting Up the Host
Step 1 - Obtain the eIQ toolchain (3.2.9.Generating the Toolchain) from NXP eIQ(TM) Machine Learning Enablement.
Step 2 - Install the toolchain:
$ chmod +x <toolchain>.sh
$ ./<toolchain>.sh
This provides all the needed setup for building ARM64 applications on a x86 machine.
Step 3 - Download the application from eIQ Sample Apps.
Step 4 - Get the models and dataset. The following command-lines create the needed folder structure for the demos and retrieves the mnist dataset, and the Caffe and TensorFlow models:
$ mkdir -p bin data model
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/data/t10k-images-idx3-ubyte -P data/
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/data/t10k-labels-idx1-ubyte -P data/
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/model/lenet_iter_9000.caffemodel -P model/
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/model/simple_mnist_tf.pb -P model/
$ wget -qN https://github.com/ARM-software/ML-examples/raw/master/armnn-mnist/model/simple_mnist_tf.prototxt -P model/
$ wget -qN https://github.com/ARM-software/Tool-Solutions/raw/master/ml-tool-examples/mnist-draw/model/optimize... -P model/
Step 5 - Compile the source code using the eIQ toolchain:
$ ${CXX} -Wall -Wextra -O3 -std=c++14 caffe_inference.cpp -o caffe_inference -larmnn -larmnnCaffeParser $ ${CXX} -Wall -Wextra -O3 -std=c++14 tensorflow_inference.cpp -o tensorflow_inference -larmnn -larmnnTfParser
Step 6 - Deploy the built files to the board:
$ scp -r caffe_inference tensorflow_inference data/ model/ root@${IMX_INET_ADDR}:/opt/mnist
Inference Comparison Applications
Step 1 - At user space, enter the mnist folder which holds the demo files:
root@imx8mmevk:/opt/mnist#
This is how the mnist folder structure should look like:
│...
├── caffe_inference
├── tensorflow_inference
├── data
│├── t10k-images-idx3-ubyte
│└── t10k-labels-idx1-ubyte
├── model
│├── lenet_iter_9000.caffemodel
│├── optimized_mnist_tf.pb
│├── simple_mnist_tf.pb
│└── simple_mnist_tf.prototxt
Step 2 - Run the applications:
NOTE: For running these applications, please provide the wanted number of predictions, which can vary from 0 to 9999 since the dataset has 10K images.
1 - Handwritten Digit Recognition using Caffe
root@imx8mmevk:/opt/mnist# ./caffe_inference 10
[0] Caffe >> Actual: 7 Predict: 7 Time: 0.0336484s
[1] Caffe >> Actual: 2 Predict: 2 Time: 0.028399s
[2] Caffe >> Actual: 1 Predict: 1 Time: 0.0283713s
[3] Caffe >> Actual: 0 Predict: 0 Time: 0.0284133s
[4] Caffe >> Actual: 4 Predict: 4 Time: 0.0280637s
[5] Caffe >> Actual: 1 Predict: 1 Time: 0.0281574s
[6] Caffe >> Actual: 4 Predict: 4 Time: 0.0285136s
[7] Caffe >> Actual: 9 Predict: 9 Time: 0.0283779s
[8] Caffe >> Actual: 5 Predict: 5 Time: 0.0283902s
[9] Caffe >> Actual: 9 Predict: 9 Time: 0.0283282s
Total Time: 0.296081s
Sucessfull: 10 Failed: 0
2 - Handwritten Digit Recognition using TensorFlow
root@imx8mmevk:/opt/mnist# ./tensorflow_inference 10
[0] Tensor >> Actual: 7 Predict: 7 Time: 0.00670075s
[1] Tensor >> Actual: 2 Predict: 2 Time: 0.00377025s
[2] Tensor >> Actual: 1 Predict: 1 Time: 0.0036785s
[3] Tensor >> Actual: 0 Predict: 0 Time: 0.0036815s
[4] Tensor >> Actual: 4 Predict: 4 Time: 0.00372875s
[5] Tensor >> Actual: 1 Predict: 1 Time: 0.003669s
[6] Tensor >> Actual: 4 Predict: 4 Time: 0.00367825s
[7] Tensor >> Actual: 9 Predict: 9 Time: 0.0036955s
[8] Tensor >> Actual: 5 Predict: 6 Time: 0.00367488s FAILED
[9] Tensor >> Actual: 9 Predict: 9 Time: 0.0036025s
Total Time: 0.0414569s Sucessfull: 10 Failed: 1
NOTE: The argument 10 refers to the number of predictions for each test.
These tests run the inference on the input MNIST dataset images (Actual), showing the inference results (Predict) and how long it took to complete the prediction. The input images for this test are in the binary form and can be found at the t10k-images-idx3-ubyte.gz package from Yann Lecun.
By the output results, it is possible to notice that the Caffe model is slower than TensorFlow, however, it is also more accurate than the latter. Change the argument to compare further results between the two models.
Go to the eIQ Sample Apps - Object Recognition using OpenCV DNN.
Hi diegodorta,
I encountered some problems when reproducing this demo.
1. Then link in "Step 3 - Download the application from eIQ Sample Apps" is unavailable. It shows "path not found" when I open it. So what is the correct path?
2. I guess the link src\handwritten_digit\examples-armnn - eiq_sample_apps may be the correct one and downloaded it. But I could not compile it because there are no "armnn/****.hpp".
Could you please help me solve them?
Hi Iris,
Thanks for reporting it, you are absolutely right! I will fix the link in the document and I will also re-write the readme file and let it more clear for further references :smileygrin:
Meanwhile, about the "mnist_loader.hpp" you can find the header file here. Could you please test it and see if it works for you?
Thanks,
Diego
Hi diegodorta,
Thanks for your solutions. I downloaded "mnist_loader.hpp" and placed it in the right path, then compiling successfully. And it shows the similar results when running on my board.
For convenience, I created a Makefile for compiling, which is attached for your reference. (I don't know how to attach a file in comments. So I write it directly below.)
all: caffe_inference tensorflow_inference
caffe_inference: caffe_inference.cpp
${CXX} -Wall -Wextra -O3 -std=c++14 $^ -o $@ -larmnn -larmnnCaffeParsertensorflow_inference: tensorflow_inference.cpp
${CXX} -Wall -Wextra -O3 -std=c++14 $^ -o $@ -larmnn -larmnnTfParserclean:
-rm -f caffe_inference tensorflow_inference
Hi Iris,
Glad that all worked out for you. Thanks for the makefile rules, it will be useful :smileygrin:
I will update the documents in a few days on CAF.
Thanks,
Diego