About this document
This document describe the setup detail for Interfacing, Installing, programming (basis) and testing depth cameras with MX6QDL based boards using Ubuntu as rootfs on i.MX6 processors boards.
Supported NXP HW boards:
- i.MX 6QuadPlus SABRE-SD Board and Platform
- i.MX 6Quad SABRE-SD Board and Platform
- i.MX 6DualLite SABRE-SD Board
- i.MX 6Quad SABRE-AI Board
- i.MX 6DualLite SABRE-AI Board
- Depth sensors tested: Microsoft Kinect, ASUS Xtion. Prime Sense Carmine
Software: Gcc, Ubuntu 14.04v, Openni, Python, ROS.
1. Depth Sensor
Depth sensors are 3D vision sensors, mainly used in 3D vision application and motion gaming and robotics. For this paper we are going to use Kinect sensor and Asus Xtion connection to i.MX6 Processor boards and will get 3D images that are converted to finer points called point cloud.
a) Microsoft Kinect
Kinect mainly has an IR camera, IR projector and RGB camera, the IR and projector generates the 3D point cloud of the surroundings. It also has a mic array and motorized tilt for moving up and down.
Kinect reveal that both the video and depth sensor cameras have a 640 x 480-pixel resolution and run at 30 FPS (frames per second). The RGB camera capture 2D color images, whereas the depth camera captures monochrome depth images. Kinect has a depth sensing range from 0.8mts to 3.5 mts
b) Asus Xtion:
Another alternative to kinect is Asus Xtion Pro. Is a 3D sensor designed for motion sensing applications, this sensor is only for 3D sensing and it doesn’t have sound sensing. It has an infrared projector and a monochrome CMOS sensor to capture the infrared data. Xtion can be powered from USB itself and can calculate a sense depth from 0.8mts to 3.5 mts from the sensor.
c) Prime Sense Carmine:
The Prime Sense team (apple bought this company in November 2013) developed the Microsoft Kinect 3D vision, later develop their own 3D vision sensor Carmine. It also works with IR projector, RGB CMOS sensor and a depth CMOS sensor. All sensor are interfaced in System On Chip and is powered trough USB. Carmine capture 640x480 at 30 FPS and can sense from 0.35 mts to 3 mts. Developers can program the device using OpenNI and its wrapper libraries.
All these sensors (Kinect, Carmine and Xtion) support the same software, so there is no special need for programming or general usage. Can be interface to i.MX processor using USB 2.0 interface and programmed using OPENNI and OPENCV.
2. Installation on Ubuntu
For installation steps of Ubuntu trusty on iMX6 boards in your board, please follow up:
Install the dependencies:
$ sudo apt-get install -y g++ git python libusb-1.0-0-dev libudev-dev freeglut3-dev doxygen graphviz openjdk-6-jdk libxmu-dev libxi-dev
Create a devel folder
$ sudo mkdir –p devel $ cd devel Get OpenNI and the drivers $ git clone https://github.com/OpenNI/OpenNI.git -b unstable $ git clone git://github.com/ph4m/SensorKinect.git $ git clone https://github.com/PrimeSense/Sensor.git -b unstable
Set the compile flags to build for the i.MX
$ nano OpenNI/Platform/Linux/CreateRedist/Redist_OpenNi.py From: MAKE_ARGS += ' -j' + calc_jobs_number() To: MAKE_ARGS += ' –j2'
Must also change the Arm compiler settings for this distribution
$ nano OpenNI/Platform/Linux/Build/Common/Platform.Arm From: CFLAGS += -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mfloat-abi=softfp #-mcpu=cortex-a8 To: CFLAGS += -mtune=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard
$ cd OpenNI/Platform/Linux/CreateRedist/ $ ./RedistMaker.Arm $ cd ../Redist/OpenNI-Bin-Dev-Linux-Arm-v1.5.x.x $ sudo ./install.sh
Also edit the Sensor and SensorKinect makefile CFLAGS parameters
$cd ~/devel/ $ nano Sensor/Platform/Linux/Build/Common/Platform.Arm $ nano SensorKinect/Platform/Linux/Build/Common/Platform.Arm For both files From: CFLAGS += -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mfloat-abi=softfp #-mcpu=cortex-a8 To: CFLAGS += -mtune=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard
And the Sensor and SensorKinect redistribution scripts
$ nano Sensor/Platform/Linux/CreateRedist/RedistMaker $ nano SensorKinect/Platform/Linux/CreateRedist/RedistMaker for both, change: make -j$(calc_jobs_number) -C ../Build to: make –j2 -C ../Build
The create the redistributables Sensor (xtion and primesense) and Kinect (sensor Kinect)
$ cd Sensor/Platform/Linux/CreateRedist/ $ ./RedistMaker Arm $ cd ~/devel/ $ cd SensorKinect/Platform/Linux/CreateRedist/ $ ./RedistMaker Arm $ cd ~/devel/
Then install PrimeSense and Kinect
$ cd Sensor/Platform/Linux/Redist/Sensor-Bin-Linux-Arm-v5.1.x.x $ sudo ./install.sh $ cd ~/devel/SensorKinect/Platform/Linux/Redist/Sensor-Bin-Linux-Arm-v5.1.2.x $ sudo ./install.sh
3. Testing Installation:
Connect the sensor power supply, Connect the Kinect to the NXP board USB port. (check with lsusb). For my board:
Imx6q@imx6q:~/devel$ lsusb Bus 001 Device 022: ID 045e:02ae Microsoft Corp. Xbox NUI Camera Bus 001 Device 021: ID 045e:02ad Microsoft Corp. Xbox NUI Audio Bus 001 Device 019: ID 045e:02c2 Microsoft Corp. Kinect for Windows NUI Motor
Now edit theglobal Kinect settings in Ubuntu
$ sudo nano /usr/etc/primesense/GlobalDefaultsKinect.ini and uncomment this line UsbInterface=1 and changed it to 1 instead of 2 UsbInterface=2 $ sudo modprobe -r gspca_kinect
And then blacklisting to avoid it to be auto-loaded on boot:
$ sudo sh -c 'echo "blacklist gspca_kinect" > /etc/modprobe.d/blacklist-kinect.conf'
$ cd ~/devel/OpenNI/Platform/Linux/Bin/Arm-Release $ sudo ./Sample-NiSimpleRead
You should get something like:
Reading config from: '../../../../Data/SamplesConfig.xml' … Frame 40 Middle point is: 5050. FPS: 30.771788 Frame 41 Middle point is: 5050. FPS: 30.866173 Frame 42 Middle point is: 5050. FPS: 30.850958 Frame 43 Middle point is: 5050. FPS: 30.779032 Frame 44 Middle point is: 5050. FPS: 30.767746 Frame 45 Middle point is: 5050. FPS: 30.800463 Frame 46 Middle point is: 5050. FPS: 30.653118 Frame 47 Middle point is: 5050. FPS: 30.741659 Frame 98 Middle point is: 5050. FPS: 30.339321
$ cd ~/devel/ $ git clone https://github.com/OpenKinect/libfreenect.git $ cd libfreenect $ mkdir build $ cd build $ cmake .. –L –DBUILD_AUDIO=ON $ make $ sudo make install
Connect the kinect sensor power supply, connect the Kinect to the NXP board USB port and test any sample such as:
$ sudo freenect-glview
Note: If glview gives a shared library error:
You need to refresh your ldconfig cache. The easiest way to do this is to create a file usr-local-libs.conf (or whatever name you wish) with the following lines:
Switch to root account and move it to /etc/ld.so.conf.d/usr-local-libs.conf. Then update the ldconfig cache:
$ su root $ mv ~/usr-local-libs.conf /etc/ld.so.conf.d/usr-local-libs.conf $ /sbin/ldconfig –v $ exit
3) Learning Robotics using phyton by Lentin Joseph. http://www.amazon.com/Learning-Robotics-Python-Lentin-Joseph/dp/1783287535/ref=sr_1_1