Interfacing Depth Sensors on NXP i.MX6 boards

Document created by Bio_TICFSL Employee on Apr 1, 2016Last modified by Bio_TICFSL Employee on Oct 5, 2016
Version 1Show Document
  • View in full screen mode

About this document

This document describe the setup detail for Interfacing, Installing, programming (basis) and testing depth cameras with MX6QDL based boards using Ubuntu as rootfs on i.MX6 processors boards.

Supported NXP HW boards:

  • i.MX 6QuadPlus SABRE-SD Board and Platform
  • i.MX 6Quad SABRE-SD Board and Platform
  • i.MX 6DualLite SABRE-SD Board
  • i.MX 6Quad SABRE-AI Board
  • i.MX 6DualLite SABRE-AI Board
  • Depth sensors tested: Microsoft Kinect, ASUS Xtion. Prime Sense Carmine

 

Software:   Gcc, Ubuntu 14.04v,  Openni, Python, ROS.

 

1. Depth Sensor

Depth sensors are 3D vision sensors, mainly used in 3D vision application and motion gaming and robotics. For this paper we are going to use Kinect sensor and Asus Xtion connection to i.MX6 Processor boards and will get 3D images that are converted to finer points called point cloud.

 

a) Microsoft Kinect

Kinect mainly has an IR camera, IR projector and RGB camera, the IR and projector generates the 3D point cloud of the surroundings. It also has a mic array and motorized tilt for moving up and down.

Kinect reveal that both the video and depth sensor cameras have a 640 x 480-pixel resolution and run at 30 FPS (frames per second). The RGB camera capture 2D color images, whereas the depth camera captures monochrome depth images. Kinect has a depth sensing range from 0.8mts to 3.5 mts

 

b) Asus Xtion:

Another alternative to kinect is Asus Xtion Pro.  Is a 3D sensor designed for motion sensing applications, this sensor is only for 3D sensing and it doesn’t have sound sensing. It has an infrared projector and a monochrome CMOS sensor to capture the infrared data. Xtion can be powered from USB itself and can calculate a sense depth from 0.8mts to 3.5 mts from the sensor.

c) Prime Sense Carmine:

The Prime Sense team (apple bought this company in November 2013) developed the Microsoft Kinect 3D vision, later develop their own 3D vision sensor Carmine. It also works with IR projector, RGB CMOS sensor and a depth CMOS sensor. All sensor are interfaced in System On Chip and is powered trough USB. Carmine capture 640x480 at 30 FPS and can sense from 0.35 mts to 3 mts. Developers can program the device using OpenNI and its wrapper libraries.

 

All these sensors (Kinect, Carmine and Xtion) support the same software, so there is no special need for programming or general usage. Can be interface to i.MX processor using USB 2.0 interface and programmed using OPENNI and OPENCV.

2. Installation on Ubuntu

For installation steps of Ubuntu trusty on iMX6 boards in your board, please follow up:

https://community.freescale.com/docs/DOC-330147

Install the dependencies:

$ sudo apt-get install -y g++ git python libusb-1.0-0-dev libudev-dev freeglut3-dev doxygen graphviz openjdk-6-jdk libxmu-dev libxi-dev

Create a devel folder

$  sudo mkdir –p devel
$ cd devel 
 Get OpenNI and the drivers
 $ git clone https://github.com/OpenNI/OpenNI.git -b unstable 
 $ git clone git://github.com/ph4m/SensorKinect.git
$ git clone https://github.com/PrimeSense/Sensor.git -b unstable

 

Set the compile flags to build for the i.MX

$ nano OpenNI/Platform/Linux/CreateRedist/Redist_OpenNi.py
From:
 MAKE_ARGS += ' -j' + calc_jobs_number()
To:
 MAKE_ARGS += ' –j2'

 

Must also change the Arm compiler settings for this distribution

$ nano OpenNI/Platform/Linux/Build/Common/Platform.Arm
From:
 CFLAGS += -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mfloat-abi=softfp #-mcpu=cortex-a8
To:
 CFLAGS += -mtune=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard

 

Then run

$ cd OpenNI/Platform/Linux/CreateRedist/
 $ ./RedistMaker.Arm
 $ cd ../Redist/OpenNI-Bin-Dev-Linux-Arm-v1.5.x.x
 $ sudo ./install.sh

 

Also edit the Sensor and SensorKinect makefile CFLAGS parameters

$cd ~/devel/
$ nano Sensor/Platform/Linux/Build/Common/Platform.Arm
$ nano  SensorKinect/Platform/Linux/Build/Common/Platform.Arm
For both files From:
 CFLAGS += -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mfloat-abi=softfp #-mcpu=cortex-a8
To:
 CFLAGS += -mtune=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard

 

And the Sensor and SensorKinect redistribution scripts

$ nano Sensor/Platform/Linux/CreateRedist/RedistMaker
$ nano SensorKinect/Platform/Linux/CreateRedist/RedistMaker
 for both, change:
 make -j$(calc_jobs_number) -C ../Build
to:
 make –j2 -C ../Build

 

The create the redistributables Sensor (xtion and primesense) and Kinect (sensor Kinect)

$ cd Sensor/Platform/Linux/CreateRedist/
 $ ./RedistMaker Arm
$ cd ~/devel/
$ cd SensorKinect/Platform/Linux/CreateRedist/
 $ ./RedistMaker Arm
$ cd ~/devel/

 

Then install PrimeSense and Kinect

$ cd Sensor/Platform/Linux/Redist/Sensor-Bin-Linux-Arm-v5.1.x.x
 $ sudo ./install.sh
$ cd ~/devel/SensorKinect/Platform/Linux/Redist/Sensor-Bin-Linux-Arm-v5.1.2.x
 $ sudo ./install.sh

 

 

3. Testing Installation:

Connect the sensor power supply, Connect the Kinect to the NXP board USB port. (check with lsusb). For my board:

Imx6q@imx6q:~/devel$ lsusb
Bus 001 Device 022: ID 045e:02ae Microsoft Corp. Xbox NUI Camera
Bus 001 Device 021: ID 045e:02ad Microsoft Corp. Xbox NUI Audio
Bus 001 Device 019: ID 045e:02c2 Microsoft Corp. Kinect for Windows NUI Motor

Now edit theglobal Kinect settings in Ubuntu

$ sudo nano /usr/etc/primesense/GlobalDefaultsKinect.ini
and uncomment this line 
 UsbInterface=1
and changed it to 1 instead of 2
UsbInterface=2

$ sudo modprobe -r gspca_kinect

And then blacklisting to avoid it to be auto-loaded on boot:

$ sudo sh -c 'echo "blacklist gspca_kinect" > /etc/modprobe.d/blacklist-kinect.conf'

Then:

$ cd ~/devel/OpenNI/Platform/Linux/Bin/Arm-Release
$ sudo ./Sample-NiSimpleRead

You should get something like:

Reading config from: '../../../../Data/SamplesConfig.xml'
…
Frame 40 Middle point is: 5050. FPS: 30.771788
Frame 41 Middle point is: 5050. FPS: 30.866173
Frame 42 Middle point is: 5050. FPS: 30.850958
Frame 43 Middle point is: 5050. FPS: 30.779032
Frame 44 Middle point is: 5050. FPS: 30.767746
Frame 45 Middle point is: 5050. FPS: 30.800463
Frame 46 Middle point is: 5050. FPS: 30.653118
Frame 47 Middle point is: 5050. FPS: 30.741659
Frame 98 Middle point is: 5050. FPS: 30.339321

 

LibFreenect:

$ cd ~/devel/
$ git clone https://github.com/OpenKinect/libfreenect.git
 $ cd libfreenect
 $ mkdir build
 $ cd build
 $ cmake .. –L –DBUILD_AUDIO=ON
 $ make
 $ sudo make install

 

Connect the kinect sensor power supply, connect the Kinect to the NXP board USB port and test any sample such as:

$ sudo freenect-glview

 

 

Note: If glview gives a shared library error:

You need to refresh your ldconfig cache. The easiest way to do this is to create a file usr-local-libs.conf (or whatever name you wish) with the following lines:

/usr/local/lib

Switch to root account and move it to /etc/ld.so.conf.d/usr-local-libs.conf. Then update the ldconfig cache:

$ su root
$ mv ~/usr-local-libs.conf /etc/ld.so.conf.d/usr-local-libs.conf
$ /sbin/ldconfig –v
$ exit

 

References:

1)      https://openkinect.org/

2)      https://www.asus.com/us/3D-Sensor/Xtion_PRO_LIVE/

3)      Learning Robotics using phyton by Lentin Joseph. http://www.amazon.com/Learning-Robotics-Python-Lentin-Joseph/dp/1783287535/ref=sr_1_1

4)      Computer Vision on i.MX Processors

1 person found this helpful

Attachments

    Outcomes