Dumping the pipeline elements into a image file # On target, run the pipeline $ export GST_DEBUG_DUMP_DOT_DIR=<folder where dot files are created> $ gst-launch playbin2 uri=file://${avi} $ # Move the .dot files to a host machine (scp, etc) # On Host dot <dot file> -Tpng -o out.png # dot command is part the the graphviz package Querying which elements are being used on a gst-launch command GST_DEBUG=GST_ELEMENT_FACTORY:3 gst-launch playbin2 uri=file://`pwd`/<media file> Interrupting a gst-launch process running in the background kill -INT $PID # where $PID is the process ID Using only SW codecs # Backup and remove $ find /usr/lib/gstreamer-0.10 -name "libmfw*" | grep -v sink | xargs tar cvf /libmfw_gst.tar $ find /usr/lib/gstreamer-0.10 -name "libmfw*" | grep -v sink | xargs rm # Run your pipeline. This time SW codecs are used $ gst-launch playbin2 uri=file://`pwd`/media_file # To 'install' FSL plugins again, just untar the file $ cd / && tar xvf libmfw_gst.tar && cd - # then run your pipeline. This time HW codecs are used $ gst-launch playbin2 uri=file://`pwd`/media_file

Video, bad performance gst-launch filesrc location=test.mp4 typefind=true ! aiurdemux ! vpudec ! mfw_v4lsink Video, better performance gst-launch filesrc location=sample.mp4 typefind=true ! aiurdemux ! queue max-size-time=0 ! vpudec ! mfw_v4lsink # typefind=true allows to 'type find' the source file before negotiating # max-size-time=0 indicates to ignore possible blocking issues # In case of ASF files gst-launch filesrc location=sample.asf typefind=true ! aiurdemux ! queue max-size-time=0 ! mfw_wmvdecoder ! mfw_v4lsink Audio gst-launch filesrc location=sample.mp3  typefind=true ! beepdec ! audioconvert  ! 'audio/x-raw-int, channels=2' ! alsasink Audio with visualization gst-launch filesrc location=sample.mp3 typefind=true ! beepdec ! tee name=t ! queue ! audioconvert  ! 'audio/x-raw-int, channels=2' ! alsasink t. ! queue ! audioconvert ! goom ! autovideoconvert ! autovideosink Video/Audio long version gst-launch filesrc location=sample.avi typefind=true ! aiurdemux name=demux demux. ! queue max-size-buffers=0 max-size-time=0 ! vpudec ! mfw_v4lsink demux. ! queue max-size-buffers=0 max-size-time=0 ! beepdec ! audioconvert ! 'audio/x-raw-int, channels=2' ! alsasink # queue properties, max-size-buffers=0 and max-size-time=0, allows a smoother playback; type 'gst-inspect queue' for more info VA short version gplay sample.avi VA short version gst-launch playbin2 uri=file://<full path to sample file>

Description       this doc is explain how to develop a audio card driver base on i.MX6 platform. which explain the ASOC architecture struction basic knowledage and then give some sample for the audio driver development like: 1:NXP SGTL5000: NXP i.MX BSP sabrelite board default support it. 2: Wolfson WM8524.    A: 3.0.35 BSP support: i.MX6 setbox BSP support it:(which in elder fsl community link and out of data)    B: 3.14.28 BSP support pls check attachment: 3: Wolfson WM8960.     which include how to add the android middle-layer and driver, pls check attachment. 4: TI TLV320AIC3120      which include how to add the android middle-layer and driver, pls check attachment. 5: TI TLV320AIC3X   Products Product Category NXP Part Number URL MPU i.MX6 Family https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-6-processors:IMX6X_SERIES   Tools NXP Development Board URL i.MX6 SabreSDP https://www.nxp.com/design/development-boards:EVDEBRDSSYS#/collection=softwaretools&start=0&max=25&query=typeTax%3E%3Et633::archived%3E%3E0::Sub_Asset_Type%3E%3ETSP::deviceTax%3E%3Ec731_c380_c127_c126&sorting=Buy%2FSpecifications.desc&language=en&siblings=false which have a doc MX6X_ASOC_V5-20191115.pdf and related driver sample codes.

This document describes all the i.MX 8 MIPI-CSI use cases, showing the available cameras and daughter cards supported by the boards, the compatible Device Trees (DTS) files, and how to enable these different camera options on the i.MX 8 boards. Plus, this document describes some Advanced camera use cases too, such as multiples cameras output using imxcompositor_g2d plugin, GStreamer zero-copy pipelines and V4L2 API extra-controls examples.

[中文翻译版] 见附件   原文链接： https://community.nxp.com/docs/DOC-343528 

///////////////////////////create device node /dev/galcore///////////////////////////// $home/myandroid/kernel_imx/drivers/mxc/gpu-viv/Kbuild MODULE_NAME ?= galcore /* define node name*/ $home/myandroid/kernel_imx/drivers/mxc/gpu-viv/hal/os/linux/kernel/gc_hal_kernel_linux.h define DEVICE_NAME "galcore" $home/myandroid/kernel_imx/drivers/mxc/gpu-viv/hal/os/linux/kernel/gc_hal_kernel_probe.c drv_init call ret = register_chrdev(major, DEVICE_NAME, &driver_fops); ///////////////////////////////opengles2 functios/////////////////////////////////////////// myandroid/device/fsl-proprietary/gpu-viv/lib/egl/libGLESv2_VIVANTE.so glActiveTexture glBindBuffer ... ... ... //those glxxxxxx call into sub_D40C int __fastcall sub_D40C(int a1, int a2, int a3) //address 0x0000D40C { int result; // r0@1 int v4; int v5; v4 = a2;   v5 = a3;   gcoOS_GetTLS(&v4);  //------------> goto libGAL.so   result = v4;   if ( v4 )     result = *(_DWORD *)(v4 + 36);   return result; } and $home/myandroid/device/fsl-proprietary/gpu-viv/lib/libGAL.so //export function signed int __fastcall gcoOS_GetTLS(void **a1) { ... ... gcoOS_GetTLS v4 = open("/dev/galcore", 2); ... ... } and device node /dev/galcore pass command into module galcore $home/myandroid/kernel_imx/drivers/mxc/gpu-viv/hal/kernel/gc_hal_kernel.c gckKERNEL_Dispatch This document was generated from the following discussion: Share Vivante 3d gc2000 work flow

In some cases it is desired to directly have progressive content available from a TV-IN interface through the V4L2 capture device. In the BSP, HW accelerated de-interlacing is only supported in the V4L2 output stream. Below is a patch created against a rather old BSP version that adds support for de-interlaced V4L2 capture. The patch might need to be adapted to newer BSPs, However, the logic and functionality is there and should shorten the development time. This patch adds another input device to the V4L2 framework that can be selected to perform the deinterlacing on the way to memory. The selection is done by passing the index “2” as an argument to the VIDIOC_S_INPUT  V4L2 ioctl. Attached is also a modified the tvin unit test to give an example of how to use the new driver. An example sequence for running the test is as follows: modprobe mxc_v4l2_capture ./mxc_v4l2_tvin_vdi.out -ow 720 -oh 480 -ol 10 -ot 20 -f YU12 Some key things to note: This driver does not support resize or color space conversion on the way to memory. The requested format and size should match what can be provided directly by the sensor. The driver was tested on a Sabre AI Rev A board running Linux 12.02. This code is not an official delivery and as such no guarantee of support for this code is provided by Freescale.

MIPI can support video streaming over 1, 2, 3 and 4 lanes. On i.MX6 Sabre boards, the OV5640 camera supports 1 or 2 lanes and the NXP Linux Kernel uses 2 lanes as default. In order to use only one lane, follow the steps below: 1 - Change the board Device Tree on Linux Kernel. On file <linux kernel folder>/arch/arm/boot/dts/imx6qdl-sabresd.dtsi, find the entry "&mipi_csi" and change lanes from 2 to 1. 2 - Configure OV5640 to use only one lane instead of two. On file <linux kernel folder>/drivers/media/platform/mxc/capture/ov5640_mipi.c, change the register 0x300e value from 0x45 to 0x05. This register setup is located at struct ov5640_init_setting_30fps_VGA. 3 - Build the kernel and device tree files. 4 - Test the camera. Unit test can be used to test the video capture: /unit_tests/mxc_v4l2_overlay.out -di /dev/video1 -ow 1024 -oh 768 -m 1 5 - Checking if it's really using one lane. MIPI_CSI_PHY_STATE resgister (address 0x021D_C014) provides the status of all data and clock lanes. During video streaming using 2 lanes, the register value constantly changes its value between 0x0000_0300 and 0x0000_0330. When using only one lane, this register value constantly changes its value between 0x0000_0300 and 0x0000_0310. To read the register value during the stream, run the video test with &: /unit_tests/mxc_v4l2_overlay.out -di /dev/video1 -ow 1024 -oh 768 -m 1 & Now, run the memtool: /unit_tests/memtool -32 0x021dc014 1 i.MX6DL running mxc_v4l2_overlay.out with only one lane:

All Boards Creating App MP3 OpenEmbedded

Description about VPU & IPU usage in Android R13.4 GA release for i.MX6DQ

Overview As more and more communication required between online and offline, the QR code is widely used in the mobile payment, mobile small apps, industry things identification and etc. The i.MX6UL/ULL has the IP of CSI and PXP for camera connection and image CSC/FLIP/ROTATION acceleration. A LCDIF IP is supporting the display, but no 3D IP support. This means this low power and low end AP is very suitable for the industry HMI segment, which does not require a cool 3D graphic display, but a simple and straightforward GUI for interaction. QR code scanner is one of the use cases in the industry segment, which more and more customer are focusing on. The i.MX6UL CPU freq of i.MX6UL is about 500Mhz, and it does not have GPU IP, so a lightweight GUI and window system is required. Here we recommend the QT with wayland backend (without X11), which would make the window system small and faster than traditional X11 UI. Why chose QT is because of it has open source version, rich components, platform independent, good performance for embedded system and strong development staffs like QtCreator for creating application. How to enable the QT development environment, check this: Enable QT developement for i.MX6UL (v2)  Here I made a QR code scanner demo based on QT5.6 + QZXing (QR/Bar code scan engine) running on the i.MX6UL EVK board with a UVC camera (at least 640x480 resolution is required) and 480x272px LCD. Source code is open here (License Apache2.0): https://github.com/muddog/QRScanner  Implementation To do camera preview and capture, you must think on the gstreamer first, which is easy use and has the acceleration pads which implemented by NXP for i.MX6UL. Yes, it's very easy for you to enable the preview in console like: $ gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=640,height=320 ! imxvideoconvert_pxp ! video/x-raw,format=RGB16 ! waylandsink It works under the i.MX6UL EVK, with PXP IP to do color space convert from YUY2 -> RGB16 acceleration, also the potential scaling of the image. The CPU loading of this is about 20-30%, but if you use the component of "videoconvert" to replace the "imxvideoconvert_pxp", we do CSC and scale by CPU, then the loading would increase to 50-60%. The "/dev/video1" is the device node for UVC camera, it may different in your environment. So our target is clear, create such pipeline (with PXP acceleration) in the QT application, and use a appsink to get preview images, do simple "sink" to one QWidget by drawing this image on the widget surface for preview (say every 50ms for 20fps). Then in other thread, we fetch the preview buffer in a fixed frequency (like every 0.5s), then feed it into the ZXing engine to decode the strings inside this image. Here are the class created inside the source code: ScannerQWidgetSink It act as a gstreamer sink for preview rendering. Init the pipeline, create a timer with timeout every 50ms. In the timer handler, we use appsink to copy the camera buffer from gstreamer, and tell the ViewfinderWidget to do update (re-draw event). ViewfinderWidget This class inherit from the QWidget, which draw the preview buffer as a QImage onto it's own surface by using QPainter. The QImage is created at the very begining with the image buffer created by the ScannerQWidgetSink. Because QImage itself does not maintain the image buffer, so the buffer must be alive during it's usage. So we keep this buffer during the ScannerQWidgetSink life cycle, copy the appsink buffer from pipeline to it for preview. MainWindow Create main window, which does not have title bar and border. Start any animation for the red line scan bar. Create instance of DecoderThread and ScannerQWidgetSink. Setup and start them. DecoderThread A infinite loop, to wait for a available buffer released by the ScannerQWidgetSink every 0.5s. Copy the buffer data to it's own buffer (imgData) to avoid any change to the buffer by sink when doing decoding. Then feed this copy of buffer into ZXing engine to get decoder result. Then show on the QLabel. Screenshot under wayland (weston) desktop: Customize Camera instance Now I use the UVC camera which pluged in the USB host, which device node is /dev/video1. If you want to use CSI or other device, please change the construction parameters for ScannerQWidgetSink(): sink = new ScannerQWidgetSink(ui->widget, QString("v4l2src device=/dev/video1")); Image resolution captured and review Change the static member value of ScannerQWidgetSink class: uint ScannerQWidgetSink::CAPTURE_HEIGHT = 480; uint ScannerQWidgetSink::CAPTURE_WIDTH = 640; Preview fps and decoding frequency Find the "framerate=20/1" strings in the ScannerQWidgetSink::GstPipelineInit(), change to your fps. You also have to change the renderTimer start timeout value in the ::StartRender(). The decoding frequency is determined by renderCnt, which determine after how many preview frames showed to feed the decoder. Main window size It's fixed size of main window, you have to change the mainwindow.ui. It's easy to do in the QtCreate Designer. FAQ Why not use CSI camera in demo? Honestly, I do not have CSI camera module, it's also DNP when you buying the board on NXP.com. So a widely used UVC camera is preferred, it's also easy for you to scan QR code on your phone, your display panel etc. Why not use QCamera to do preview and capture? The QCamera class in the Qtmultimedia component uses the camerabin2 gstreamer plugin, which create a very long pipeline for different usage of viewfinder, image capture and video encoder. Camerabin2 would eat too much CPU and memory resource, take picture and recording are very very slow. The preview of 30fps would eat about 70-80% CPU loading even I hacked it using imxvideoconvert_pxp instread of software videoconvert. Finally I give up to implement the QRScanner based on QCamera. How to make sure only one instance of QT app is running? We can use QSharedMemory to create a share memory with a unique KEY. When second instance of app is started, it would check if the share memory with this KEY is created or not. If the shm is there, it means there's already one instance running, it has to exit(). But as the QT mentioned, the QSharedMemory can not be destroyed correctly when app crashed, this means we have to handle each terminate signal, and do delete by ourselves: static QSharedMemory *gShm = NULL; static void terminate(int signum) {    if (gShm) {       delete gShm;       gShm = NULL;    }    qDebug() << "Terminate with signal:" << signum;    exit(128 + signum); } int main(int argc, char *argv[]) {    QApplication a(argc, argv);    // Handle any further termination signals to ensure the    // QSharedMemory block is deleted even if the process crashes    signal(SIGHUP, terminate ); // 1    signal(SIGINT, terminate ); // 2    signal(SIGQUIT, terminate ); // 3    signal(SIGILL, terminate ); // 4    signal(SIGABRT, terminate ); // 6    signal(SIGFPE, terminate ); // 8    signal(SIGBUS, terminate ); // 10    signal(SIGSEGV, terminate ); // 11    signal(SIGSYS, terminate ); // 12    signal(SIGPIPE, terminate ); // 13    signal(SIGALRM, terminate ); // 14    signal(SIGTERM, terminate ); // 15    signal(SIGXCPU, terminate ); // 24    signal(SIGXFSZ, terminate ); // 25    gShm = new QSharedMemory("QRScannerNXP");    if (!gShm->create(4, QSharedMemory::ReadWrite)) {       delete gShm;       qDebug() << "Only allow one instance of QRScanner";       exit(0);    } .....

When streaming, if you want to play a streaming URL, it can be inconvenient if the browser cannot recognize the URL as a media stream and downloads the content rather than using Gallery to play it. To create this kind of media streaming, you need to write an apk to use VideoView to play the URL/media stream from the console. Here is the command of how to play a media file or network stream from console. Gingerbread am start -n com.cooliris.media/com.cooliris.media.MovieView -d "<URL>"       The URL can be file position or network stream URL, such as: you can play a local file by: am start -n com.cooliris.media/com.cooliris.media.MovieView -d "/mnt/sdcard/test.mp4" You can also play a http stream by: am start -n com.cooliris.media/com.cooliris.media.MovieView -d "http://v.iask.com/v_play_ipad.php?vid=76710932" Or play a rtsp stream by: am start -n com.cooliris.media/com.cooliris.media.MovieView -d "rtsp://10.0.2.1:554/stream" ICS am start -n com.android.gallery3d/com.android.gallery3d.app.MovieActivity -d "<URL>"        The URL has the same definition of Gingerbread.

This PDF is training material for showing examples on video encoding, video decoding, video streaming on an i.MX53QSB board.

Test digital zoom with ipu for camera preview.   Board :sarbre-sd (imx6dq) BSP   : android 13.4ga In the above flow, one frame buffer is processed in four steps at camera preview. Add the step to change the frame buffer before step 4 , the added step which  zoom one preview frame.   The figure below shows the crop function of ipu lib, we use this function scale the frame.   Test result: preview zoom levle 0:   preview zoom level max:     When taking pictures with 5M pixels and the zoom is over level 1, the picture size is not 2592x1944 but 2016x1512. The underlying reason for it is that ipu crop function only supports the 2048x2048 maximum output .   Thumbnails of test result :  

The following document contains a list of document, questions and discussions that are relevant in the community based on amount of views. If you are having a problem, doubt or getting started in i.MX processors, you should check the following links to see if your doubt is in there. Yocto Project Freescale Yocto Project main page‌ Yocto Training - HOME‌ i.MX Yocto Project: Frequently Asked Questions‌ Useful bitbake commands‌ Yocto Project Package Management - smart  How to add a new layer and a new recipe in Yocto  Setting up the Eclipse IDE for Yocto Application Development Guide to the .sdcard format  Yocto NFS &amp; TFTP boot  YOCTO project clean  Yocto with a package manager (ex: apt-get)  Yocto Setting the Default Ethernet address and disable DHCP on boot.  i.MX x Building QT for i.MX6  i.MX6/7 DDR Stress Test Tool V3.00  i.MX6DQSDL DDR3 Script Aid  Installing Ubuntu Rootfs on NXP i.MX6 boards  iMX6DQ MAX9286 MIPI CSI2 720P camera surround view solution for Linux BSP i.MX Design&amp;Tool Lists  Simple GPIO Example - quandry  i.MX6 GStreamer-imx Plugins - Tutorial &amp; Example Pipelines  Streaming USB Webcam over Network  Step-by-step: How to setup TI Wilink (WL18xx) with iMX6 Linux 3.10.53  Linux / Kernel Copying Files Between Windows and Linux using PuTTY  Building Linux Kernel  Patch to support uboot logo keep from uboot to kernel for NXP Linux and Android BSP (HDMI, LCD and LVDS)  load kernel from SD card in U-boot  Changing the Kernel configuration for i.MX6 SABRE  Android  The Android Booting process  What is inside the init.rc and what is it used for.  Others How to use qtmultimedia(QML) with Gstreamer 1.0

BSP version: 11.03 Multimedia Package version: 11.03 1. Install BSP and Multi Media package (11.03 release) 2. Avoid Display Timeout: append the following line to rootfs/etc/oprofile: echo -e -n '\033[9]' > /dev/tty0 3. Set VGA port as the primary display in the kernel command line: video=mxcdi1fb:GBR24,VGA-XGA di1_primary vga 4. Connect a VGA monitor and WVGA display to the MX53 Quick Start 5. Boot Linux on MX53 Quick Start board (NFS is used in this example) 6. Unblank WVGA display (fb1): $ echo 0 > /sys/class/graphics/fb1/blank 7. On the target enter into /dev/shm directory. If the following files are present: vss_lock vss_shmem ,delete them. 8. On your host, edit the ltib/rootfs/usr/share/vssconfig as following: vss device definition Master=VGA, Slave=WVGA master display [VGA] type = framebuffer format = RGBP fb_num = 2 main_fb_num = 0 vs_max = 4 slave display [WVGA] type = framebuffer format = RGBP fb_num = 1 vs_max = 4 9. Run the Gstreamer pipeline below: gst-launch filesrc location=file.mp4 ! qtdemux ! mfw_vpudecoder ! mfw_isink display=VGA display-1=WVGA Video is played on the VGA and WVGA panels. A 720p file can be played at the same time.

                                                                                         Watch the Freescale i.MX team boot up Android 5.0 Lollipop in i.mx6 application processors—在线播放—优酷网，视频高清在线观看 The Freescale i.MX Android team has booted up Android 5.0 Lollipop in the SABRE platform for i.mx6 series. Google pushed all of the latest source for its Android release to AOSP on Nov. 5, and the Freescale Android Team started their work. With the previous 6 days to boot Android Lollipop up, the Freescale i.MX Android team enabled the basic features like connectivity, audio/video playback, sensors, inputs and display on day 7! You can see the some changes in the demo video at the beginning of the post. The Freescale i.MX Android team has closely followed almost every version of Android since it is released by AOSP and has good experience on it. Below are some snapshots and pictures for the Android Lollipop.

Audio transcoding # Transcode the input file into MP3 gst-launch filesrc location=media_file typefind=true ! beepdec ! mfw_mp3encoder ! matroskamux ! filesink location=output_audio_file.mk Audio transcoding + streaming # Transcode the input file into MP3 and stream it # On host, get the transcoded audio data gst-launch udpsrc port=8880 ! <CAPS_FROM_THE_TARGET> ! queue ! ffdec_mp3 ! alsasink # where <CAPS_FROM_THE_TARGET> can be something like 'audio/mpeg, mpegversion=(int)1, layer=(int)3, rate=(int)44100, channels=(int)2' # run the pipeline and check the caps gst-launch filesrc location=media_file typefind=true ! queue ! beepdec ! mfw_mp3encoder ! udpsink host=10.112.103.77 port=8880 -v Video Transcoding* # Transcode the input file into AVC (H-264) gst-launch filesrc location=media_file typefind=true ! aiurdemux name=demux ! queue ! vpudec ! vpuenc codec=avc ! matroskamux name=mux ! filesink location=output_media_file.mk Video Transcoding + scaling* # Transcode the input file into AVC (H264) and rescale video to 480p gst-launch filesrc location=media_file typefind=true ! aiurdemux name=demux ! queue ! vpudec ! mfw_ipucsc ! 'video/x-raw-yuv, width=(int)720, height=(int)480' ! vpuenc codec=avc ! matroskamux name=mux ! filesink location=output_media_file.mk Video transcoding + scaling + streaming* # NOTE: Run the pipelines in the presented order # On host, get the transcoded+scaled video data $ gst-launch udpsrc port=8888 ! <CAPS_FROM_THE_TARGET> ! queue ! ffdec_h264 ! xvimagesink # On target, run the pipeline and check the caps gst-launch filesrc location=media_file typefind=true ! aiurdemux name=demux ! queue ! vpudec ! mfw_ipucsc ! 'video/x-raw-yuv, width=(int)720, height=(int)480' ! vpuenc codec=avc ! udpsink host=$HOST_IP port=8888 -v * There is a limit for the number of pipelines which can be run simultaneously, for high resolution input files, at most two for 1080p and four for 720p.

This document describes the i.MX 8MM EVK mini-SAS connectors features on Linux and Android use cases, covering the supported daughter cards, the process to change Device Tree (DTS) files or Boot images, and enable these different display options on the board.

Video decoding gst-launch filesrc location=sample.mp4 ! qtdemux ! ffdec_h264 ! mfw_v4lsink Notes: On LTIB BSP 3.0.35_4.0.0, prep the package and apply the attached patch on top, then build. On Yocto, the easy way to add the gst-ffmpeg package is by adding these two lines on the conf/local.conf file: IMAGE_INSTALL_append = " gst-ffmpeg" LICENSE_FLAGS_WHITELIST = 'commercial'