Anyone who want to use this solution should get reference design and firmware from Lontium. Hardware Here is the block diagram of LT9611UXC Demo Board. As the MIPI port of our EVK can provide 5V, 3V3 and 1V8.We can remove useless DC-DC chips from reference design. Below is the LT9611UXC Demo Board. Software Download the firmware into LT9611UXC. In Linux side, we need to drive the MIPI to output signals with standard timings of 1080P. Panel type diff --git a/arch/arm64/boot/dts/freescale/imx8mp-evk.dts b/arch/arm64/boot/dts/freescale/imx8mp-evk.dts index 1732b5c72380..c6a829be541f 100644 --- a/arch/arm64/boot/dts/freescale/imx8mp-evk.dts +++ b/arch/arm64/boot/dts/freescale/imx8mp-evk.dts @@ -696,13 +716,17 @@ &ldb_phy { &mipi_dsi { status = "okay"; + panel@0{ + compatible = "nxp,lt9611uxc"; + reg = <0>; + status = "okay"; }; }; &snvs_pwrkey { diff --git a/drivers/gpu/drm/panel/panel-simple.c b/drivers/gpu/drm/panel/panel-simple.c index 4f78bbf63f33..90d99f12515b 100644 --- a/drivers/gpu/drm/panel/panel-simple.c +++ b/drivers/gpu/drm/panel/panel-simple.c @@ -4997,6 +4997,34 @@ struct panel_desc_dsi { unsigned int lanes; }; +static const struct drm_display_mode lt9611_panel_mode = { + .clock = 148500, + .hdisplay = 1920, + .hsync_start = 1920 + 88, + .hsync_end = 1920 + 88 + 44, + .htotal = 1920 + 88 + 44 + 148, + .vdisplay = 1080, + .vsync_start = 1080 + 4, + .vsync_end = 1080 + 4 + 5, + .vtotal = 1080 + 4 + 5 + 36, +}; + +static const struct panel_desc_dsi lt9611_panel = { + .desc = { + .modes = &lt9611_panel_mode, + .num_modes = 1, + .bpc = 8, + .size = { + .width = 62, + .height = 110, + }, + .connector_type = DRM_MODE_CONNECTOR_DSI, + }, + .flags = MIPI_DSI_MODE_VIDEO_HSE | MIPI_DSI_MODE_VIDEO | MIPI_DSI_MODE_NO_EOT_PACKET | MIPI_DSI_MODE_VIDEO_SYNC_PULSE, + .format = MIPI_DSI_FMT_RGB888, + .lanes = 4, +}; + static const struct drm_display_mode auo_b080uan01_mode = { .clock = 154500, .hdisplay = 1200, @@ -5201,6 +5229,9 @@ static const struct panel_desc_dsi osd101t2045_53ts = { static const struct of_device_id dsi_of_match[] = { { + .compatible = "nxp,lt9611uxc", + .data = &lt9611_panel, + },{ .compatible = "auo,b080uan01", .data = &auo_b080uan01 }, {

On behalf of Gopise Yuan. In case some customer may make mistake in mechanical design, they may need to flip/mirror the screen. DPR in DPU can do this in a simple way. This patch demonstrate how to enable VFLIP and HFLIP in DPR to do a V+H flip (=180 rotate) of the screen

This is simple known-how for how to implement "boot animation" with DRM under i.MX8/X + Linux:   Code to refer to: ========================================================================= 1. kmscube: Either open source one or the customized on for i.MX will be OK: https://cgit.freedesktop.org/mesa/kmscube/ https://source.codeaurora.org/external/imx/kmscube-imx/ 2. Android display HAL: KmsDisplay.cpp   Known-how: ========================================================================= 1. Only one application can grab the master role of the DRM device. If need to control DRM from two applicaiton simultanously, possible solution:     A, Use "controlD" node instead of "card" node in /dev/dri/. This requires L4.14 or before. This device node was removed by two commits in L4.14.x:           8a357d10043c75e980e7fcdb60d2b913491564af           6449b088dd51dd5aa6b38455888bbf538d21f2fc     Can be brought back by reverting these two commits in L4.14.98.     B, Use framebuffer emulator to emulate a FB device (/dev/fb0). (not recommended due to lack of vsync). 2. Some kernel functions will re-config the DRM device during boot. This will cause display abnormal after user application has configured the DRM device. Better to disable these kernel features:       CONFIG_DRM_FBDEV_EMULATION       CONFIG_FRAMEBUFFER_CONSOLE 3. Use atomic mode of KMS API instead of legacy mode for any dynamically screen drawing application, such as video, game and etc. Atomic mode will have much better performance compare to legacy mode. The kmscube has sample code for both mode. 4. Better to do commit checking before doing any real commit, especially when doing display during boot. Sometimes some internal component in DRM is not fully ready after card device is present.       DRM_MODE_ATOMIC_ALLOW_MODESET 5. If video playback will be used, some points to remind:     a, Sample code for direct video decoding (in unit-test): imx-test/test/mxc_v4l2_vpu_test/     b, VPU in i.MX8/X only support tiled NV12 output and it has pixel alignment requirement (128). Need to use CPU or G2D to do un-tile, CSC and cropping. Sample code: <android>/vendor/nxp/fsl_imx_omx/OpenMAXIL/src/component/v4l2_common/G2dProcess.cpp If using G2D under Linux, it will support un-tile directly (through OpenCL internally).

  From L5.4 BSP, the iMX8QM HDMI RX feature is removed from BSP, but it is added back in L5.10.52 2.1.0 BSP. The followed is the detail steps to use HDMI RX.   We need enable the followed kernel config to make hdmirx driver work:     CONFIG_IMX8_MEDIA_DEVICE=y     CONFIG_MHDP_HDMIRX=y apply the attached kernel patch. put hdmi firmware “hdmirxfw.bin” and “hdmitxfw.bin” to SD card’s FAT partition test command:     gst-launch-1.0 v4l2src device=/dev/video2 ! autovideosink   Note: To test the hdmi feature, the display should also use the HDMI TX. And in Uboot, to load the hdmirx firmware, we can run the followed commands first, then run the "boot" command:     run loadhdprx     hdprx load 0x9c800000     setenv fdt_file imx8qm-mek-hdmi-rx.dtb  

This is based on L5.10.35 BSP where you have to install QT static build: Qt 5.15 static build: Assuming your sysroot is at "/sysroot-cross" and your toolchain is at "/Toolchain" your qt-source is at /Qt-5.15 PATH=/sysroot-cross/bin:/sysroot-cross/sbin:/Toolchain/bin mkdir /Qt-5.15/mkspecs/qws/linux-imx6-g++ create in this dir the textfile "qmake.conf" with this content: ####################### snip qmake.conf ############################## include(../../common/linux.conf) include(../../common/qws.conf) # modifications to g++.conf QMAKE_CC                = arm-linux-gnueabi-gcc QMAKE_CFLAGS            = -pipe -isystem /sysroot-cross/include -isystem /sysroot-cross/usr/include QMAKE_CXX               = arm-linux-gnueabi-g++ QMAKE_CXXFLAGS          = -pipe -isystem /sysroot-cross/include -isystem /sysroot-cross/usr/include QMAKE_INCDIR            = /sysroot-cross/include /sysroot-cross/usr/include QMAKE_LIBDIR            = /sysroot-cross/lib /sysroot-cross/usr/lib QMAKE_LINK              = arm-linux-gnueabi-g++ QMAKE_LINK_SHLIB        = arm-linux-gnueabi-g++ QMAKE_LFLAGS            = -L/sysroot-cross/lib -L/sysroot-cross/usr/lib -Wl,-rpath-link -Wl,/sysroot-cross/lib QMAKE_LFLAGS           += -Wl,-rpath-link -Wl,/sysroot-cross/usr/lib #Opengl QMAKE_INCDIR_OPENGL = /Vivante/include QMAKE_INCDIR_OPENGL += /Vivante/include/GL QMAKE_INCDIR_OPENGL += /Vivante/include/EGL QMAKE_INCDIR_OPENGL += /Vivante/include/GLES2 QMAKE_LIBDIR_OPENGL = /Vivante/lib QMAKE_INCDIR_OPENGL_ES1 = $$QMAKE_INCDIR_OPENGL QMAKE_LIBDIR_OPENGL_ES1 = $$QMAKE_LIBDIR_OPENGL QMAKE_INCDIR_OPENGL_ES1CL = $$QMAKE_INCDIR_OPENGL QMAKE_LIBDIR_OPENGL_ES1CL = $$QMAKE_LIBDIR_OPENGL QMAKE_INCDIR_OPENGL_ES2 = /Vivante/include QMAKE_INCDIR_OPENGL_ES2 += /Vivante/include/EGL QMAKE_INCDIR_OPENGL_ES2 += /Vivante/include/GLES2 QMAKE_LIBDIR_OPENGL_ES2 = $$QMAKE_LIBDIR_OPENGL QMAKE_INCDIR_EGL = $$QMAKE_INCDIR_OPENGL_ES2 QMAKE_LIBDIR_EGL = $$QMAKE_LIBDIR_OPENGL QMAKE_LIBS_EGL = -lEGL -lGAL -lGLESv2 -lGLES_CM QMAKE_LIBS_OPENGL_ES2 = -lEGL -lGAL -lGLESv2 -lGLES_CM QMAKE_LIBS_OPENGL = -lEGL -lGAL -lGLESv2 -lGLES_CM QMAKE_LIBS_OPENGL_QT = -lEGL -lGAL -lGLESv2 -lGLES_CM QMAKE_LIBS_OPENGL_ES1 = QMAKE_LIBS_OPENGL_ES1CL = # modifications to linux.conf QMAKE_AR                = arm-linux-gnueabi-ar cqs QMAKE_OBJCOPY           = arm-linux-gnueabi-objcopy QMAKE_STRIP             = arm-linux-gnueabi-strip QMAKE_CFLAGS_RELEASE   = -pipe -isystem /sysroot-cross/include -isystem /sysroot-cross/usr/include load(qt_config) ####################### snip qmake.conf ############################## create in the same dir the text file "qplatformdefs.h" ####################### snip qplatformdefs.h ############################## #include "../../linux-g++/qplatformdefs.h" ####################### snip qplatformdefs.h ############################## now goto dir /Qt-5.15 cd /Qt-5.15 call configure with ./configure -opensource -confirm-license -release -no-rpath -no-fast \     -no-sql-ibase -no-sql-mysql -no-sql-odbc -no-sql-psql -no-sql-sqlite2 \     -no-qt3support -no-mmx -no-3dnow -no-sse -no-sse2 -no-sse3 -no-ssse3 \     -no-sse4.1 -no-sse4.2 -no-avx -no-optimized-qmake -no-nis -no-cups -pch \     -reduce-relocations -force-pkg-config -prefix /usr -no-armfpa -make libs \     -nomake docs -little-endian -embedded armv6 -qt-decoration-styled \     -depths all -xplatform qws/linux-imx6-g++ -iconv -largefile -qt-gfx-linuxfb \     -qt-gfx-multiscreen -qt-mouse-pc -qt-mouse-linuxinput -qt-libpng \     -plugin-gfx-directfb -system-zlib -no-accessibility -no-gfx-transformed \     -no-gfx-qvfb -no-gfx-vnc -no-kbd-tty -no-kbd-linuxinput -no-kbd-qvfb \     -no-mouse-linuxtp -no-mouse-tslib -no-mouse-qvfb -no-libmng -no-libtiff \     -no-gif -no-libjpeg -no-freetype -no-stl -no-glib -no-openssl -no-egl \     -no-xmlpatterns -no-exceptions -no-multimedia -no-audio-backend -no-phonon \     -no-phonon-backend -no-webkit -no-script -no-scripttools -no-svg -no-script \     -no-declarative -no-sql-sqlite -no-qdbus -no-opengl -static -nomake tools \     -nomake examples -nomake demos when configuring is finished call make after a looong time, when everything goes right, we have a staticly compiled Qt. DO NOT call "make install". We will install manually: copy from /Qt-5.15/bin the files moc, uic, rcc and qmake to somewhere in PATH, eg. /sysroot-cross/bin copy the contents of dir /Qt-5.15/mkspecs to /sysroot-cross/usr/mkspec copy the contents of dir /Qt-5.15/plugins to /sysroot-cross/usr/plugins copy the contents of dir /Qt-5.15/include to /sysroot-cross/usr/include copy the contents of dir /Qt-5.15/lib to /sysroot-cross/usr/lib Test application camtest: if you don't have/want directfb plugin remove from camtest.pro the lines LIBS += -L/sysroot-cross/usr/plugins/gfxdrivers QTPLUGIN += QDirectFBScreen and the lines from main.cpp #include <QtPlugin> Q_IMPORT_PLUGIN(qdirectfbscreen) generate makefile by typing /sysroot-cross/bin/qmake -spec /sysroot-cross/usr/mkspecs/qws/linux-imx6-g++ camtest.pro then make you should set and activate your framebuffers with this script ################# snip ################################ fbset -fb /dev/fb0 -g 1024 768 1024 2304 16 echo -n 0 > /sys/class/graphics/fb0/blank fbset -fb /dev/fb1 -g 1024 768 1024 1536 32 echo -n 0 > /sys/class/graphics/fb1/blank modprobe galcore modprobe uvcvideo modprobe mxc_v4l2_capture ################# snip ################################ if you use directfb then your /etc/directfbrc file should look like this: ######################## snip /etc/directfbrc ############# system=fbdev fbdev=/dev/fb1 mode=1024x768 depth=32 pixelformat=ARGB no-cursor window-surface-policy=systemonly ######################## snip /etc/directfbrc ############# to start the application with directfb: ./camtest -qws -display directfb without directfb using linuxfb: ./camtest -qws -display linuxfb:/dev/fb1 Notes about application: 1. The application shows 2 webcams in background-framebuffer (BG-FB). The foreground-framebuffer (FG-FB) shows the qt-gui. FG-FB is configured to be fully opaque and uses color-keying. On the BG-FB one cam is overlayed on the other cam using IPU. Optimization possibilities: the app copies the frames from the cams with memcpy. This wouldn't be necessary, when the kernel usb-webcam interface (uvc) would support V4L2_MEMORY_USERPTR method. through this way, you could pass the mapped IPU mmapped inbufs directly to v4l2 output buffers. If you get errors like NOSPC (-28) from uvc, this is a limitation of USB. My board is a MX6QSabre, where the two webcams are connected to the same usb-controller. With both webcams I had to limit the frame size to 320x250 and 160x120 at 25Hz. You might try higher res if you have other type of webcams (not usb). Have fun  

Use case: iMX8QXP system can be a video input source to another system.   Hardware Pins： LCDIF_D00 ~ LCDIF_D07 LCDIF_CLK LCDIF_VSYNC LCDIF_HSYNC LCDIF_EN   Reference patch： It is based on L5.4.70_2.3.0 GA BSP.  File: L5.4.70_2.3.0-iMX8QXP-LCDIF-add-YUV422-8-bits-output.patch Customer can change the timing parameters in file "panel-lcdif-yuv422.c" as needed, the default timing is a 1280x720 P30 mode: static const struct display_timing yuv422_lcd_timing = {     .pixelclock = { 74250000, 74250000, 74250000 },     .hactive = { 1280, 1280, 1280 },     .hfront_porch = { 220, 220, 220 },     .hback_porch = { 110, 110, 110 },     .hsync_len = { 40, 40, 40 },     .vactive = { 720, 720, 720 },     .vfront_porch = { 20, 20, 20 },     .vback_porch = { 5, 5, 5 },     .vsync_len = { 5, 5, 5 },     .flags = DISPLAY_FLAGS_DE_HIGH, };   Test application: drm_test_yuv.zip: it can set framebuffer to UYVY mode, in this case, no CSC is needed, the data in framebuffer memory will be same as output on display data interface. drm_test_rgb.zip: it can set framebuffer to RGBA mode, in this case, RGB to YUV CSC is needed, application can draw RGB data into framebuffer as normal, the LCDIF will convert it to YUV422 format on the fly, then output the YUV data to display interface.    

This doc share one OpenGL ES sample code, it is running on i.MX8 MEK board with QNX SDP7.1. HW: i.MX8 MEK board, HDMI display SW: QNX SDP7.1, i.MX8 MEK board BSP, and this sample code   This sample code will draw 3D object model, and with some animation. Reference: https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-8-processors/i-mx-8-family-arm-cortex-a53-cortex-a72-virtualization-vision-3d-graphics-4k-video:i.MX8 https://github.com/NXPmicro/gtec-demo-framework https://github.com/syoyo/tinyobjloader-c https://github.com/nothings/stb https://3dhaupt.com/futuristic-car-game-ready-download/ https://wallpapersafari.com/w/Y5JZNh https://www.pngwing.com/en/free-png-ysaus https://www.shadertoy.com/view/Ms2SWW#

For iMX6DQ, there are two IPUs, so they can support up to 4 cameras at the same time. But the default BSP can only support up to two cameras at the same time. The attached patch can make the BSP support up to 4 cameras based on 3.10.53 GA 1.1.0 BSP.   The 4 cameras can be: - 1xCSI, 3xMIPI - 2xCSI, 2xMIPI - 4xMIPI   For 4xMIPI case, the four cameras should be combined on the single MIPI CSI2 interface, and each camera data should be transfered on a mipi virtual channel.   In this patch, we given the example driver for Maxim MAX9286, it was verified working on iMX6DQ SabreAuto board. The input to MAX9286 is four 720P30 cameras. The verified camera boards:     (1) Onsemi AR0140+AP0101+MAX9271 boards.     (2) OmniVision OV10635+MAX9271 boards.   The MIPI CSI2 CVBS camera surround view solution can be found at: iMX6DQ ISL79985/79987 MIPI CSI2 CVBS camera surround view solution for Linux BSP The MIPI CSI2 CVBS HD camera surround view solution can be found at: iMX6DQ TP2854 MIPI CSI2 720P CVBS camera surround view solution for Linux BSP   The kernel patches: 0001-IPU-update-IPU-capture-driver-to-support-up-to-four-.patch      Updated IPU common code to support up to four cameras.   0002-Add-Max9286-support-on-SabreAuto-board-which-can-sup.patch      MAX9286 driver, it includes MAX9271, AP0101 and AR0140 drivers.   0003-Remove-the-page-size-align-requirement-for-v4l2-capt.patch      With this patch, the mxc_v4l2_tvin test application can use overlay framebuffer as V4l2 capture buffer directly.   0004-Max9286-skip-AP0101-camera-re-initialization.patch      If the camera board's power had been kept after initialized, this patch will bypass the re-initialization to reduce the start up time.   0005-Max9286-set-I2C-speed-to-400Kbps.patch     Set I2C to 400Kbps to reduce the AP0101+AR0140 initialization time.   0006-Max9286-add-retry-for-MAX9271-I2C-access.patch     Added retry for MAX9271 I2C access.   0007-Max9286-Add-support-for-OV10635-camera.patch     Updated code for OV10635 camera.   0008-Max9286-support-auto-detect-camera-number.patch     Make the Max9286 driver can detect the camera number automatically.     How to builld the kernel with MAX9286 support:       make imx_v7_defconfig       make menuconfig (In this command, you should select the MAX9286 driver:             Device Drivers  --->                   <*> Multimedia support  --->                         [*]   V4L platform devices  --->                               <*>   MXC Video For Linux Video Capture                                       MXC Camera/V4L2 PRP Features support  --->                                           <*>Maxim max9286 GMSL Deserializer Input support                                               Select Camera Sensor (OmniVision OV10635 camera sensor)  // Or (Onsemi AP0101 and AR0140 camera sensor)                                           <*>mxc VADC support                                           <*>Select Overlay Rounting (Queue ipu device for overlay library)                                           <*>Pre-processor Encoder library                                           <*>IPU CSI Encoder library)       make zImage       make dtbs   The built out image file:       arch/arm/boot/dts/imx6q-sabreauto.dtb       arch/arm/boot/zImage   "mxc_v4l2_tvin_max9286.tar.gz" is the test application, test command to capture the four cameras and render on 1080P HDMI display: /mxc_v4l2_tvin.out -ol 0 -ot 0 -ow 960 -oh 540 -d 1 -x 0 -g2d & /mxc_v4l2_tvin.out -ol 960 -ot 0 -ow 960 -oh 540 -d 1 -x 1 -g2d & /mxc_v4l2_tvin.out -ol 0 -ot 540 -ow 960 -oh 540 -d 1 -x 2 -g2d & /mxc_v4l2_tvin.out -ol 960 -ot 540 -ow 960 -oh 540 -d 1 -x 3 -g2d &   Some hardware check point on AR0140+AP0101+MAX9271 camera board (Please get MAX9286 and OV10635 schematics from Maxim): 1. In this patch, MAX9286's I2C address is 0x4D, so ADD0 and ADD1 should be connected to high. AP0101's I2C address is 0xBA, so SADDR should be connected to high.   2. AP0101's DOUT0~DOUT7 should be connected to MAX9271's DIN7~DIN0, the order should be switched, MSB connected to LSB.   3. MAX9271's GPO pin should be connected to AP0101's FRAME_SYNC pin. The pull down resistance on FRAME_SYNC pin should not be 0 ohm.   Some known limitation: 1. AP0101's VSYNC invalid time, last video line's HSYNC to VSYNC porch's max value is 255 pixel clocks, it is not enough for MAX9286 to generate the Frame End MIPI packets for each camera. So in order to let iMX6DQ to capture 1280x720 video for each camera, we had let AP0101 output 1280*724 frame size, and iMX6 will only capture 720 lines, the remained video data and Frame End will be ignored. This solution will not impact the function, but there will be "Error matching Frame Start with Frame End for Virtual Channel x" error reported from iMX6 MIPI_CSI_ERR1 register. Maxim suggested to use MAX96705 to relace the MAX9271, it can delay the VSYNC invalid time, then the MIPI error will be fixed.     2015-11-17 update: Updated for OV10635 camera support. File: L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2015-11-17.zip   2015-12-04 update: File: L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2015-12-04.zip Added patch 0009-Max9286-updated-PCLK-edge-setting-for-OV10635.patch to correct the OV10635 PCLK edge setting     2016-03-07 update: File L3.14.38_GA_MAX9286_Surroundview_Patch_2016-03-07.zip Added kernel patch for L3.14.38 GA 1.1.0 BSP.   2016-07-26 update: Files: L3.10.53_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip; L3.14.38_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip; L3.14.52_GA1.1.0_MAX9286_Surroundview_Patch_2016-07-26.zip. Added gstreamer support. Added MAX96705 support. Added patch for L3.14.52_GA1.1.0.   2017-12-11 update: Added CVBS surround view link: iMX6DQ TP2854 MIPI CSI2 720P CVBS camera surround view solution for Linux BSP     2021-04-26 update: Some customer reported, when system loading is heavy, sometimes, some camera will flicker left and right. It is caused by SFMC FIFO data lost. The original patch used IDMAC 0 and IDMAC 1 for two cameras on one IPU, this is not the best setting.  IDMAC 1 is fixed to use 1/4 SMFC FIFO and it will cause IDMAC 0 to use 1/4 SMFC FIFO too. And another 1/2 of SMFC FIFO can't be used in this case. Some code update to improve it: For each IPU, please use IDMAC 0 and IDMAC 2 to capture the two cameras. This needs change the hard coding in "drivers\media\platform\mxc\capture\ipu_csi_enc.c", "CSI_MEM1" and "IPU_IRQ_CSI1_OUT_EOF" should be changed to "CSI_MEM2" and "IPU_IRQ_CSI2_OUT_EOF". In this case, all SMFC FIFO can be used. And in "ipu_common.c", function ipu_probe(), the followed code should be changed to make IDMAC2 use high priority too. /* Set sync refresh channels and CSI->mem channel as high priority */ - ipu_idmac_write(ipu, 0x18800003L, IDMAC_CHA_PRI(0)); + ipu_idmac_write(ipu, 0x1880000FL, IDMAC_CHA_PRI(0));

In this doc will show how to use i.MX8QXP DPU do image warp.   SW: i.MX Linux BSP L5.4.24_2.1.0 bsp release and patch in this doc HW: i.MX8QXP MEK board, ov5640 camera, HDMI display   Introduction Image Warping is the process of digitally manipulating image data such that the image’s projection precisely matches a specific projection surface or shape.   i.MX8QXP DPU controller could do image warp work by its blit engine and display engine. I choose to enable blit engine’s fetchwarp9 unit to do warp work. Check i.MX8QXP RM, Blit Engine support Image Warp as: “Performs a re-sampling of the source image with any pattern. The sample point positions are read from a compressed coordinate buffer.” So you need prepare two input buffers, one buffer store original image data, the other buffer store resample point coordinate, DPU blit engine will read that two buffer by fetchwarp9 unit, then output result image buffer which contain warped image data. Note i.MX8QXP DPU blit engine fetchwarp9 unit, for the input original image buffer, support RGB and YUV 4:4:4 format. The resample point coordinate buffer contents is depend on what kind warp transformation in your use case; and for each resample point coordinate format check i.MX8QXP RM fecthwarp unit description as below. In this doc, using the 2xs12.4 format, each point x coordinate use (12+4) bit, same as y coordinate.   For DPU fetchwarp9 unit, to enable it work for image warp, check i.MX8QXP RM:   2.Patch notes and test code imx8-dpu-warp-kernel.diff contain the kernel side change for drm ioctl api permission and add vmap function of ion dma_buf_ops. libg2d.so contain the binary for adding warp feature. g2d.h is header file which add define for G2D_WARP and G2D_YUV4. imx8-ov5640-dpu-warp-render.c is a sample code which show how to call g2d lib to image warp, need open the G2D_WARP flag. And this code contain some example calculate the coordinate buffer of rotate, swirl, barrel distortion, affine transformation, perspective transformation, wave transformation. And this code will show read camera input frame then add warp process , then render warp image frame to display.   The test cmd usage as below, read 1080P frame from ov5640 camera, do warp then render warp image to drm plane. Note as dpu fetchwarp9 unit support YUV 4:4:4 input image frame, so below cmd need set parameter YUV4, which will ask ISI driver output YUV 4:4:4 image frame. imx8-ov5640-dpu-warp-render  -i /dev/video0 -f YUV4  -S 1920,1080  -M imx-drm -p 91:38 -F XB24  -b 6  -e g2d  -t 5         -i <video-node> set video node (default: /dev/video0)         -f <fourcc>     set input format using 4cc         -S <width,height>       set input resolution         -s <width,height>@<left,top>    set crop area         -M <drm-module> set DRM module         -o <connector_id>:<crtc_id>:<mode>      set a mode         -p <connector_id>:<crtc_id>     output to a plane         -F <fourcc>     set output format using 4cc         -t <warptype>   set 0 neutual 1 rotate 2 swirl 3 divisionmodel 4 affine 5 perpsptive 6 wave         -b buffer_count set number of buffers        3.Example original image:                     Reference: https://www.nxp.com/webapp/Download?colCode=IMX8DQXPRM https://www.nxp.com/webapp/Download?colCode=L5.4.24_2.1.0_MX8QXPC0&appType=license https://en.wikipedia.org/wiki/Image_geometry_correction https://lists.freedesktop.org/archives/dri-devel/2012-March/019778.html https://store.kde.org/p/1246558 https://github.com/ImageMagick/ImageMagick        

BSP: L5.4.47-2.2.0-rc2 Board: imx8QM B0 HW:  LVDS2HDMI , MIPIDSI2HDMI. It is the porting of i.MX8QM dpu loopback to isi. to the 5.4.y, with the addition of the MIPI-DSI loopback and the HDMI loopback.  Overview of the DC capture configuration: For enabling the capture: only DC 0 Stream 0  and DC 1 Stream 1 can be captured The pixel link Master address should be set to 3 because the Receiver Address at ISI is 3 and can't be changed. To continue displaying the stream, the Receiver Address at LVDS and DSI or HDMI should be changed to 3. It is possible to change the RA by using GPIO of the modules.   Patches: Create V4L2 device enabling the capture of by the ISI of DC loop-backs. Enable ISI capture from DSI 0 / LVDS 1 in 1920x1080 (at the same time.) Enable ISI capture from HDMI in 2840x2160 (half with even pixel) in 1920x2160. While capturing with the ISI, the captured screen continue to be displayed. Remark: Ov5640 cameras are also enabled in the same dtb. So 4 stream in 1920x1080 can be captured at the same time. Installation and gstreamer command: See readme

This document is a user guide for the GStreamer version 1.0 based accelerated solution included in all the i.MX 8 family SoCs supported by NXP BSP L5.4.24_1.1.0. Some instructions assume a host machine running a Linux distribution, such as Ubuntu, connected to i.MX 8 device. These commands were tested using Ubuntu 18.04 LTD, and while Ubuntu is not required on the host machine, other distributions have not been tested. These instructions are targeted for use with the following hardware: • i.MX 8MQ EVK • i.MX 8MN EVK • i.MX 8MN EVK • i.MX 8QXP MEK B0 • i.MX 8QM MEK B0   Release History v1.0 - Mar 2020 - Initial release. v2.0 - Sep 2020: Added the following content: - Mux/Demux Examples - Audio Examples - Image Examples - Transcode Examples - Streaming Examples - Multi-Display Examples - Scaling and Rotation Examples - Zero-copy Examples - Debug Examples Maintainers: . Marco Franchi . Pedro Jardim

[中文翻译版] 见附件   原文链接： https://community.nxp.com/docs/DOC-345148 

This document describes the i.MX 8MM EVK mini-SAS connectors features on Linux and Android use cases, covering the supported daughter cards, the process to change Device Tree (DTS) files or Boot images, and enable these different display options on the board.

This document describes all the i.MX 8 MIPI-CSI use cases, showing the available cameras and daughter cards supported by the boards, the compatible Device Trees (DTS) files, and how to enable these different camera options on the i.MX 8 boards. Plus, this document describes some Advanced camera use cases too, such as multiples cameras output using imxcompositor_g2d plugin, GStreamer zero-copy pipelines and V4L2 API extra-controls examples.

SW: Linux 4.14.98_2.2.0 bsp release , gpu sdk 5.3.0 and patch in this doc HW: i.MX8QXP MEK board Inspired by the GIMP adjust color curve tool,  I write similar gui test code for i.MX8QXP gamma curve tuning. That is on display will show one linear curve first, use mouse to change any part of this curve, after that calculate value of point on changed curve , then pass related value to i.MX8QXP DPU gammcor unit , at last the display effect will be changed by DPU. When test code start up , will draw Y=X curve on display, that will from (0,0) to (500, 500) to (1023,1023). After you use mouse drag some part of the curve, will generated a new point (x_new, y_new) which is not on original Y=X curve. Then need draw a new curve that pass  through the (0,0) , (500, 500) , (x_new, y_new), (1023,1023).  That new curve , i choose to use Catmull-Rom Splines to get it. Use Catmull-Rom Splines for approximate a curve that pass through n point, then need n+2 control point. Extra 2 point could be selected as you want ,and they will affect the shape of the curve. Catmull-Rom Splines could pass through all control point, it is C1 continuity, no convex hull property. For example, there is four control point p0, p1 , p2 , p3,  to draw the curve pass through all of them, it could divide to segment p0 to p1 , segment p1 to p2 , segment p2 to p3. For segment p1 to p2 , select four control point as p0,p1, p2, p3, use Catmull-Rom Splines as below: For segment p0 to p1 , need use four control point as p0, p0, p1, p2: For segment p2 to p3, need use four control point as p1, p2, p3, p3: In this test code,  i will use gpu vertex shader to calculate each segment of curve, then use transform feedback to read back point value of each segment to cpu side, cpu side will pass related value to DPU gammcor unit for gamma tuning.   Test steps: Apply 8qxp_dpu_gammacor_4.14.98_2.2.0.diff on linux-imx for i.MX8QXP DPU device driver. Apply dpu_gamma_curve_gpusdk5.3.0.diff on imx-gpu-sdk for build this gui test code. Update the new kernel image, and copy test code to rootfs. Run any other application first to draw some thing on screen, for example  /usr/share/cinematicexperience-1.0/Qt5_CinematicExperience. Then run gui test code in this code, S01_SimpleTriangle_Wayland. Then there will show one linear curve on display , use mouse to change the curve as you like by put mouse cursor close to the curve, press the mouse button , drag it and release the mouse button, you will see the new curve on display , and also the display effect also be changed. Reference: 1>https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-8-processors/i-mx-8x-family-arm-cortex-a35-3d-graphics-4k-video-dsp-error-correcting-code-on-ddr:i.MX8X  2>https://www.nxp.com/design/software/embedded-software/linux-software-and-development-tools/embedded-linux-for-i-mx-applications-processors:IMXLINUX?tab=Design_Tools_Tab  3>https://github.com/NXPmicro/gtec-demo-framework  4>https://en.wikipedia.org/wiki/Cubic_Hermite_spline  5>http://www.mvps.org/directx/articles/catmull

[中文翻译版] 见附件   原文链接： https://community.nxp.com/docs/DOC-342059 

This doc show how to use i.MX8QXP Display Controller GammaCor unit to tune gamma. HW: i.MX8QXP MEK board, HDMI monitor SW: i.MX Linux 4.14.98_2.2.0 BSP release, patch in this doc 1.Introduce gamma The gamma, gamma correction, gamma encoding, gamma compression , these words all related one kind operation , see wiki page of it: The device used for image capture/print/display follow this power-law. For example the camera captured image , to view this image on display device as good as original captured image : gamma encoding when camera saved sensor data to image file,  and  gamma decoding when that image file display on your PC LCD monitor. That is : 2. i.MX8QXP Display Controller Gamma Correction Unit The Gamma Correction unit position is located between Frame Gen unit and TCon unit.   More detail see below contents from i.MX8QXP RM: So GammaCor unit could be used as adjust display gamma , or brightness or contrast. To used it, need follow the steps at RM 15.9.2.4.4.8.3.   Something need to note: You need program 33 sample point value into the register, these sample point value range is from 0 to 1023. Note, first write is start sample point value , then the other is delta value: current sample point minus previous sample point value. You can use GammaCor unit on any channel of R/G/B. If you use normalized function f(x), the following formula should be used to clut[i = 0..32] = round( f(i * 32 / 1023) * 1023) 3. i.MX8QXP Linux device driver patch and test code Apply attached  patch 8qxp_dpu_gammacor_4.14.98_2.2.0.diff on Linux kernel. In the kernel patch, function dpu_gammacor_update, I choose not calculate delta value between each sample pint , let user space application calculate delta value and passed to kernel. Apply 8qxp-dpu-gammacor-modetst.diff on libdrm-imx, to get test application which is based on modetest.  Test app will read one greyscale image file 720P.rgb, put it under same folder of test application , calculate sample point value by pow function  , and calling drmModeCrtcSetGamma to pass related value to kernel,  next loop will change sample point value, and will see that greyscale image will changed on HDMI monitor. After system boot up, run below cmd to check result of test application systemctl stop weston ./gamma_show_rgba.out -P 29@32:1280x720@AB24 Reference: a>https://www.nxp.com/webapp/Download?colCode=IMX8DQXPRM b>https://www.nxp.com/webapp/Download?colCode=L4.14.98_2.2.0_MX8QXP&appType=license c> https://source.codeaurora.org/external/imx/libdrm-imx/ d> https://en.wikipedia.org/wiki/Gamma_correction

The i.MX8QuadMax SMARC System On Module integrates Dual Cortex A72 + Quad Cortex A53 Cores, Dual GPU systems, 4K H.265 capable VPU dual failover-ready display controller based i.MX8 QuadMax SoC with on SOM Dual 10/100/1000 Mbps Ethernet PHY, USB 3.0 hub and IEEE 802.11a/b/g/n/ac Wi-Fi & Bluetooth 5.0 module.

The D-PHY PLL (in the red circle in the picture below) is the PLL that drives the MIPI Clock lane. It must be set in accordance with the video to be sent to the display.   Calculating the video bandwidth The video bandwidth is calculated with the following equation: Pixels per second = Horizontal res. x Vertical res. x Frame rate x Bits per pixel Taking as example the 1080p60 OLED display RM67191: Pixels per second = 1920 x 1080 x 60 x 24 Pixels per second = 2985984000 = 2,98Gpixels/sec Pixel clock calculation The Display pixel clock can be obtained on the display driver. In this example for RM67191, the pixel clock is 132Mpixel/sec, see file: panel-raydium-rm67191.c\panel\drm\gpu\drivers - linux-imx - i.MX Linux kernel  Line 530: .pixelclock = { 66000000, 132000000, 132000000 }, Or the number can be obtained with the following equation: pixel clock = (hactive + hfront_porch + hsync_len + hback_porch) x (vactive + vfront_porch + vsync_len + vback_porch) x frame rate pixel clock = (1080 + 20 + 2 +34) × (1920 + 10 + 2 + 4) x 60 pixel clock = 132000000 (rounded up) Bit clock calculation (clock lane) The mipi-dphy bit_clk is the output clock and is calculated on file sec-dsim.c (line 1283): sec-dsim.c\bridge\drm\gpu\drivers - linux-imx - i.MX Linux kernel  Bit clock can be calculated with the following equation: bit_clk = Pixel clock * Bits per pixel / Number of lanes In the case of 1980p60 (Raydium display), It is:   bit_clk = pixel clock * bits per pixel / number of lanes bit_clk = 132000000 * 24 / 4 bit_clk = 792000000 Other important timing parameters like 'p', 'm', 's' are obtained on the table in the following header file: sec_mipi_dphy_ln14lpp.h\imx\drm\gpu\drivers - linux-imx - i.MX Linux kernel 

This doc show: on i.MX8QXP MEK board, configure ov5640 sensor(parallel interface) output 5MP(2592x1944) RAW(Bayer) data at 15fps, and Parallel Capture Subsystem and Image Sensor Interface capture RAW RGB data, and i.MX8QXP GPU debayer RAW data then display image. HW: i.MX8QXP MEK board, MEK base board (to place the parallel camera), ov5640 sensor. SW: Linux 4.14.98_2.0.0 BSP, and patches in this doc.   Configure at camera sensor side A Bayer filter is a color filter array (CFA) for arranging RGB color filters on a square grid of photosensors. The filter pattern is 50% green, 25% red and 25% blue, hence is also called BGGR, RGBG ,GRGB, or RGGB. The ov5640 has an image array capable of operating at up to 15 fps in 5 megapixel (2592x1944) resolution. OV5640 support output formats: RAW(Bayer), RGB565/555/444,CCIR656, YUV422/420, YCbCr422, and compression. To make ov5640 output 5MP RAW data at 15fps, check my kernel patch imx8-ov5640-raw-capture-driver-4.14.98_2.0.0.diff which apply on i.MX Linux 4.14.98_2.0.0 BSP kernel code. Parallel interface ov5640, use ov5640_raw_setting[] array of drivers/media/platform/imx8/ov5640_v3.c. This register setting is come from ov5640 software application note and data sheet. Configure at i.MX8QXP side The Parallel Capture Subsystem consists of the Parallel Capture Interface (BT 656) and associated peripherals. It interfaces to the Parallel CSI sensor. This allows for up to 24 RGB data bits in parallel or for RGB components on consecutive clocks (up to 10-bit color depth). The formats supported are RGB, RAW and YUV 422. Below is Parallel Capture Subsystem diagram: For RAW format data, CSI_CTRL_REG of Parallel Capture Subsystem need configured as my patch, otherwise found cannot get correct data.   The multiple input sources (MIPI CSI, Parallel Capture) captures the pixel data and feeds it to the ISI. The ISI is responsible for capturing and pre-processing the pixel data from multiple input sources and storing them into the memory. Below is ISI diagram: For RAW format data, it should be bypass any processing pipeline of ISI, just use ISI to save it to memory.   Capture test code To capture the RAW data and save it to file, check my patch imx8_ov5640_raw_captupre_test_4.14.98_2.0.0_ga.diff which apply on i.MX Linux 4.14.98_2.0.0 BSP unit test code. Note the usage is: ./imx8_cap.out -of -cam 1 -fr 15 -fmt BA81 -ow 2592 -oh 1944 -num 100   Display RAW data The RAW data cannot be displayed directly, debayer process is needed to get complete red, green, blue color for each pixel. The debayer process if run on CPU, will cost much CPU time. To save CPU time, debayer could done by GPU. The method is, captured RAW data upload to GPU as texture , then GPU will do the debayer, then full color of each pixel will be got, then display it. To upload RAW camera data to GPU with zero memory copy, i will use i.MX8QXP GPU extension GL_VIV_direct_texture. It create a texture with direct access support. API glTexDirectVIVMap,  which support mapping a user space memory or a physical address into the texture surface. The API glTexDirectVIVMap need logic and physical address of data buffer, so i will allocate data buffer from g2d lib, it is dma-buffer also get logic/physical address of buffer, then queue it as DMABUF to v4l2 capture driver, after dequeue got RAW camera data, pass it to GPU for debayer. GPU side, I will use OpenGL shader code from "Efficient, High-Quality Bayer Demosaic Filtering on GPUs". Check my patch imx8_debayer-gpusdk-5.3.0.diff which apply on i.MX GPU SDK 5.3.0 code. Note, here i only do is debayer, no extra process.   Known issue One thing is ov5640 output 5MP at 15fps, compare with output 5MP at 5fps, there are more noise of camera data at 15fps case. My debug found is, this noise seems come from ov5640 itself.   Reference: a>https://www.nxp.com/webapp/Download?colCode=IMX8DQXPRM b>https://www.nxp.com/webapp/Download?colCode=L4.14.98_2.0.0_MX8QXP&appType=license c>https://github.com/NXPmicro/gtec-demo-framework d>ov5640 data sheet e>ov5640 software application note f>Efficient, High-Quality Bayer Demosaic Filtering on GPUs https://www.semanticscholar.org/paper/Efficient%2C-High-Quality-Bayer-Demosaic-Filtering-on-McGuire/088a2f47b7ab99c78d41623bdfaf4acdb02358fb