Implementing GREY8 → NV12 Conversion Inside Kernel on i.MX8M Plus (Yocto 6.1.36 Mikeldore)

[Olivia]

Hello everyone,

I’m working with a monochrome camera on an i.MX8M Plus platform running Yocto 6.1.36 Mikeldore. The camera outputs raw grayscale frames in GREY8 pixel format.

Using this command:

v4l2-ctl -d /dev/video2 --set-fmt-video=width=1280,height=720,pixelformat=GREY --stream-mmap

I can capture video at around 108 fps, which is excellent.

However, for displaying the video, I need to convert from GREY8 to NV12 because most display sinks and hardware accelerators expect formats like NV12 for efficient rendering and colorspace handling. Without this conversion, the raw grayscale format cannot be displayed correctly or hardware-accelerated, which is why software conversion is currently required.

Currently, this conversion is done in user space using GStreamer’s videoconvert, but it causes high CPU usage and reduces streaming performance drastically — the effective fps drops from 30 fps down to about 10 fps.

My main question is:

Is it possible or recommended to implement the GREY8 → NV12 pixel format conversion inside the kernel driver or V4L2 pipeline on the i.MX8M Plus platform, ideally leveraging hardware acceleration?

I want to move this conversion to kernel space to reduce CPU load and achieve high fps streaming like the raw capture.

Any insights or pointers would be appreciated!

Thanks!

[Manuel_Salas]

Hello @Olivia 

I hope you are doing very well.

After evaluating the available hardware capabilities, I can confirm that this conversion is not supported by the G2D engine, and may also not be natively supported by the ISI block(Image Sensor Interface).

While it might be possible to implement the conversion within the V4L2 pipeline using a custom mem-to-mem driver or a software-based approach, doing so it would likely require significant development effort.

The GREY8 to NV12 conversion is not a simple pixel reformatting, it involves injecting artificial U/V chroma data to form a valid NV12 frame, which is not a typical use case for hardware accelerators optimized for full color format processing.

I hope this can helps to you.

Best regards,

Salas.

[Olivia]

Thank you for your detailed response regarding the limitations of the G2D and ISI blocks with respect to GRAY8 to NV12 conversion.

I understand that such conversion is not typically handled by the hardware accelerators, and implementing this within a V4L2 mem-to-mem pipeline would require additional development effort, especially due to the need for artificial chroma plane generation.

To help accelerate development, could you please let me know if NXP provides:

Any reference implementation of a V4L2 mem-to-mem driver (software or hardware-based)

Sample code, application notes, or Yocto recipes that demonstrate how to implement or integrate such a driver

Or any recommendation on how to best approach this conversion efficiently on NXP platforms

Any documentation or starting point would be greatly appreciated.

[Manuel_Salas]

Hello @Olivia 

Yes, you can take a look to the driver imx8-isi-m2m.c.

Best regards,

Salas.

[Olivia]

Hi Salas,

Thank you for your previous assistance. I'm continuing development of my custom V4L2 M2M driver (grey8_to_nv12.ko) for the i.MX 8M Plus EVK.

I am consistently encountering a critical kernel panic (NULL pointer dereference) in cgroup_file_write (or cgroup_procs_start) shortly after my module loads. This occurs spontaneously, typically within seconds of insmod, after the system's automatic V4L2 device discovery briefly opens and closes /dev/video2.

Problem Timeline & Observations:

1. Module Loading & Probing: My module loads, the platform device registers, and the /dev/video2 node is successfully created. (Logs confirm: my_m2m_probe: Probing M2M device..., Video device video2 created at /dev/video2, driver probed successfully!)
2. Automatic V4L2 Device Scan: A userspace process (likely udev or v4l2-id) automatically opens and then closes /dev/video2. My driver's my_m2m_open and my_m2m_release callbacks execute successfully and report no errors. (Logs confirm: my_m2m_open: Opening device, Device opened, context created, my_m2m_release: Releasing device, Device released).
3. Kernel Panic: Immediately following this successful my_m2m_release, the kernel panics. The stack trace consistently points to a NULL pointer dereference within the kernel's cgroup subsystem (cgroup_file_write or cgroup_procs_start). This suggests a deeper system instability, rather than a direct error in my V4L2 driver's core logic.

Detailed Logs :

[ 37.387466] grey8_to_nv12: loading out-of-tree module taints kernel.
[ 37.394944] [imx-gray8-nv12] Module init: Registering platform driver.
[ 37.402287] imx-gray8-nv12-m2m imx-gray8-nv12-m2m: [imx-gray8-nv12] my_m2m_probe: Probing M2M device...
[ 37.411952] imx-gray8-nv12-m2m imx-gray8-nv12-m2m: [imx-gray8-nv12] Video device video2 created at /dev/video2
[ 37.422015] imx-gray8-nv12-m2m imx-gray8-nv12-m2m: [imx-gray8-nv12] i.MX8M Plus GRAY8 to NV12 M2M driver probed successfully!
root@imx8mpevk:~# [ 37.467196] imx-gray8-nv12-m2m imx-gray8-nv12-m2m: [imx-gray8-nv12] my_m2m_open: Opening device
[ 37.475969] imx-gray8-nv12-m2m imx-gray8-nv12-m2m: [imx-gray8-nv12] my_m2m_open: Device opened, context created
[ 37.486270] imx-gray8-nv12-m2m imx-gray8-nv12-m2m: [imx-gray8-nv12] my_m2m_release: Releasing device
[ 37.495480] imx-gray8-nv12-m2m imx-gray8-nv12-m2m: [imx-gray8-nv12] my_m2m_release: Device released
[ 39.709494] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000578
[ 39.710213] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000578
[ 39.718385] Mem abort info:
[ 39.727124] Mem abort info:
[ 39.727126] ESR = 0x0000000096000004
[ 39.727130] EC = 0x25: DABT (current EL), IL = 32 bits
[ 39.727134] SET = 0, FnV = 0
[ 39.727137] EA = 0, S1PTW = 0
[ 39.727140] FSC = 0x04: level 0 translation fault
[ 39.727143] Data abort info:
[ 39.727145] ISV = 0, ISS = 0x00000004
[ 39.729968] ESR = 0x0000000096000004
[ 39.732753] CM = 0, WnR = 0
[ 39.732759] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000115015000
[ 39.736525] EC = 0x25: DABT (current EL), IL = 32 bits
[ 39.741831] [0000000000000578] pgd=0000000000000000, p4d=0000000000000000
[ 39.744917] SET = 0, FnV = 0
[ 39.748033] 
[ 39.748037] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
[ 39.748042] Modules linked in:
[ 39.752937] EA = 0, S1PTW = 0
[ 39.755789] grey8_to_nv12(O) flexcan overlay fsl_jr_uio caam_jr
[ 39.759673] FSC = 0x04: level 0 translation fault
[ 39.763372] caamkeyblob_desc caamhash_desc caamalg_desc crypto_engine authenc libdes dw_hdmi_cec snd_soc_wm8962 snd_soc_fsl_sai snd_soc_fsl_xcvr snd_soc_fsl_aud2htx imx8_media_dev(C)
[ 39.766375] Data abort info:
[ 39.772789] snd_soc_fsl_micfil snd_soc_fsl_easrc snd_soc_fsl_asrc snd_soc_fsl_utils can_dev caam secvio error
[ 39.778116] ISV = 0, ISS = 0x00000004
[ 39.784887] crct10dif_ce polyval_ce polyval_generic snd_soc_fsl_asoc_card snd_soc_imx_audmux snd_soc_imx_card snd_soc_imx_hdmi
[ 39.787945] CM = 0, WnR = 0
[ 39.789433] imx_dsp_rproc fuse
[ 39.789439] CPU: 3 PID: 900 Comm: (t_isp.sh) Tainted: G       C O       6.1.36+g04b05c5527e9 #1
[ 39.795713] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000111e59000
[ 39.798752] Hardware name: NXP i.MX8MPlus EVK board (DT)
[ 39.798755] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 39.798760] pc : cgroup_file_write+0x50/0x1b0
[ 39.801922] [0000000000000578] pgd=0000000000000000
[ 39.807902] lr : kernfs_fop_write_iter+0x120/0x1f0
[ 39.807910] sp : ffff80000e6e3ca0
[ 39.807912] x29: ffff80000e6e3ca0 x28: ffff0000ddab8000 x27: 0000000000000000
[ 39.807919] x26: 0000000000000000 x25: 0000000000000000
[ 39.812826] , p4d=0000000000000000
[ 39.829124] x24: ffff0000d4b17d20
[ 39.829127] x23: ffff80000e6e3d88 x22: ffff0000d5434800 x21: ffff800009db77f8
[ 39.829134] x20: ffff0000d5ed9580 x19: 0000000000000004
[ 39.832012] 
[ 39.842004] x18: 0000000000000000
[ 39.842007] x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffffc42c0b48
[ 39.842013] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 39.842019] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[ 39.958914] x8 : 0000000000000000 x7 : 0000000000000200 x6 : ffff0000dd8c5184
[ 39.958921] x5 : 0000000000000000 x4 : ffff0000d4b17d00 x3 : 0000000000000000
[ 39.958928] x2 : ffff0000d4b17c00 x1 : ffff0000dd8c5280 x0 : 0000000000000000
[ 39.958935] Call trace:
[ 39.958937] cgroup_file_write+0x50/0x1b0
[ 39.958944] kernfs_fop_write_iter+0x120/0x1f0
[ 39.958950] vfs_write+0x1b8/0x2e0
[ 39.958956] ksys_write+0x70/0x104
[ 39.958960] __arm64_sys_write+0x1c/0x2c
[ 39.958964] invoke_syscall+0x48/0x114
[ 39.958970] el0_svc_common.constprop.0+0xcc/0xec
[ 39.958976] do_el0_svc+0x2c/0xd0
[ 39.958981] el0_svc+0x2c/0x84
[ 39.958989] el0t_64_sync_handler+0xf4/0x120
[ 40.236597] el0t_64_sync+0x18c/0x190
[ 40.236609] Code: f9400401 f9403015 f9403036 f940e6c0 (b9457800) 
[ 40.236613] ---[ end trace 0000000000000000 ]---
[ 40.236786] Internal error: Oops: 0000000096000004 [#2] PREEMPT SMP
[ 40.236792] Modules linked in: grey8_to_nv12(O) flexcan overlay fsl_jr_uio caam_jr caamkeyblob_desc caamhash_desc caamalg_desc crypto_engine authenc libdes dw_hdmi_cec snd_soc_wm8962 snd_soc_fsl_sai snd_soc_fsl_xcvr snd_soc_fsl_aud2htx imx8_media_dev(C) snd_soc_fsl_micfil snd_soc_fsl_easrc snd_soc_fsl_asrc snd_soc_fsl_utils can_dev caam secvio error crct10dif_ce polyval_ce polyval_generic snd_soc_fsl_asoc_card snd_soc_imx_audmux snd_soc_imx_card snd_soc_imx_hdmi imx_dsp_rproc fuse
[ 40.236792] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 40.236792] SMP: stopping secondary CPUs
[ 40.240449] Kernel Offset: disabled
[ 40.240450] CPU features: 0x00000,00800084,0000421b
[ 40.240453] Memory Limit: none
[ 40.274121] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

System Environment:

• Board: NXP i.MX 8M Plus EVK board
• Kernel Version: Linux imx8mpevk 6.1.36
• Build System: NXP's standard Yocto build for i.MX8M Plus
• Process involved in panic: The panic trace points to cgroup_file_write being called by PID: 900 Comm: (t_isp.sh) or PID: 1 Comm: systemd.

---

Questions for NXP Support:

1. Given that cgroup_file_write / cgroup_procs_start is part of the core kernel's cgroup subsystem, has NXP observed similar NULL pointer dereferences or kernel instabilities in this area on the i.MX 8M Plus EVK with kernel 6.1.x?
2. Could this be a known bug or a particular configuration issue related to cgroups in the NXP-provided kernel (or Yocto layer)?
3. Are there any recommended workarounds or kernel patches that address cgroup stability issues on this platform?
4. Are there specific NXP-recommended debugging tools or kernel configurations (e.g., enabling certain CONFIG_DEBUG_... options) that could help us further diagnose this low-level kernel crash?
5. The dmesg output shows imx8_media_dev(C) linked in. Is it possible that this proprietary module interacts with the system or cgroups in a way that might expose this issue, or is it likely a completely separate problem?

Thank you for your continued assistance.