imx8mq: Facing kernel hang(dcss driver) issue with Industrial imx8mq.

[abhishekojhasfl]

Hi all,

I am facing an issue where kernel boots successfully with imx8mq's commercial version but kernel stuck with industrial version.

After some debugging found that, if we disabled dcss node in device tree then kernel boots successfully with industrial version.

Looks like dcss driver hangs every time when tries to access 0x32e20000. Logs with Industrial version.

[    2.128217] imx-dcss 32e00000.display-controller: dcss_dev_create:262 submodules initialization
[    2.151964] imx-dcss 32e00000.display-controller: dcss_submodules_init:81 dcss_blkctl_init
[    2.160257] imx-dcss 32e00000.display-controller: dcss_blkctl_init:phy_addr=0x32e2f000 ioremap to addr=0x822f6000
[    2.170542] imx-dcss 32e00000.display-controller: dcss_submodules_init:85 dcss_ctxld_init
[    2.179547] imx-dcss 32e00000.display-controller: dcss_ctxld_init:phy_addr=0x32e23000 ioremap to addr=0x823e1000
[    2.189891] imx-dcss 32e00000.display-controller: dcss_submodules_init:89 dcss_dtg_init
[    2.197940] imx-dcss 32e00000.display-controller: dcss_dtg_init:phy_addr=0x32e20000 ioremap to addr=0x823e3000
[    2.208025] imx-dcss 32e00000.display-controller: dcss_dtg_irq_config: dcss_update

logs with commercial version:

[    2.549979] imx-dcss 32e00000.display-controller: dcss_submodules_init:81 dcss_blkctl_init
[    2.564396] imx-dcss 32e00000.display-controller: dcss_blkctl_init:phy_addr=0x32e2f000 ioremap to addr=0x824e5000
[    2.580739] imx-dcss 32e00000.display-controller: dcss_submodules_init:85 dcss_ctxld_init
[    2.638699] imx-dcss 32e00000.display-controller: dcss_ctxld_init:phy_addr=0x32e23000 ioremap to addr=0x824fd000
[    2.648945] imx-dcss 32e00000.display-controller: dcss_submodules_init:89 dcss_dtg_init
[    2.656961] imx-dcss 32e00000.display-controller: dcss_dtg_init:phy_addr=0x32e20000 ioremap to addr=0x82505000
[    2.666996] imx-dcss 32e00000.display-controller: dcss_dtg_irq_config: dcss_update
[    2.674615] imx-dcss 32e00000.display-controller: dcss_submodules_init:93 dcss_ss_init
[    2.682552] imx-dcss 32e00000.display-controller: dcss_submodules_init:97 dcss_dtrc_init
[    2.690715] imx-dcss 32e00000.display-controller: dcss_submodules_init:101 dcss_dpr_init
[    2.700975] [drm] Initialized imx-dcss 1.0.0 20190917 for 32e00000.display-controller on minor 1
[    2.762816] hub 1-1:1.0: USB hub found
[    2.763902] hub 1-1:1.0: 4 ports detected
[    2.803613] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 4G
[    2.942151] ili9881c-dsi-5 30a00000.dsi.0: Skipping enable of already enabled panel
[    2.990711] Console: switching to colour frame buffer device 90x80
[    3.036026] imx-dcss 32e00000.display-controller: [drm] fb0: imx-dcssdrmfb frame buffer device

Are there changes that needs to be done for Industrial version processor.

I am using latest NXP SDK

• Linux version 6.6.3-lts-next-g594ecad341c5-dirty
• nanbield

Attaching full dmesg logs for reference.

Thanks a lot!

[joanxie]

pls share the consumer and industrial part number with me, the only difference between your two boards is the chip? are they HW total the same? and did you connect any display via dcss?

[abhishekojhasfl]

Hi

Here are the part number details.

1. Commercial: MIMX8MQ6DVAJZAB

2. Industrial: MIMX8MQ6CVAHZAB

• DDR chip is different, other than that there are no changes on the HW side. On working(commercial) setup, we are using "MT53E768M32D2ZW-046 WT:" DDR and on Industrial setup we are using "MT53E1G32D2" DDR.
• We are using mipi-dsi to connect LCD panel.

I used the NXP DDR Tool to generate the new set of timings for "MT53E1G32D2". Attaching all the configurations and test results. Please review the configurations and let me know if any changes required.

Thanks

[abhishekojhasfl]

Hi

Kernel hang issue on Industrial imx8mq still persists. Do you have any inputs on this?

Are there any code changes that needs to be added for Industrial imx8mq processor?

Thanks

[abhishekojhasfl]

Adding debugging logs:

• Logs from Industrial imx8mq processor setup:

aarch64-poky-linux-musl-gdb ./vmlinux
GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pokysdk-linux --target=aarch64-poky-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./vmlinux...

warning: could not convert 'main' from the host encoding (UTF-8) to UTF-32.
This normally should not happen, please file a bug report.
(gdb) b dcss-dtg.c:134
Breakpoint 1 at 0xffff80008095ecac: file /usr/src/kernel/drivers/gpu/drm/imx/dcss/dcs
s-dtg.c, line 134.
(gdb) target remote localhost:4441
Remote debugging using localhost:4441
warning: multi-threaded target stopped without sending a thread-id, using first non-e
xited thread
[Switching to Thread 4294967294]
arch_kgdb_breakpoint () at /usr/src/kernel/arch/arm64/include/asm/kgdb.h:21
21      /usr/src/kernel/arch/arm64/include/asm/kgdb.h: No such file or directory.
(gdb) c
Continuing.
[New Thread 59]
[New Thread 60]
[New Thread 61]
[New Thread 62]
[New Thread 63]
[New Thread 64]
[New Thread 65]
[New Thread 66]
[New Thread 67]
[New Thread 68]
[New Thread 69]
[New Thread 70]
[New Thread 71]
[New Thread 72]
[New Thread 73]
[New Thread 74]
[New Thread 75]
[New Thread 76]
[New Thread 77]
[Switching to Thread 47]

Thread 51 hit Breakpoint 1, dcss_dtg_irq_config (pdev=<optimized out>, 
    dtg=0xffff0000c0409fc0)
    at /usr/src/kernel/drivers/gpu/drm/imx/dcss/dcss-dtg.c:134
134     /usr/src/kernel/drivers/gpu/drm/imx/dcss/dcss-dtg.c: No such file or director
y.
(gdb) info frame
Stack level 0, frame at 0xffff80008275baf0:
 pc = 0xffff80008095ecac in dcss_dtg_irq_config
    (/usr/src/kernel/drivers/gpu/drm/imx/dcss/dcss-dtg.c:134); 
    saved pc = 0xffff80008095dcc0
 inlined into frame 1
 source language c.
 Arglist at unknown address.
 Locals at unknown address, Previous frame's sp in sp
(gdb) p dtg
$1 = (struct dcss_dtg *) 0xffff0000c0409fc0
(gdb) p *dtg
$2 = {dev = 0xffff0000c02c5810, ctxld = 0xffff0000c0457280, 
  base_reg = 0xffff800082363000, base_ofs = 853671936, ctx_id = 0, in_use = false, 
  hdmi_output = false, dis_ulc_x = 0, dis_ulc_y = 0, control_status = 4278190216, 
  alpha = 255, alpha_cfg = 0, ctxld_kick_irq = 203, ctxld_kick_irq_en = false}
(gdb) p dtg->base_reg
$3 = (void *) 0xffff800082363000
(gdb) p *(0xffff800082363000)
Ignoring packet error, continuing...
Invalid hex digit 116
(gdb) info registers 
x0             0xcb                203
x1             0xfffffdfb          4294966779
x2             0xcb                203
x3             0x4                 4
x4             0x0                 0
x5             0xffff0000ff7ff8a0  -281470690133856
x6             0xcb                203
x7             0xffff0000c0fb1700  -281471739029760
x8             0x1fffe000181f62e1  2305807825246315233
x9             0xffff800080d85b54  -140735326692524
x10            0xcb                203
x11            0xffff0000c0304600  -281471752321536
x12            0xffff0000c048dc00  -281471750710272
x13            0xffff0000c043ace0  -281471751050016
x14            0xffff0000c043ace4  -281471751050012
x15            0xffff80008275b450  -140735299603376
x16            0xffff0000c0302800  -281471752329216
x17            0xffff0000c0302c00  -281471752328192
x18            0xffffffffffffffff  -1
x19            0xffff0000c0409fc0  -281471751249984
x20            0xcb                203
x21            0x32e20000          853671936
x22            0x32e00000          853540864
x23            0xffff800082110f10  -140735306199280
x24            0x17                23
x25            0xffff0000c06b0100  -281471748472576
x26            0x0                 0
x27            0x0                 0
x28            0x0                 0
x29            0xffff80008275bac0  -140735299601728
x30            0xffff80008095eca0  -140735331046240
sp             0xffff80008275bac0  0xffff80008275bac0
pc             0xffff80008095ecac  0xffff80008095ecac <dcss_dtg_init+196>
cpsr           0x40000005          [ SP EL=1 BTYPE=0 Z ]
fpsr           0x0                 [ ]
fpcr           0x0                 [ Len=0 Stride=0 RMode=0 ]
(gdb)

• Logs from Commercial imx8mq processor setup:

aarch64-poky-linux-musl-gdb ./vmlinux
GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pokysdk-linux --target=aarch64-poky-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./vmlinux...

warning: could not convert 'main' from the host encoding (UTF-8) to UTF-32.
This normally should not happen, please file a bug report.
(gdb) 
(gdb) target remote localhost:4441
Remote debugging using localhost:4441
warning: multi-threaded target stopped without sending a thread-id, using first non-exited thread
[Switching to Thread 4294967294]
arch_kgdb_breakpoint () at /usr/src/kernel/arch/arm64/include/asm/kgdb.h:21
21      /usr/src/kernel/arch/arm64/include/asm/kgdb.h: No such file or directory.
(gdb) b dcss-dtg.c:134
Breakpoint 1 at 0xffff80008095ecac: file /usr/src/kernel/drivers/gpu/drm/imx/dcss/dcss-dtg.c, line 134
.
(gdb) c
Continuing.
[New Thread 61]
[New Thread 62]
[New Thread 63]
[New Thread 64]
[New Thread 65]
[New Thread 66]
[New Thread 67]
[New Thread 68]
[New Thread 69]
[New Thread 70]
[New Thread 71]
[New Thread 72]
[New Thread 73]
[New Thread 74]
[Switching to Thread 47]

Thread 51 hit Breakpoint 1, dcss_dtg_irq_config (pdev=<optimized out>, dtg=0xffff0000c10675c0)
    at /usr/src/kernel/drivers/gpu/drm/imx/dcss/dcss-dtg.c:134
134     /usr/src/kernel/drivers/gpu/drm/imx/dcss/dcss-dtg.c: No such file or directory.
(gdb) p dtg
$1 = (struct dcss_dtg *) 0xffff0000c10675c0
(gdb) p *dtg
$2 = {dev = 0xffff0000c02c5810, ctxld = 0xffff0000c093da00, base_reg = 0xffff800082363000, base_ofs = 
853671936, ctx_id = 0, 
  in_use = false, hdmi_output = false, dis_ulc_x = 0, dis_ulc_y = 0, control_status = 4278190216, alph
a = 255, alpha_cfg = 0, 
  ctxld_kick_irq = 203, ctxld_kick_irq_en = false}
(gdb) x/1x 0xffff800082363000
0xffff800082363000:     0x00000400
(gdb) x/1x 0xffff800082363068
0xffff800082363068:     0x00000000
(gdb) info frame
Stack level 0, frame at 0xffff80008275baf0:
 pc = 0xffff80008095ecac in dcss_dtg_irq_config (/usr/src/kernel/drivers/gpu/drm/imx/dcss/dcss-dtg.c:1
34); 
    saved pc = 0xffff80008095dcc0
 inlined into frame 1
 source language c.
 Arglist at unknown address.
 Locals at unknown address, Previous frame's sp in sp
(gdb) p *(0xffff800082363068)
$3 = 0
(gdb) p *(0xffff800082363000)
$4 = 1024

 Thanks

[joanxie]

thanks for your update, after you disable dcss in the Industrial part, could you access 0x32e20000 address?

[abhishekojhasfl]

As per my observation, 0x32e20000 address is not accessible if we disable the dcss in device tree, irrespective of the type of processor.

Thanks

[joanxie]

do your two boards are total the same? how many industrial board do you have？only one? try  to replace the board with commercial part by the industrial part, still has the same issue?

[abhishekojhasfl]

Yes Both the boards are same except processor and DDR.

Its bit difficult to change the processor right now. Do you see any issue on the DDR configurations side? I have already attached DDR configuration in previous messages.

Thanks

[joanxie]

I checked your log file, it seems your ddr pass the stress test successfully, the lpddr4_timing.c is from the test, right? and you replace the original file by the lpddr4_timing.c from the test, right? 

[abhishekojhasfl]

Hi

Thanks for reviewing the DDR logs.

Yes, the lpddr4_timing.c file is generated after the calibration and then stress test. Also, the same lpddr4_timing.c is used to build the u-boot image.

Thanks

[joanxie]

let's double confirm your ddr again, any update, mail to you direclty