imx7d crashes

[asmaa]

Hello Everyone,

My company is working on imx7d boards, kernel version we are using is 4.19.35. 

I have written a char driver, which is being called from userspace to turn ON/OFF a GPIO. I know I can do it in userspace as well, however, I have to turn GPIO ON/OFF very frequently from both the userspace and kernel space. So Instead of resource sharing from userspace and kernel space, I thought of making a driver in kernel space which will be called from userspace and handle LED toggling for both the worlds.

Now after 1 o- 2 hours of running without any issue, the kernel crashes. Here is the output I see, I will attach the full log.

WARNING: CPU: 0 PID: 1060562623 at kernel/sched/core.c:3187 search_module_extables+0x10/0x88
DEBUG_LOCKS_WARN_ON((preempt_count() < 0))
Modules linked in:
CPU: 0 PID: 1060562623 Comm: Hã  ã Tainted: G W 4.19.35-00051-g3d6c7ab1fd59-dirty #1
Hardware name: Freescale i.MX7 Dual (Device Tree)
[<8010e0ec>] (unwind_backtrace) from [<8010b868>] (show_stack+0x10/0x14)
[<8010b868>] (show_stack) from [<80128078>] (__warn+0xe0/0xf8)
[<80128078>] (__warn) from [<801280d8>] (warn_slowpath_fmt+0x48/0x6c)
[<801280d8>] (warn_slowpath_fmt) from [<8018caf4>] (search_module_extables+0x10/0x88)
[<8018caf4>] (search_module_extables) from [<80111c74>] (fixup_exception+0x10/0x28)
[<80111c74>] (fixup_exception) from [<80111eb8>] (do_page_fault+0x7c/0x36c)
[<80111eb8>] (do_page_fault) from [<80112408>] (do_PrefetchAbort+0x48/0xcc)
---[ end trace 802dce8a39c2e1b0 ]---
Unable to handle kernel paging request at virtual address 01010000
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel paging request at virtual address 2000e2a9
Unable to handle kernel NULL pointer dereference at virtual address 0000018c
pgd = a5f496ab
[0000018c] *pgd=a6bb1835, *pte=00000000, *ppte=00000000
Internal error: Oops: 5 [#1] PREEMPT ARM
Modules linked in:
CPU: 0 PID: 224 Comm: klogd Tainted: G W 4.19.35-00051-g3d6c7ab1fd59-dirty #1
Hardware name: Freescale i.MX7 Dual (Device Tree)
PC is at do_page_fault+0x30/0x36c
LR is at do_DataAbort+0x4c/0xe8
pc : [<80111e6c>] lr : [<80112324>] psr: 000f0193
sp : a6b6c098 ip : ffffe000 fp : 00000000
r10: 00000000 r9 : a6b6c000 r8 : 0000018c
r7 : 00000005 r6 : 0000018c r5 : 00000000 r4 : a6b6c198
r3 : a6b6c000 r2 : 000f0193 r1 : 00000005 r0 : 0000018c
Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
Control: 10c53c7d Table: a6088059 DAC: 00000051
Process klogd (pid: 224, stack limit = 0x594c3bfb)
Stack: (0xa6b6c098 to 0xa6b6c000)
[<80111e6c>] (do_page_fault) from [<80112324>] (do_DataAbort+0x4c/0xe8)
[<80112324>] (do_DataAbort) from [<80101978>] (__dabt_svc+0x58/0x80)
Exception stack(0xa6b6c198 to 0xa6b6c1e0)
c180: 0000018c 00000005
c1a0: 000f0193 a6b6c000 a6b6c2e8 00000000 0000018c 00000005 0000018c a6b6c000
c1c0: 00000000 00000000 ffffe000 a6b6c1e8 80112324 80111e6c 000f0193 ffffffff
[<80101978>] (__dabt_svc) from [<80111e6c>] (do_page_fault+0x30/0x36c)
[<80111e6c>] (do_page_fault) from [<80112324>] (do_DataAbort+0x4c/0xe8)
[<80112324>] (do_DataAbort) from [<80101978>] (__dabt_svc+0x58/0x80)
Exception stack(0xa6b6c2e8 to 0xa6b6c330)
c2e0: 0000018c 00000005 000f0193 a6b6c000 a6b6c438 00000000
c300: 0000018c 00000005 0000018c a6b6c000 00000000 00000000 ffffe000 a6b6c338
c320: 80112324 80111e6c 000f0193 ffffffff
[<80101978>] (__dabt_svc) from [<80111e6c>] (do_page_fault+0x30/0x36c)
[<80111e6c>] (do_page_fault) from [<80112324>] (do_DataAbort+0x4c/0xe8)
[<80112324>] (do_DataAbort) from [<80101978>] (__dabt_svc+0x58/0x80)
Exception stack(0xa6b6c438 to 0xa6b6c480)

.....

bf80: 81008f9c a6b6c048 a6b6c000 00000000 00000000 80112324 a6b6a000 00000067
bfa0: 00000000 80101000 000a8845 00000000 00000002 000a87f8 000003ff 000003fe
bfc0: 000a8845 00000000 000a87f8 00000067 00000000 00000000 00000000 00000000
bfe0: 000a8300 7e8f3cd4 0004864c 76e8b55c 600c0010 00000002 00000000 00000000
[<80152a4c>] (autoremove_wake_function) from [<80151f78>] (__wake_up_common+0x78/0x134)
[<80151f78>] (__wake_up_common) from [<80152660>] (__wake_up_common_lock+0x7c/0x158)
[<80152660>] (__wake_up_common_lock) from [<80152750>] (__wake_up+0x14/0x1c)
[<80152750>] (__wake_up) from [<80197e30>] (irq_work_run_list+0x6c/0xa0)
[<80197e30>] (irq_work_run_list) from [<8017264c>] (update_process_times+0x4c/0x60)
[<8017264c>] (update_process_times) from [<80181dac>] (tick_sched_timer+0x54/0xb0)
[<80181dac>] (tick_sched_timer) from [<8017315c>] (__hrtimer_run_queues.constprop.4+0x148/0x1f0)
[<8017315c>] (__hrtimer_run_queues.constprop.4) from [<80173878>] (hrtimer_interrupt+0xcc/0x2f0)
[<80173878>] (hrtimer_interrupt) from [<807ee074>] (mxc_timer_interrupt+0x34/0x3c)
[<807ee074>] (mxc_timer_interrupt) from [<8015d054>] (__handle_irq_event_percpu+0x50/0x128)
[<8015d054>] (__handle_irq_event_percpu) from [<8015d158>] (handle_irq_event_percpu+0x2c/0x7c)
[<8015d158>] (handle_irq_event_percpu) from [<8015d1f4>] (handle_irq_event+0x4c/0x84)
[<8015d1f4>] (handle_irq_event) from [<8016075c>] (handle_fasteoi_irq+0xb8/0x194)
[<8016075c>] (handle_fasteoi_irq) from [<8015c2d4>] (generic_handle_irq+0x24/0x34)
[<8015c2d4>] (generic_handle_irq) from [<8015c9d8>] (__handle_domain_irq+0x70/0xdc)
[<8015c9d8>] (__handle_domain_irq) from [<80426550>] (gic_handle_irq+0x4c/0x80)
[<80426550>] (gic_handle_irq) from [<80101a0c>] (__irq_svc+0x6c/0xa8)
Exception stack(0xa6b6bea0 to 0xa6b6bee8)
bea0: 00000001 80d76fc8 00000003 00000003 a6390980 0000000b 600f0193 a6b6a000
bec0: 7f000000 ffffe000 a6390980 0000000b 00000007 a6b6bef0 801291e0 801291e4
bee0: 800f0113 ffffffff
[<80101a0c>] (__irq_svc) from [<801291e4>] (do_exit+0xd4/0xb18)
[<801291e4>] (do_exit) from [<8010b9dc>] (die+0x170/0x274)
[<8010b9dc>] (die) from [<801124f0>] (__do_kernel_fault.part.0+0x64/0x74)
[<801124f0>] (__do_kernel_fault.part.0) from [<801121a8>] (do_bad_area+0x0/0x84)
[<801121a8>] (do_bad_area) from [<00000051>] (0x51)
Code: e5942010 e284300c e594100c e5812004 (e5821000)
---[ end trace 802dce8a39c2e1b2 ]---
Kernel panic - not syncing: Fatal exception in interrupt
---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

I have stuck with this problem for a few days, Any help would be really appreciated.

@igorpadykov 

Thanks,

Asma

[igorpadykov]

Hi Asma

for that error one can run ddr test and put new calibration coefficients found from test

to uboot dcd header

https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX-6-7-DDR-Stress-Test-Tool/ta-p/11082...

https://source.codeaurora.org/external/imx/uboot-imx/tree/board/freescale/mx7dsabresd/imximage.cfg?h...

Best regards
igor

[asmaa]

Hi @igorpadykov ,

Every other peripheral and hardware are working correctly for ou board. why do you think DDR is a problem?

Thanks,

Asma

[igorpadykov]

Hi Asma

without running ddr test & rebuild image some memory regions may be faulty,

so for new board recommended first to run ddr test and rebuild image with new

ddr coefficients found from test.

Best regards
igor