iMX6Q SOM freeze or Kernel crashes at High CPU Frequency

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

iMX6Q SOM freeze or Kernel crashes at High CPU Frequency

3,883 Views
balasubramaniyam
Contributor II

Hi,

I'm using iMX6Q SOM with Linux 4.9.11 kernel and Yocto Morty package.

When I stress the CPU at High frequency (996MHz), I got a kernel crash or SOM freeze issue.

Error log:

Unable to handle kernel paging request at virtual address 80e45d50
pgd = 80004000
[80e45d50] *pgd=10e1141e(bad)
Unable to handle kernel paging request at virtual address 80e45d50
pgd = d97f0000
[80e45d50] *pgd=10e1141e(bad)
Internal error: Oops: 8000000d [#1] PREEMPT SMP ARM
Modules linked in: wl12xx wlcore mxc_v4l2_capture ipu_bg_overlay_sdc ov5640_camera_mipi(O) ov5640_camera_parallel(O) ipu_still ar1820_camera_mipi(O) ipu_prp_enc ov5640_camera(O) mt9p031(O) ipu_csi_enc ov10633_camera_parallel(O) ipu_fg_overlay_sdc v4l2_int_device wlcore_sdio mxc_dcic evbug
CPU: 0 PID: 859 Comm: i1_killer Tainted: G O 3.14.28 #1
task: d8ff0900 ti: d8aa4000 task.ti: d8aa4000
PC is at thread_notify_head+0x0/0x8
LR is at rootfs_mount+0x24/0x68
pc : [<80e45d50>] lr : [<80008948>] psr: 000f0193
sp : d8aa5ee0 ip : 00000000 fp : d8aa5f94
r10: d8f46c40 r9 : 00000002 r8 : d823e000
r7 : ffffffff r6 : 8004c0bc r5 : ffffffff r4 : d823e000
r3 : 80009868 r2 : 00000001 r1 : 80e45c6c r0 : ffffffed
Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
Control: 10c53c7d Table: 697f004a DAC: 00000015
Process i1_killer (pid: 859, stack limit = 0xd8aa4238)
Stack: (0xd8aa5ee0 to 0xd8aa6000)
5ee0: 00000000 d817f980 00000000 8004c340 00000000 d8aa4008 d823e01c d8ff0900
5f00: ee71ebc0 80012598 00000001 80de4084 00000000 00000000 80e47bc0 00000000
5f20: 80e45a38 80ddfbc0 efc0676d 00005516 efc0676d 00000000 80de4080 80dde458
5f40: 80e47bc0 0000000a 80727398 008e4c85 80de40c0 00404140 ee71ebc0 80ddee54
5f60: 0000001d 600f0193 f4a00100 800717f8 80ddee54 d8aa4018 00000000 d8aa5fb0
5f80: d8aa4000 00000000 d8aa4000 68c00018 00000320 800112bc 6f847eec 76a20ffa
5fa0: 400f0030 ffffffff 6942d7f0 8000e4a0 6f847eec 6f841ae4 00000103 00000265
5fc0: 000002d0 6f84726c 000001a8 6942d7f0 00000000 68c4a898 68c00018 00000320
5fe0: 000002d0 719ada60 6942d900 76a20ffa 400f0030 ffffffff dcf12125 cd6bfb12
[<80008948>] (rootfs_mount) from [<d8aa4008>] (0xd8aa4008)
Code: 00000000 00000000 00000000 8047abf0 (00010001)
---[ end trace 630f7791a35e068a ]---
note: i1_killer[859] exited with preempt_count 2
Internal error: Oops: 8000000d [#2] PREEMPT SMP ARM
Modules linked in: wl12xx wlcore mxc_v4l2_capture ipu_bg_overlay_sdc ov5640_camera_mipi(O) ov5640_camera_parallel(O) ipu_still ar1820_camera_mipi(O) ipu_prp_enc ov5640_camera(O) mt9p031(O) ipu_csi_enc ov10633_camera_parallel(O) ipu_fg_overlay_sdc v4l2_int_device wlcore_sdio mxc_dcic evbug
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D O 3.14.28 #1
task: d8082d00 ti: d80aa000 task.ti: d80aa000
PC is at thread_notify_head+0x0/0x8
LR is at rootfs_mount+0x24/0x68
pc : [<80e45d50>] lr : [<80008948>] psr: 000e0093
sp : d80abf30 ip : 00000000 fp : d80abfe4
r10: d879e000 r9 : 00000002 r8 : d8204000
r7 : ffffffff r6 : 8004c0bc r5 : ffffffff r4 : d8204000
r3 : 80009868 r2 : 00000001 r1 : 80e45c6c r0 : ffffffed
Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c53c7d Table: 687a004a DAC: 00000015
Process swapper/1 (pid: 0, stack limit = 0xd80aa238)
Stack: (0xd80abf30 to 0xd80ac000)
bf20: 112078d3 d839f500 00000000 8004c340
bf40: 00000000 d80aa018 d820401c d8082d00 ee726bc0 80012598 efc05200 00005516
bf60: 80dde2d0 00000000 efc05200 00005516 80e45a38 80ddfbc0 80dde2d0 ee725a00
bf80: 00005516 00000000 00005516 ef29704a 80e8bb08 efc05200 00005516 8004b26c
bfa0: 00000000 00000002 00000001 80e8bb08 d80aa000 8007c99c 00000000 00000002
bfc0: ef29704a d80aa000 80dea574 8072738c d80aa030 80e45a3d 80e45a3d d80aa000
bfe0: d80abfec 80721e68 00000000 80067ac0 80e45f5c 10008624 ada1db5f 4ccddb48
[<80008948>] (rootfs_mount) from [<d80aa018>] (0xd80aa018)
Code: 00000000 00000000 00000000 8047abf0 (00010001)
---[ end trace 630f7791a35e068b ]---
Kernel panic - not syncing: Attempted to kill the idle task!
CPU2: stopping
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D O 3.14.28 #1
[<800155f0>] (unwind_backtrace) from [<800116d4>] (show_stack+0x10/0x14)
[<800116d4>] (show_stack) from [<8071ea2c>] (dump_stack+0x7c/0xbc)
[<8071ea2c>] (dump_stack) from [<80014590>] (handle_IPI+0x144/0x158)
[<80014590>] (handle_IPI) from [<80008598>] (gic_handle_irq+0x58/0x5c)
[<80008598>] (gic_handle_irq) from [<800121c0>] (__irq_svc+0x40/0x70)
Exception stack(0xd80adf50 to 0xd80adf98)
df40: d80adf98 3b9aca00 0385309f 00005517
df60: 80df0508 80df0508 ee72e0d0 00000000 efc4e912 00005516 d80ac000 00000000
df80: 00000017 d80adf98 00000009 804db154 00080013 ffffffff
[<800121c0>] (__irq_svc) from [<804db154>] (cpuidle_enter_state+0x54/0xe4)
[<804db154>] (cpuidle_enter_state) from [<804db298>] (cpuidle_idle_call+0xb4/0x14c)
[<804db298>] (cpuidle_idle_call) from [<8000f07c>] (arch_cpu_idle+0x8/0x44)
[<8000f07c>] (arch_cpu_idle) from [<80067ae8>] (cpu_startup_entry+0x100/0x14c)
[<80067ae8>] (cpu_startup_entry) from [<10008624>] (0x10008624)
SMP: failed to stop secondary CPUs
þ

This issue is not coming while running the CPU stress test under 792MHz and 396MHz.

And I also tried to boot the QuadSOM with maximum CPU cores as one and two by using maxcpus environment variables in U-Boot.

While booting the SOM with one or two core CPU, I can able to put the CPU Stress test at all frequencies(396MHz, 792MHz, and 996MHz). And I didn't face these SOM freeze or kernel crash issue with maxcpus=2 or maxcpus=1.

NOTE:

  • Issue only occurs in High CPU frequency(996MHz) only and that too while accessing CPU 3 and CPU 4 core. And I didn't face these issue while running CPU stress test at CPU High frequency(996MHz) with maxcpus=2.
  • And these issue comes with some SOM only.

Hoping for better guidance.

 

Thanks and Regards,

Bala

Labels (4)
0 Kudos
18 Replies

3,230 Views
balasubramaniyam
Contributor II

Hi Igor,

We are facing this SOM freeze or Kernel crash issue in 5 out of 100 SOM in the same batch.

And these issues are coming only in U-Boot 2014.04 and issue solved in U-Boot 2016.03.

We really need more details why these issues are coming only in U-Boot 2014.04 and solved in U-Boot 2016.03.

Changing the u-boot on all the devices is very difficult for us.

Thanks & Regards,

Bala

0 Kudos

3,230 Views
igorpadykov
NXP Employee
NXP Employee

Hi Bala

instabilities may be caused by both ddr errors and power supplies,

so one can run ddr test on problematic boards and update image with new

calibration coefficients

i.MX6/7 DDR Stress Test Tool V3.00 

for power supplies use recommendations given in Hardware Guide

IMX6DQ6SDLHDG, Hardware Development Guide for i.MX 6Quad, 6Dual, 6DualLite, 6Solo Families of Applic...

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

0 Kudos

3,230 Views
balasubramaniyam
Contributor II

Hi Igor,

I tested DDR with memtester application, it was working fine. And I probed the CPU voltage too, it was looking stable and I didn't find any fluctuation in it.

I'm getting this issue only in High CPU frequencies with 4 CPU core.

RAM details while running memtester application at higher frequencies (996MHz):

$ free -m
total used free shared buff/cache available
Mem: 2015 171 1770 1 73 1818
Swap: 0 0 0

RAM details while running memtester application at 792 MHz:
$ free -m
total used free shared buff/cache available
Mem: 2015 198 1693 1 123 1786
Swap: 0 0 0

Thanks & Regards,

Bala

0 Kudos

3,234 Views
igorpadykov
NXP Employee
NXP Employee

Hi Bala

such issue may be caused by wrong capacitors placement.

One can check the layout:  are the bulk caps placed outside of perimeter?
    If yes, then try the following experiments to determine if it is a hardware PCB issue.
    Hand solder 22uF capacitors directly under center of chip. Solder on top of the 0402 chips.
    Have software Increase voltages of ARM_CAP, SOC_CAP, PU_CAP.

Best regards
igor

0 Kudos

3,234 Views
balasubramaniyam
Contributor II

Hi Igor,

Q) Have software Increase voltages of ARM_CAP, SOC_CAP, PU_CAP.

[Ans]: Yes, we increased the voltage of ARM_CAP, SOC_CAP, PU_CAP and test the same cpustresstest application, still the issue occurs.

These are the default voltage for Quad SOM and we increased the ARM voltage to 1.275, 1.3 & 1.35 for 996MHz frequencies.

operating-points = <
/* kHz uV */
1200000 1275000
996000 1250000
852000 1250000
792000 1175000
396000 975000
>;
fsl,soc-operating-points = <
/* ARM kHz SOC-PU uV */
1200000 1275000
996000 1250000
852000 1250000
792000 1175000
396000 1175000
>;

Thanks & Regards,

Bala

0 Kudos

3,230 Views
igorpadykov
NXP Employee
NXP Employee

Hi Bala

 

please try to hand solder 22uF capacitors directly under center of chip.
Solder on top of the 0402 chips.

Best regards
igor

0 Kudos

3,230 Views
balasubramaniyam
Contributor II

Hi Igor,

we are already using 063 caps with 22uF capacity with a rated voltage of 603V

Thanks & Regards,

Bala

0 Kudos

3,230 Views
igorpadykov
NXP Employee
NXP Employee

Hi Bala

 

could you confirm that you tried to solder 22uF capacitors directly under center of chip.

Best regards
igor

0 Kudos

3,230 Views
balasubramaniyam
Contributor II

Hi Igor,


Please check the following image, the highlighted part is the 22uF capacitors which is solder under the center of chip.

picturemessage_tjl4qnp2.ore.png

Thanks & Regards,

Bala

0 Kudos

3,230 Views
igorpadykov
NXP Employee
NXP Employee

Hi Bala

 

what bsp used in the case.

Best regards
igor

0 Kudos

3,230 Views
balasubramaniyam
Contributor II

Hi Igor,

 

we tested with Linux 4.9.11 kernel and Yocto Morty rootfs

 

Thanks & Regards,

Bala

0 Kudos

3,230 Views
balasubramaniyam
Contributor II

Hi Igor,

And we are using U-Boot 2014.04 is this create any issue?

Thanks & Regards,

Bala

0 Kudos

3,230 Views
igorpadykov
NXP Employee
NXP Employee

Hi Bala

 

Linux 4.9.11 kernel and Yocto Morty rootfs are provided by
many vendors, for example variscite: GitHub - varigit/linux-imx: i.MX Linux Kernel 
Which one are you using ?
General recommendation is to use linux from official nxp source.codeaurora.org/external/imx/linux-imx
repository, preferably latest one

Best regards
igor

0 Kudos

3,230 Views
balasubramaniyam
Contributor II

Hi Igor,

 

We are using the customized Kernel(L4.9.11) and U-Boot(2014.04). But we took the base from official NXP source only.

 

Thanks & Regards,

Bala

0 Kudos

3,229 Views
igorpadykov
NXP Employee
NXP Employee

Hi Bala

 

suggest to try oficial nxp linux from source.codeaurora.org/external/imx/linux-imx

Support for customized Kernel(L4.9.11) and U-Boot(2014.04)

may be provided with extended support with NXP Professional Services | NXP 

Best regards
igor
0 Kudos

3,230 Views
balasubramaniyam
Contributor II

Hi Igor,

 

where can I find the release notes for the latest version?

 

Thanks & Regards,

Bala

0 Kudos

3,230 Views
igorpadykov
NXP Employee
NXP Employee

Hi Bala

 

it is provided in Linux documentation
i.MX Software | NXP 

 

Best regards
igor

0 Kudos

3,230 Views
yves_grillet
Contributor I

Hi Igor

With the Quad core, at the kernel boot, I've personnaly detected and seen that the CPU is heating a lot and can freeze

normaly, with an heater it should stop

Do you use an heatsink to cool down the CPU?

BR

Yves

0 Kudos