We are stuck in an urgent and strange issue which needs some help.
1) Please allow me to describe the Application Operation first.
Our customer is doing a file I/O stress test and within hours to days fail with a Linux kernel panic, thus cause i.MX6SL board to be locked up and requiring a power cycle to recover, and it happens all the time.
- i.MX6SL <-- UART2--> Control board (TI MCU)
- Communicates through UART2 (baudrate 115200, with RTS/CTS enabled), and start running the file I/O stress test application from the control board.
- The sequence operation of the file I/O stress test application: Continuously open file, write data, close file, then open file, read data, close file, compare the data. (Note: data is stored in DRAM; not the physical files.)
Attachment is our current cpuidle-imx6sl.c
Sorry, this kernel panic only happens while using Control board with customer's file I/O stress test application, cannot reproduce on EVB.
2) The following is the Kernel Panic Log:
[39585.024726] Internal error: Oops: 17 [#1] PREEMPT SMP ARM[39585.030369] Modules linked in: bcmdhd [last unloaded: evbug][39585.036109] CPU: 0 PID: 0 Comm: swapper/0 Not tainted3.14.28-1.0.0_ga+g91cf351 #1[39585.043697] task: 80d8bc28 ti: 80d80000 task.ti: 80d80000[39585.049131] PC is at ldo2p5_dummy_probe+0x10/0x90[39585.053858] LR is at imx6sl_enter_wait+0x90/0x94[39585.058493] pc : [<8001f974>] lr : [<8001f960>] psr: 400e0093[39585.058493] sp : 80d81f28 ip : fffffffa fp : 00000000[39585.069988] r10: 80d80000 r9 : 00002400 r8 : 9b263479[39585.075227] r7 : ab7400d4 r6 : 00000000 r5 : 00000001 r4 : 00000000[39585.081767] r3 : 00000000 r2 : 80d8e674 r1 : 80df81a0 r0 : 00000000[39585.088313] Flags: nZcv IRQs off FIQs on Mode SVC_32 ISA ARM Segmentkernel[39585.095723] Control: 10c53c7d Table: a885004a DAC: 00000015[39585.101483] Process swapper/0 (pid: 0, stack limit = 0x80d80238)[39585.107503] Stack: (0x80d81f28 to 0x80d82000)[39585.111880] 1f20: ffffffff 00000000 80df8b00 800237a480d8e7a8 00000001[39585.120080] 1f40: ab7400d0 8001f8e0 8001f8d0 80d8e7a8 80d8e75c ab7400d0ab7400d4 804a0230[39585.128277] 1f60: 9b263479 00002400 00000005 ab7400d0 00000000 ab7400d000000000 00000001[39585.136476] 1f80: ab7400d4 80e4f808 80d8e75c 804a03d0 00000000 80d8000080d88574 806f238c[39585.144673] 1fa0: 80d80038 80df72bd 80df72bd 8000f0bc 00000000 80066cb4ffffffff 80d2db10[39585.152871] 1fc0: ffffffff ffffffff 80d2d58c 00000000 00000000 80d7126000000000 10c53c7d[39585.161067] 1fe0: 80d884fc 80d7125c 80d8ccc0 8000406a 412fc09a 8000807400000000 00000000[39585.169279] [<8001f974>] (ldo2p5_dummy_probe) from [<ab7400d0>] (0xab7400d0)[39585.176350] Code: e92d4070 e24dd020 e59f207c e1a03000 (e590c130)[39585.182461] ---[ end trace 8249e52007c6a60b ]---[39585.187096] Kernel panic - not syncing: Attempted to kill the idle task![39588.850973] imx-uart 2024000.serial: overwrite![39592.850942] imx-uart 2024000.serial: overwrite!
3) I also checked the LDO 2P5 in “i.MX 6SoloLite Applications Processor Reference Manual”.
The LDO_2P5 module on the chip implements a programmable linear-regulator function from a higher analog supply voltage (2.8V-3.3V) to produce a nominal 2.5V output voltage. The output of the regulator can be programmed in 25mV steps from 2.0V to 2.75V. The regulator has been designed to be stable with a minimum external low-ESR decoupling capacitance, though the actual capacitance required should be determined by the application. A programmable brown-out detector is included in the regulator which can be used by the system to determine when the load capability of the regulator is being exceeded to take the necessary steps.
- There is no memory leakage
- Not enter standby mode (no suspend/resume)
- CPU utilization is 3.3% while running the stress test application
- Not use eFuse and LVDS
- No reboot log is shown
- “ldo2p5_dummy_probe()” only be called once while system boot up (loading LDO module), afterward this function won’t be called.
4) So my questions would be:
- Any idea what could cause this kernel panic issue?
- Who else will call this ldo2p5_dummy_probe()? and When?
- What's the relationship between ldo2p5_dummy_probe() and imx6sl_enter_wait()?
- Is there any possibility caused by DVFS (Dynamic Voltage Frequency Scaling)?
Any pieces of advice can share with us to help to proceed, we will be appreciated.
Thanks,
Terry
