i.MX6 kernel panic on 921600kbps uart comminication

[yurirellosa]

Hi All

Information about our device

yocto version: 1.7

kernel version: 3.10.17

serial driver version: 3.10.17

The application that manages the uart on our device has an open->close cycle on changing serial port settings or during retry error processes.

During our test we flood the serial port with data and during the [close] process, a kernel panic error occurs and the serial driver no longer functions.

Sample kernel panic message below:

Unhandled fault: imprecise external abort (0x1406) at 0x76f89000
Internal error: : 1406 [#1] PREEMPT SMP ARM
Modules linked in: aerial_sdio(O)
CPU: 0 PID: 1104 Comm: ft-test Tainted: G O 3.10.17yocto-for-pad+g33597e3 #93
task: ac72e940 ti: ad2fe000 task.ti: ad2fe000
PC is at _raw_spin_unlock_irqrestore+0x30/0x58
LR is at uart_close+0x1b4/0x1c4
pc : [<805d2f74>] lr : [<80314b60>] psr: 40030013
sp : ad2ffe68 ip : ad2ffe78 fp : ad2ffe74
r10: 00000001 r9 : 00000000 r8 : ac697528
r7 : ac44de10 r6 : a0030013 r5 : ac7c1600 r4 : ac3510c4
r3 : 00005c0f r2 : 00000002 r1 : a0030013 r0 : ac44de10
Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Control: 10c53c7d Table: 3d18004a DAC: 00000015
Process ft-test (pid: 1104, stack limit = 0xad2fe238)
Stack: (0xad2ffe68 to 0xad300000)
fe60: ad2ffe9c ad2ffe78 80314b60 805d2f50 ac7c1600 00000000
fe80: ad160480 ac587010 ac697528 00000000 ad2fff14 ad2ffea0 802f92d0 803149b8
fea0: ad2ffebc ad2ffeb0 800bf280 800f8708 ad2fff34 ad2ffec0 800de79c 800bf26c
fec0: acfa5a40 00000000 ad2b2478 00000001 ad160488 ac697528 800f17c4 00000000
fee0: 00000000 00000000 ac72e940 ad160480 00000000 aa3347f8 ac587010 ac697528
ff00: ac697528 00000008 ad2fff54 ad2fff18 800f85e4 802f91d8 00000000 00000000
ff20: ad2fe000 ad160488 ad2fff54 ac72ec20 00000000 80900d70 ac72e940 8000e9e4
ff40: ad2fe000 00000000 ad2fff64 ad2fff58 800f8864 800f8554 ad2fff8c ad2fff68
ff60: 80049b48 800f8858 ad2fff94 299d299d ad2fe010 8000e9e4 ad2fffb0 ad2fe000
ff80: ad2fffac ad2fff90 800123ac 80049a94 000152e8 00000000 0000a7f4 00000006
ffa0: 00000000 ad2fffb0 8000e800 80012310 00000000 00000000 00000000 00000001
ffc0: 000152e8 00000000 0000a7f4 00000006 00000000 00000000 76f3c000 00000000
ffe0: 00000000 7eea4c4c 0000b754 763fd2ec 400a0010 00000003 3ff7e821 3ff7ec21
[<805d2f74>] (_raw_spin_unlock_irqrestore+0x30/0x58) from [<80314b60>] (uart_close+0x1b4/0x1c4)
[<80314b60>] (uart_close+0x1b4/0x1c4) from [<802f92d0>] (tty_release+0x104/0x590)
[<802f92d0>] (tty_release+0x104/0x590) from [<800f85e4>] (__fput+0x9c/0x250)
[<800f85e4>] (__fput+0x9c/0x250) from [<800f8864>] (____fput+0x18/0x1c)
[<800f8864>] (____fput+0x18/0x1c) from [<80049b48>] (task_work_run+0xc0/0xfc)
[<80049b48>] (task_work_run+0xc0/0xfc) from [<800123ac>] (do_work_pending+0xa8/0xb4)
[<800123ac>] (do_work_pending+0xa8/0xb4) from [<8000e800>] (work_pending+0xc/0x20)
Code: e1c030b0 f57ff04f e320f004 e121f001 (e3a00001)
---[ end trace 6b3720b3c1a690ba ]---

We have tried looking for patches to the serial driver that might fix this, but still unable to find a proper solution.

Also, tried putting a [wait_for_completion] inside the [imx_shutdown] function, waiting for IRQs to finish.

This prevented the serial driver, but the problem still occurs when the flooding test continues.

Please advise, thank you

Yuri

[igorpadykov]

Hi Yuri

solution could be use uart flow control. Also recommended to try

latest L4.1.15 nxp kernels as since L3.10.17 uart driver greatly improved.

BSPs   link :
http://www.nxp.com/products/microcontrollers-and-processors/arm-processors/i.mx-applications-process...
================================================================
Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[yurirellosa]

Hello Igor

Regarding flow control, the module attached to the uart port does not use flow control,

so it is not possible to implement this.

As we cannot upgrade the whole kernel, we have only tried upgrading the imx uart driver 

to the latest driver, the situation fared much better compared than it used.

Crashing within 5 secs now occurs around after 5 mins.

Is it perhaps, that this problem would not be remedied unless the kernel is upgraded?

I also wonder if the i.MX is tested under such stress on these baudrates(921kbps)

Any additional advice would be much appreciated.

Best Regards

[conteczxy]

As we cannot upgrade the whole kernel, we have only tried upgrading the imx uart driver 

to the latest driver, the situation fared much better compared than it used.

Crashing within 5 secs now occurs around after 5 mins.

which version have you tried?

Could you share the latest imx uart driver?

Thanks,

[yurirellosa]

Hi Xiyuan Zhou

We used Freescale's Linux 4.1.15  version of the kernel.

Follow the git from link below:

linux-2.6-imx.git - Freescale i.MX Linux Tree 

What we did was replace the following files

・/drivers/tty/serial/imx.c

(http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/plain/drivers/tty/serial/imx.c?h=imx_4.1... )

・/drivers/tty/serial/serial_core.c

(http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/plain/drivers/tty/serial/serial_core.c?h... )

There will probably quite some errors because of version differences, but we mostly just needed to work on these two files.

Be aware that this does NOT really solve the problem, just makes it more "stable".

What we did was reduce the amount of open->close calls on the serial driver in the application layer.

Regards

[conteczxy]

Hello,

I met similar problem as you.

I open/close the uart frequently, then sometime the kernel will crash, same kernel panic message as you post above,

And how do you update your serial driver,which version are you using;

I want to fix the problem in my 3.0.35 kernel, but don't know how to.

Best Regards,