Kernel lockup/hang when attempting to reuse console UART

[sethrichards]

Hi,

I'm integrating a Linux 3.0.35 kernel on an i.MX6D platform (based loosely on the SABRE SDB design), and am encountering a kernel hang/lockup related to the UARTs. We're attempting to disable the serial console by specifying "console=" on the u-boot kernel command line, and then reuse the serial port for other purposes. However, we occasionally encounter locks or hangs when we attempt to open the UART from userspace.

It appears to be very similar to the problem described in this thread, but I don't see a clear resolution there:

https://community.freescale.com/thread/308092

I've attempted debugging this issue by adding a USB-serial converter to the platform and using that as a serial console, and it looks like it may be getting hung up in the imx_startup function in the i.MX serial driver; it appears that things may go off the rails during configuration of the UARTs registers. I dumped the registers to the USB console and it appears there are some leftover configuration changes from either U-boot or from kernel console setup that are apparently causing problems.

I attempted resetting the UART by writing to the SRST bit in UCR2 and it doesn't seem to have any effect. Does this bit not clear out the config registers and any interrupts? Has anyone else run into similar problems?

Thanks.

[rajendrabaniyav]

I am using Kernel version 3.0.35 on iMX6 solo and I have same issue.

To disable serial log, I configure bootargs=null, though no messages are seen during kernel boot up, any read/write operation on /dev/ttymxc1 lead to system hang.

[rajendrabaniyav]

sorry, its console=null above.

[cyborgnegotiato]

Hi Rajendra,

Please try new Linux kernel version.

Jozef

[rajendrabaniyav]

Hello Jozef,

We can't move to latest kernel version as we are in advanced stage of project. Is this possible to get patch on 3.0.35?

regards

Rajendra

[cyborgnegotiato]

Hi,

3.0.35 is old version, we have released another newest versions. There is no specific patch for 3.0.35, you must backport patches by yourself.

Jozef

[alejandrolozan1]

Hi,

I have not tested this, but I found the below steps. I wonder if you can try:

* Configure the kernel and disable (just search for it in config):

CONFIG_SERIAL_CORE_CONSOLE

CONFIG_SERIAL_IMX_CONSOLE

*In /etc/inittab comment

#mxc::respawn:/etc/init.d/rc_mxc.S

* In U-boot set the console to

setenv console console=none

Please let  me know how it goes.

Regards,

Alejandro

[sethrichards2]

Hi Alejandro,

We had previously tried removing those config variables from the kernel config, and made those u-boot changes, and they reduce the number of locks/hangs but don't completely eliminate them. We have our own custom getty being called from /etc/inittab, which we need for our final product. Thanks.

[sethrichards2]

To follow up, my investigation indicates that the problem seems to be a leftover interrupt from a previous initialization of the UART; when we go to open the UART in userspace I think this interrupt (specifically RRDY, which apparently can't be cleared by writing to the USR registers) fires off before initialization is finished and causes problems.

I thought I should be able to clear out any leftover interrupts by hitting the SRST bit in the UCR2 register; there is code to do this in /drivers/tty/serial/imx.c around line 997 in the Freescale Linux 3.0 code. In the stock code it's only performed if the driver is setting up the UART as an IRDA port, but when I change the if statement from if(USE_IRDA(sport)){ to if(1){ it doesn't seem to affect the status registers. Am I misreading what this code or the bit in the config register does?  Thanks.

[alejandrolozan1]

Hi,

Sorry for the delay. The RRDY should be set only when there is data in the Rx FIFO. Also writing 0 to SRST should reset the FIFO and registers. I wonder how you noticed that this interrupt was the problem. Cna you share more details about it?

Thanks,

Alejandro

[sethrichards2]

Sure, I can provide some more detail. I tracked it down to this section of code by using printks to determine how far it was getting in the TTY open process before the system hung, which obviously isn't ideal but got me close enough. When I dumped the USR registers to the USB debug console, the RRDY interrupt was set along with some others. Some more investigation showed that if I didn't enable the RRDY interrupt when the .membase address matched the one for the UART we're having problems with, the problem went away (although obviously the UART wasn't usable after that). So I'm assuming that interrupt is causing us issues.

[alejandrolozan1]

Ok, Have you tried to reproduce the issue in the SABRE-SD. I will try to reproduce the problem on my side and see if I can find the issue.

Regards,

Alejandro

[sethrichards2]

Great, thanks. I have not seen the issue on the SABRE board, but one of our other engineers has. To reproduce the issue:

1) Disable the console with "console=" on the u-boot command line.

2) SSH/telnet into the board and run "while true; do stty -F /dev/ttymxc4 ; done" at the command prompt (change to fit the UART used on the SABRE board). Sometimes it will lock up immediately, but it usually does within a couple minutes.

[sethrichards2]

Alejandro,

Have you managed to recreate this problem on your end? Any insights into what's going on? Thanks.

  --Seth

[alejandrolozan1]

Hi Seth,

I have tried with kernel 3.10.17 with Yocto and it is not failing. Let me see how I can test it with 3.0.35.

Best Regards,

Alejandro

[sethrichards2]

Alejandro,

I got one of our SABRE boards and reprogrammed it with the stock Freescale 3.0.35 demo image from here:

SABRE Board Reference Design|Freescale

And I'm able to reproduce the problem. I'll check it with the 3.10.17 image from the same page and see if I can confirm your results. Have you had a chance to check against the 3.0.35 kernel on your end?

  --Seth

[fabio_estevam]

Hi Seth,

Please check if you have this patch applied:

http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/commit/drivers/tty/serial/imx.c?h=imx_3....

Regards,

Fabio Estevam

[ramgopalkota]

HI Fabio

I am facing the same issue. I am not able to access the patch. Can you send me the patch please ? I will verify if it is present or not in my kernel code?

[ramgopalkota]

HI Fabio

I am facing the same issue. I am not able to access the patch. Can you send me the patch please ? I will verify if it is present or not ?

[fabio_estevam]

Hi Ramgopal,

Path is available at http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/commit/drivers/tty/serial/imx.c?h=imx_3.... 

Please note that this applies against a 3.0 kernel, which is very old. I suggest you to use a more recent kernel, such as 4.9.

Regards,

Fabio Estevam

[ramgopalkota]

Hi Fabio,

I am using 3.x kernel since I need RS485 support in imx.c

Can you point to branch where RS485 via CTS is supported ?

Ramgopal Kota

On Mon, Nov 6, 2017 at 4:16 PM, FabioEstevam <admin@community.nxp.com>