i.mx8MM tty buffer full, DMA donot insert data to tty buffer any more

[simon_cai]

kernel version: 5.4.7 (yocto 3.0)
soc: imx8mm

By using the attachment testing program, connect the PC and IMX8 via a two-wire serial port. Run the program "./test /dev/ttymxc2 9600" on the IMX side. On the PC side, set a random string for cyclic transmission with an interval of 10ms, as shown in the figure below.

simon_cai_0-1680174924052.jpeg

The attached  testing program on the IMX side receives 20 bytes per second. Due to the slower receiving speed on the IMX side compared to the sending speed on the PC side, the receive buffer on the IMX side quickly becomes full and overflows. At this point, it is found that the application program on the IMX side cannot receive data through ttymxc2.

simon_cai_1-1680174971975.png

we suppose that it is something wrong with  imx_uart_dma_rx_callback   function in this test situation （/drivers/tty/serial/imx.c）

Do you have any idea to fixed the issue? 

Thanks

[Dhruvit]

Hi @simon_cai,

I hope you are doing well.

This issue could arise if flow control is not used.

Please try using the latest kernel as it has some improvements in the/drivers/tty/serial/imx.c driver.

One can refer to /Documentation/devicetree/bindings/serial/fsl-imx-uart.yaml where uart and DMA-related configurations are documented.

Additionally, One can take look at the uart unit test at /imx/imx-test/log/test/mxc_uart_test

Thanks & Regards,
Dhruvit Vasavada.

[simon_cai]

@Dhruvit   great thanks

We are preparing it with lastest bsp and nxp official evb

I read the L6 mix.c uart driver and sdma driver，maybe it'll not fix this issue.

But we will do this testing when some hardware is really.

Please also see my reply to Sherry，

Dmamode cause CPU stalling error，but CPU uart mode don't！

Please help to comment. Or any ideas to fix It

Thank you so much

[Dhruvit]

Hi @simon_cai,

I hope you are doing well.

Is there any error logs printed after CPU stalling ?

Have you tried applying patches shared by @sherry_sun ?

Please provide me with boot logs for further debugging.

Thanks & Regards,
Dhruvit Vasavada

[Dhruvit]

Hi @simon_cai,

I hope you are doing well.

Waiting for your reply.

Thanks & Regards,
Dhruvit Vasavada

[simon_cai]

Sorry
Kernel version is 5.4.70_2.3.0

[sherry_sun]

Hi Simon,

If I understand it correctly, you are not using the uart flow control. And the uart baud rate here is very slow -- 9600bps, that is to say, it takes more than 1ms to receive a byte(10bits/9600*1000=1.0416ms).

While the sender seems to be sending more than a hundred data every 10ms, this will definitely cause the receiver to overflow, because even if we don't consider the time required for sw processing, the receiver can only process less than 10 bytes of data every 10ms.

So seems to me this is not an issue, it is an expected behavior, too much data will cause the tty buffer overflow, tty core layer will discard the data that cannot be processed. The reason why we need flow control is to avoid such case.

It should no error in imx_uart_dma_rx_callback(), a simple way to prove that, you can try to disable uart dma mode as the following code, try to use cpu mode uart, I believe you can also observe the same behavior here.

diff --git a/arch/arm64/boot/dts/freescale/imx8mm.dtsi b/arch/arm64/boot/dts/freescale/imx8mm.dtsi
index 1c5b0b339c16..803976d693b7 100755
--- a/arch/arm64/boot/dts/freescale/imx8mm.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8mm.dtsi
@@ -958,8 +958,6 @@
clocks = <&clk IMX8MM_CLK_UART3_ROOT>,
<&clk IMX8MM_CLK_UART3_ROOT>;
clock-names = "ipg", "per";
- dmas = <&sdma1 26 4 0>, <&sdma1 27 4 0>;
- dma-names = "rx", "tx";
status = "disabled";
};

Best Reagrds

Sherry

[simon_cai]

Hi Sherry

After do testing about 3days，

1，CPU_MODE don't cause cpu stalling error but DMA_MODE does！！！

2，both mode lost data。

both cause frame error.and break error.

Dma_mode will happen overruns error(oe)

We can acpect lost date or frame error or break error, because it doesn't make CPU stalling.

But in dma mode，cpu stalling will happen after 12-24hours。

I think sdma channel stop to copy uart register data ，and cause uart oe error，and then CPU stalling happens。。。i don't know why...

The defers of CPU mode is that cpu never stop to copy data from uart receiver register

We need DMA_MODE in our field situation

We need your help to fix this issue why causing CPU stalling in dmamode

Such great thanks

[sherry_sun]

Hi Simon,

Thanks for your update, so now the highlight issue is DMA mode uart will cause CPU hang after 12-24hours continuous data receive, right?

We will check it internally with i.mx DMA owner. But at the same time, can you please try the following tests to help us identify this issue more faster? Thanks.

1. Please try the latest L6.1.1_1.0.0 BSP release and check if this issue still can be observed.

https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applicat...

2. Please try to increase the baud rate(such as 115200 or 3M) or increase the PC transmission interval(such as 200ms).

3. Add the fsl,dma-info property under the uart dts node and tweak DMA period number and size.

fsl,dma-info first cell contains the size of DMA buffer chunks, second cell contains the amount of chunks used for the device. Multiplying both numbers is the total size of memory used for receiving data. When not being configured the system will use default settings(<1024 16>), which are sensible for most use cases. If you need low latency processing on slow connections this needs to be configured appropriately. For more details, please check this patch(https://lore.kernel.org/all/[email protected]/).

For example:

&uart3 {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_uart3>;
    assigned-clocks = <&clk IMX8MM_CLK_UART3>;
    assigned-clock-parents = <&clk IMX8MM_SYS_PLL1_80M>;
    fsl,dma-info = <256 64>;
    status = "okay";
};

Best regards

Sherry

[sherry_sun]

And 4. Please try if the attaches patch work for you.

[simon_cai]

Hi Sherry

thanks for you quickly feedback.

yes,we didn't use uart flow control because of our field condition.

about this 10ms,  I have additional explaintion.

this 10ms is an interval not a period as shown in the figure below

that is, send 200 bytes to IMX,wait 10ms,send 200 bytes to imx immediately  and wait 10ms send 200 bytes.......

it is not , send 200bytes, wait Sending completed，and then wait 10ms，and send 200bytes, wait Sending completed.....

simon_cai_0-1680250700271.png

Suppose, it may fix UART overrun error by using cpu mode uart ，

but it will still happen UART buffer overflow error AS n_tty buffer is full becaue of IMX reading slower than PC sending.

Of course, anyway，we will try CPU_MODE suggestion.

give you feedback soon,thank you Sherry,  Thanks

[simon_cai]

@sherry_sun