System freeze when using UART with DMA enabled

[daniil_s1001]

Hi, we are trying to enable DMA for communication via UART in rs485 mode.

We are using IMX6ULL processor on a custom board. We have build the linux kernel (v4.20.x) from this repository GitHub - Freescale/linux-fslc at 4.20.x+fslc with SDMA firmware found in Linux Binary Demo Files provided here: i.MX Software | NXP (we also tried the firmware from here sdma\imx - kernel/git/firmware/linux-firmware.git - Repository of firmware blobs for use with the Li... )

The DMA is enabled in device tree like this:

sdma: sdma@20ec000 {                                   
  compatible = "fsl,imx6ul-sdma", "fsl,imx6q-sdma",    
         "fsl,imx35-sdma";                             
  reg = <0x020ec000 0x4000>;                           
  interrupts = <GIC_SPI 2 IRQ_TYPE_LEVEL_HIGH>;        
  clocks = <&clks IMX6UL_CLK_SDMA>,                    
     <&clks IMX6UL_CLK_SDMA>;                          
  clock-names = "ipg", "ahb";                          
  #dma-cells = <3>;                                    
  fsl,sdma-ram-script-name = "imx/sdma/sdma-imx6q.bin";
};                                                    

uart4: serial@21f0000 {                         
  compatible = "fsl,imx6ul-uart",               
         "fsl,imx6q-uart";                      
  reg = <0x021f0000 0x4000>;                    
  interrupts = <GIC_SPI 29 IRQ_TYPE_LEVEL_HIGH>;
  clocks = <&clks IMX6UL_CLK_UART4_IPG>,        
     <&clks IMX6UL_CLK_UART4_SERIAL>;           
  clock-names = "ipg", "per";                   
  status = "disabled";                          
};                                             

&uart4 {
  pinctrl-names = "default";
  pinctrl-0 = <&pinctrl_uart4>;
  uart-has-rtscts;
    dmas = <&sdma 31 4 0>,
                 <&sdma 32 4 0>;
    dma-names = "rx", "tx";
  status = "okay";
};

It works fine for 10-20 minutes but then the UART or the whole system hangs without any special output to debug console.

We also tried to run our software on the demo image from Linux Binary Demo Files (i.MX Software | NXP ) for IMX6ULL  (fsl-image-validation-imx-imx6ul7d.sdcard) with u-boot-imx6ull14x14evk_sd.imx. But then the system hangs after the first time function tcdrain() is called.

Can anyone advice how can we solve this issue?

[j_lambrecht]

Hi Danyil Stas ,

We use kernel 4.19+fslc. And when we load the SDMA firmware (v3.3 or v3.5) the kernel crashes in +80% of the cases (it hangs, or a kernel panic). We use SDMA for SAI (audio), but also when disabling SAI to use sdma it crashes.

Just like you it crashes after loading the driver, so not related to the latest kernel 5.2 fix 3f93a4f297961c12bb17aa16cb3a4d1291823cae. And we use a imx6UL (and imx6ULL), so not containing a HW bug that is fixed now (http://lists.infradead.org/pipermail/linux-arm-kernel/2019-June/658381.html).

Do you already know a solution?

Kind regards,

Jürgen

[daniil_s1001]

    Hi, Jürgen Lambrecht

We didn't find the solution to our problem and decided to use different SoC for our board instead (stm32mp153).

But looks like Ramon in a sibling comment provided a solution to the problem: https://community.nxp.com/message/1148345?commentID=1148345&et=watches.email.thread#comment-1239371 

    Best regards,

Danyil

[j_lambrecht]

See my mail sent to linux-arm-kernel FYI: imx-sdma firmware is not compatible with SLUB slab allocator 

We are updating our kernel on our custom board with an iMX6UL from 3.14 to 4.19, and when loading linux-firmware/imx/sdma/sdma-imx6q.bin v3.5 the kernel hangs when booting, only "Starting kernel ..." is printed (by uBoot I think).

When I remove the sdma bin (built-in the kernel via EXTRA_FIRMWARE), then it boots.

I found that the slab allocator (menu config -> General setup -> Choose SLAB allocator) is the culprit: when using imx_v6_v7_defconfig the kernel boots, but not with our config (based on a tiny config I think).
And the config that makes the difference is the slab allocator: with SLUB or SLAB the kernel (with sdma bin) boots, but not with SLOB.

[igorpadykov]

Hi  Danyil

fslc kernels and kernel 4.20.x is not supported by nxp, one

can try to rebuild custom software using nxp linux from source.codeaurora.org/external/imx/linux-imx

repository

linux-imx - i.MX Linux kernel 

linux documentation

i.MX Software | NXP 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[daniil_s1001]

Hi, Igor

We tried to use kernel from source.codeaurora.org/external/imx/linux-imx too (branch imx_4.14.98_2.0.0_ga).

The system hangs as soon as our software calls tcdrain() function.

[igorpadykov]

Hi  Danyil

one can try to reproduce issue on i.MX6ULL EVK with Demo Images

Linux Binary Demo Files - i.MX 6UltraLite, i.MX 6ULL, i.MX 7Dual

Best regards
igor

[daniil_s1001]

Hi, Igor

I could reproduce the issue on the Demo Image (fsl-image-validation-imx-imx6ul7d.sdcard with u-boot-imx6ull14x14evk_sd.imx).

Here is the minimal program that causes the system to freeze:

#include <fcntl.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <termios.h>

#include <linux/serial.h>
#include <string.h>


static char buf[100000];

int main(int argc, const char *argv[])
{
  int fd = open("/dev/ttymxc1", O_RDWR);
  if (fd < 0)
  {
    printf("ERR\n");
    return 1;
  }
  struct serial_rs485 rs485conf;
  memset(&rs485conf, 0, sizeof(rs485conf));

  rs485conf.flags |= SER_RS485_ENABLED;
  rs485conf.flags |= SER_RS485_RTS_ON_SEND;
  rs485conf.flags &= ~(SER_RS485_RTS_AFTER_SEND);

  rs485conf.delay_rts_before_send = 0;
  rs485conf.delay_rts_after_send = 0;

  if (ioctl(fd, TIOCSRS485, &rs485conf) < 0) {
      close(fd);
      printf("Unable to set rs485 mode\n");
      return 2;
  }

  struct termios newtio;
  tcgetattr(fd, &newtio);
  cfsetospeed(&newtio, B500000);
  cfsetispeed(&newtio, B500000);
  tcsetattr(fd, TCSANOW, &newtio);

  memset(buf, 0xAA, sizeof(buf));
  size_t count = sizeof(buf);
  char *pos = buf;
  while(count > 0)
  {
      ssize_t r = write(fd, pos, count);
      if (r < 0)
      {
          printf("Error writing to UART\n");
          return 3;
      }
      count -= r;
      pos += r;
  }
  tcdrain(fd);
  return 0;
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[ramon_effegi]

I had a similar issue using the rs485 ( on the linux-fslc but I had the same issue with the linux-imx).

I saw that the ull was always in interrupt.

I solved it with the following patch (tested on)

@@ -481,6 +490,11 @@ static inline void imx_transmit_buffer(struct imx_port *sport)
writel(temp, sport->port.membase + UCR1);
schedule_work(&sport->tsk_dma_tx);
}
+
+ writel(readl(sport->port.membase + UCR4) & ~UCR4_TCEN, sport->port.membase + UCR4);
+ return;
}

if (sport->dma_is_txing)
@@ -518,8 +532,16 @@ static void dma_tx_callback(void *data)
}
sport->dma_is_txing = 0;

+ // turn off dma
+ u32 ucr1 = readl(sport->port.membase + UCR1);
+ ucr1 &= ~UCR1_TDMAEN;
+ writel(ucr1, sport->port.membase + UCR1);
+
xmit->tail = (xmit->tail + sport->tx_bytes) & (UART_XMIT_SIZE - 1);
sport->port.icount.tx += sport->tx_bytes;
spin_unlock_irqrestore(&sport->port.lock, flags);

dma_unmap_sg(sport->port.dev, sgl, sport->dma_tx_nents, DMA_TO_DEVICE);

@@ -532,6 +554,14 @@ static void dma_tx_callback(void *data)

if (!uart_circ_empty(xmit) && !uart_tx_stopped(&sport->port))
schedule_work(&sport->tsk_dma_tx);
+ else {
+ spin_lock_irqsave(&sport->port.lock, flags);
+ // Abilito interrupt di shift register vuoto
+ writel(readl(sport->port.membase + UCR4) | UCR4_TCEN, sport->port.membase + UCR4);
+ spin_unlock_irqrestore(&sport->port.lock, flags);
+ }

if (waitqueue_active(&sport->dma_wait)) {
wake_up(&sport->dma_wait);

@@ -1090,6 +1128,11 @@ static void dma_rx_callback(void *data)
return;
}

+ if( state.residue == 0 )
+ count = 0;
+ else
count = RX_BUF_SIZE - state.residue;
sport->rx_buf.buf_info[sport->rx_buf.cur_idx].filled = true;
sport->rx_buf.buf_info[sport->rx_buf.cur_idx].rx_bytes = count;

Hope it helps,

Ramon La Pietra

[daniil_s1001]

    Hi, Ramon

Thanks for the information!

We decided to use different SoC for our board though (stm32mp153)...

    Best regards,

Danyil

[igorpadykov]

Hi  Danyil

rs485 is not supported by nxp in its bsps, please try to test without

it, for example with uart loopback.

For rs485 issues may be suggested to proceed with extended support of

NXP Professional Services | NXP 

Best regards
igor