i.MX8MQ: LCDIF FIFO starved by PCIe traffic

[mntmn]

I've been trying to solve this problem for a long time, but I might be missing technical information that is not in the reference manuals.

In our application (MNT Reform laptop) I want to enable a dual display application, where I have to drive the internal 1920x1080 display with LCDIF to be able to use DCSS for HDMI. The problem is when using PCIe devices together with LCDIF, PCIe transfers take priority instead of the LCDIF master. In our case this happens with an NVMe disk. A test case:

dd if=/dev/zero of=./testfile bs=1M count=100 oflag=direct

Where "testfile" is on the NVMe SSD. This will completely glitch the display while the transfer is active.

I added a kernel thread to the mxsfb driver that monitors the STAT register of LCDIF, especially the LFIFO_COUNT part. Whenever glitches appear, the fifo gets close to 0 or 0. This confirms that PCIe accesses lead to LCDIF FIFO underflow. I have tried to reconfigure NOC, but as far as I can tell from sparse documentation and experiments, LCDIF is in "Other Masters" in the NOC, at 0x32700280. I confirmed this by limiting the bandwidth hear which makes the display output starve. But I cannot find any way or register to lower the PCIe transfer priority. I imagine I could also do this on demand in a kernel thread monitoring the LFIFO status. But I cannot find anything in the documentation which could regulate PCIe master access priority/QoS/bandwidth. Is there any possibility to do this?

I have also tried another approach, which is to disable the MASTER mode of LCDIF and feed LCDIF_DATA in a kernel thread. But the accesses to this register seem to be too slow to be able to fill up the FIFO for this resolution. I was thinking to feed the data using Cortex-M4 or SDMA, but apparently the LCDIF_DATA pipeline is even more problematic than the master mode of LCDIF.

I have invested countless hours in trying to understand this part of the hardware and come up with workarounds, but at this point I need your help to progress any further.

I'm looking forward to any suggestions.

In the attached video, you can see the glitches. Near the end of the video you see the extreme case when writing to disk with dd.

[Luciano_Mougenot]

Solution solved our dark-screen issue as well. 

System with eDP/HDMI LCDs and newer NAND SSDs (older/slower were fine).

What could be monitored to then allow the system to detect and respond accordingly (ex: power cycle to recover)?

[mntmn]

Unexpectedly, I was able to obtain a workaround for this via an NXP engineer and combining with other puzzle pieces that I already had. Keep in mind that this is undocumented and unsupported, as far as I know.

There is a QoS controller (QoSC) in i.MX8MQ which is mentioned in the AIPS2 memory map in the reference manual, but the registers are not (yet?) documented. It is possible to restrict the number of outstanding (queued?) transactions of masters via this controller's registers. I was given information on how to do this for the PCIe controllers 1 and 2. In my case, this fixed the conflict between PCIe and LCDIF. I tested this first with memtool and then wrote the following function that I patched into my kernel:

void imx8mq_pcie_qos_for_lcdif(void)
{
  void __iomem *qosc = ioremap(0x307f0000, 0x2100);
  // clock and unlock QoSC registers
  writel(0x0, qosc); // un-clock-gate qosc
  writel(0x1, qosc); // enable qosc, manual mode
  writel(0x0, qosc+0x60); // enable advanced master qos control register access

  // limit number of outstanding transactions for PCIe1
  writel(0x0, qosc+0x1000); // un-clock-gate, un-reset adv regs for pcie1 master
  writel(0x1, qosc+0x1000); // advanced qos ctrl enable
  writel(0x01010100, qosc+0x1050); // 1 otW
  writel(0x01010100, qosc+0x1060); // 1 otR
  writel(0x01010100, qosc+0x1070); // 1 otT
  writel(0x1, qosc+0x1000); // advanced qos ctrl enable

  // limit number of outstanding transactions for PCIe2
  writel(0x0, qosc+0x2000);
  writel(0x1, qosc+0x2000);
  writel(0x01010100, qosc+0x2050);
  writel(0x01010100, qosc+0x2060);
  writel(0x01010100, qosc+0x2070);
  writel(0x1, qosc+0x2000);

  iounmap(qosc);
}

Depending on the bandwidth (resolution * refresh rate) you're requiring from LCDIF, you might also need to restrict the CPU bandwidth via the NOC limiter, but this affects CPU (bus) performance. For me it was enough to lower the display refresh rate to around 50Hz for 1920x1080@32bit.

Hope it helps!

[lmurillo]

Hello @mntmn,

Thanks a lot for sharing this. If I understood correctly, you call to this function that limits the PCIe from the LCD driver in your case, right?

Also, do you have some example of how to configure the NOC to limit the bandwidth from that part?

[Yuri]

@mntmn 
Hello,

  I' ve sent You some comments in e-mail.

Regards,
Yuri.

[lmurillo]

Hello @Yuri 

I am having a pretty similar problem in the i.MX8MQ that the one described by @mntmn . In my case, I have connected a camera to the USB3 and an NVMe SSD to the PCIe. 

One test that I did was that while I am capturing from the camera, at the same time I execute the command:

dd if=/dev/zero of=/tmp/ssd/testfile bs=1G count=1 oflag=direct

 where /tmp/ssd is where I mounted the SSD device. As soon as I execute this command, the camera starts failing with timeout errors.

Do you have any suggestions on how to solve this problem?

Thanks,

Luis

[Yuri]

Hello,

  I've sent You private message.

Regards,
Yuri.