swiotlb buffer is full when switching to bigger memory

[heinzhjb]

We are using i.MX8mp with 2GB/4GB of RAM.

Since we switched from 2GB to 4GB RAM, there are problems using USB.
When the device wants to send data, the following error happens:

[ 816.978697] dwc3 38100000.dwc3: swiotlb buffer is full (sz: 380928 bytes), total 32768 (slots), used 248 (slots)
[ 816.988935] dwc3 38100000.dwc3: failed to map SGs

Earlier in booting the kernel, the following line appeared:
[ 0.000000] software IO TLB: mapped [mem 0x00000000f8000000-0x00000000fc000000] (64MB)

Any hint how to solve the problem?

Edit: Using Kernel 5.10.35 hardknott

Thanks
Bernd

[heinzhjb]

Have digged a little deeper into the code.

When we use 4GB memory the memory extends beyond the 32bit 4GByte border.
This seems to trigger swiotlb module to be initted and used later on when transferring data.

The USB dwc3 gadget driver uses dma_map_sg. This leads to using swiotlb for higher memory blocks. But too large blocks of memory are requested for swiotlb to serve.

[ 139.349965] swiotlb_tbl_map_single+0x394/0x468
[ 139.354494] swiotlb_map+0x84/0x1f8
[ 139.357984] dma_direct_map_sg+0x104/0x2a8
[ 139.362080] dma_map_sg_attrs+0x28/0x58
[ 139.365918] usb_gadget_map_request_by_dev+0x48/0x1f8

Therefore these errors arise. This doesn't happen for memory below 4GB border.

Any ideas how to prevent this behaviour?

[heinzhjb · Accepted solution]

Problem was the swiotlb system can't reserve bounce buffers bigger than 64kBytes.

I've used too big blocks to write to gadgetfs endpoint in userspace. Therefore the scatter/gather list contains too big slices of memory which couldn't be allocated when swiotbls is active(for memory buffers beyond 4GB limit).

Reducing the size written in one pass to USB endpoint solved this issue for me.