USB drivers with heap in flexspi-attached memory

[stefanct]

On our board we use an external PSRAM to increase the available memory. At the moment we store the heap there (and everything else in DTCM). This works fine for software.

However, I am currently implementing a USB host part (that should access mass storage). The USB drivers in the SDK use the heap (e.g., allocate new device instances via OSA_MemoryAllocate()). I have determined that this is is a problem because it leads to empty payload data in the set_address setup packets sent at enumeration time via USB_HostStandardSetAddress(). Roughly the same code works fine on an EVK with heap in DTCM.

My current theory is that while the CPU and software running on it can access the memory addresses on the PSRAM via AHB transactions just fine, the USB controller cannot. The reference manual (of the RT1020) only talks about alignment restrictions and "physical memory addresses" but I couldn't find any hints that this is restricted to FlexRAM-attached memory only or similar.

1. Can you confirm that the USB core cannot access memory attached to the FlexSPI controller?
2. Where is the related documentation?
3. Which memories can be used to store the transfer buffers other than DTCM?

Thanks.

[Gavin_Jia]

Hi @stefanct ,

Thanks for your questions!

I need to check with an internal expert on the issue you mentioned. And I'll sync with you here once there is progress. Thanks in advance!

Best regards,
Gavin

[Gavin_Jia]

Hi @stefanct ,

Thanks for your patience!

After speaking with an expert I got the following feedback:

This issue may related to cache, please try to disable the cache to see if it can work? The USB peripheral should not have any restrict to access the external memory.  

Can you help me test it?

Best regards,
Gavin

[stefanct]

Hi @Gavin_Jia,

are you referring to SCB_DisableDCache()? I run into hardware exceptions very soon when I try to leave the data cache disabled: once the UART driver accesses the peripheral for the first time after reset a hardfault is generated. I am not entirely sure why that is... maybe the cache hides unaligned accesses?

But I have investigated further and found that my problem is actually well understood and parts of the SDK even have some provisions to deal with it: USB_HOST_CONFIG_BUFFER_PROPERTY_CACHEABLE and friends.

However, it seems that in my configuration/system it's not available... since usb_host_hci.c contains:

#if defined __CORTEX_M && (__CORTEX_M == 7U)
#if (defined(USB_HOST_CONFIG_BUFFER_PROPERTY_CACHEABLE) && (USB_HOST_CONFIG_BUFFER_PROPERTY_CACHEABLE > 0U))
#warning USB_HOST_CONFIG_BUFFER_PROPERTY_CACHEABLE is not supported.
#endif
#endif

which is triggered once I set USB_HOST_CONFIG_BUFFER_PROPERTY_CACHEABLE to 1. I could probably deal with that myself but I would like to know what the rationale for this warning is. Are there any fundamental problems with the approach behind USB_HOST_CONFIG_BUFFER_PROPERTY_CACHEABLE or was it simply not completely implemented for the host side (yet)?

I have tried to verify if the cacheability of the PSRAM is the actual culprit. To that end I have tried to change the MPU settings to indicate that the address range of the PSRAM should not be cacheable similar to what is usually set for the "NonCacheable" section. However this did not seem to make any difference (no matter what I chose for the shareable bit, which I thought might also be relevant). Can you confirm that setting the respective address range in the MPU to uncacheable should elevate the presumed problem?

[stefanct]

I've continued debugging this and became convinced that the solution would be an un-cacheable memory region for the buffer. I mentioned that we are using an external SPI PSRAM for the heap. I wanted to use a bit of that PSRAM for this buffer.

To that end, I tried configuring the MPU to make the respective region un-cacheable. However, it seems to me that on the RT1021 the AHB accesses to the memory-mapped FLEXSPI interface are not guarded by the MPU at all. No matter what I configure for the address range of the PSRAM (0x6100'0000 - 0x6180'0000 in my case with a 16MiB flash on port A and 8 MiB PSRAM on port B) all accesses go through even with ARM_MPU_AP_NONE privileges. That's just a test I did eventually because the un-cacheabilty bit in the MPU configuration did nothing.

The only reference I could find is https://community.nxp.com/t5/i-MX-RT-Crossover-MCUs/FlexSPI-Cache-Issue/m-p/1310069 which clearly shows that it's working on the RT1011. Is there a difference in their MPU configurations (apart from the SRAM size)?

My configuration:

MPU:

• PRIVDEFENA but the behavior is the same of control = 0 and 1 (i.e., without and with privileged access.
• "Default" region of 4GiB and no access at MPU slot 0.
• R/O region of 16 MiB for external flash at 0x6000'0000 at slot 3.
• Device region of 1 GiB at 0x0 at slot 4.
• DTCM region of 256 KiB at 0x2000'0000 at slot 5 (override these addresses of slot 4). I have configured SRAM to use all space for DTCM.
• Device region of 4 MiB at 0x4000'0000 at 10.
• Various configurations for the PSRAM after the flash at slot 12. For example, via ARM_MPU_SetRegionEx(12, 0x6100'0000, ARM_MPU_RASR(0, ARM_MPU_AP_NONE, 1, 0, 0, 0, 0, ARM_MPU_REGION_SIZE_8MB)).

FlexSPI:

• config.enableDoze = false;
• config.ahbConfig.enableAHBPrefetch = false;
• config.ahbConfig.enableAHBBufferable = false;
• config.ahbConfig.enableAHBCachable = false;
• config.ahbConfig.enableReadAddressOpt = false;
• config.ahbConfig.enableClearAHBBufferOpt = false;
• config.rxSampleClock = kFLEXSPI_ReadSampleClkLoopbackInternally;
• config.enableHalfSpeedAccess = true;
• config.ahbConfig.buffer[0].bufferSize = 1024;
• config.ahbConfig.buffer[0].masterIndex = 0b0000;
• config.ahbConfig.buffer[0].enablePrefetch = 0;
  (other buffers should not matter)
• The LUT contains quad write/read transactions that are configured to be executed for AHB transfers (via AWRSeqIndex/ARDSeqIndex). And in generally that works fine.

To test the access I simply try to write single bytes to addresses starting at 0x6100'0000. With the configuration above the store instructions work (although they shouldn't because of AP=none) but no writes happen unless I disable the data cache or call SCB_CleanDCache_by_Addr(&buf[i], 1); where &buf[i] is the target address of the respective store instruction within the FlexSPI AHB address range.

So my main questions currently are:

• Why do AHB accesses to the the FlexSPI/PSRAM not trigger an exception although the address range is supposed to be protected by the MPU?
• Why are the writes to the FlexSPI/PSRAM cached although the address range is configured to be un-cacheable and all configurable buffering and caching of the FlexSPI controller have been disabled?

[stefanct]

At this point I am pretty sure that the MPU handling of the second FLEXSPI port is defunct.

With my recent tests I can show that even instructions can be fetched from the PSRAM without a proper MPU mapping or with an MPU mapping that has the NX bit set for the respective range all while there is the global denial block (to workaround speculative prefetches of Arm errata 1013783-B) in the MPU at slot 0 with ARM_MPU_RASR(1, ARM_MPU_AP_NONE, 0, 0, 0, 0, 0, ARM_MPU_REGION_SIZE_4GB).

AFAICT this is an errata-worthy hardware problem but I'd love to be proven wrong.