RTCS does not work when running in External Memory

[pmt]

I am running the latest MQX (4.1.1) on a K60 Tower BSP derivative platform.

If I start RTCS before adding an external memory pool (with _mem_extend() ) everything is fine.

If I start RTCS after adding an external memory pool I can't ping or do a socket connection to the device, though DHCP client is able to get an address.

My external memory is fine.  It's is used for many other things.  In either case no MQX errors are reported, memory, stacks, tasks, etc.

Could this be some cache coherency issues (DMA related) in RTCS?  External memory is in a cache-able region.

Any ideas? 

PMT

[pmt]

Carlos,

I believe I found the issue, and it is a bug in the MQX bsp. 

In _bsp_pre_init() there is a section of code that intends to create a non-cachable SRAM memory pool for use by the Ethernet driver among others.  It does this by first _statically_ testing if the KERNEL is placed in internal or external SRAM, making the assumption that this is where the kernel heap is also placed.  If internal it allocates from the standard kernel memory pool.  If external it creates a new pool in non-cached memory and allocates from it. 

However this function fails to handle the case where the kernel memory pool is extended at run-time via _mem_extend() with cached memory.  This is why if the memory is extended before initializing Ethernet it fails. 

The logic is too convoluted.  It tries to make assumptions about where the memory pool is going to be rather than just actually testing where the memory pool is at the time the code is executed.

The logic needs to change to:

  - Test where the memory pool actually is (at run time)

  - If internal allocate from it

  - If external allocate from an internal memory pool.

A better method might be just to allocate the Ethernet buffers unconditionally and statically with the linker section to a non-cached memory region.  This would eliminate a lot of convoluted logic and obscure #define's such as BSPCFG_HAS_SRAM_POOL and BSPCFG_ENET_SRAM_BUF.

PMT

[Carlos_Musich]

Hi PMT,

Thank you very much for your inputs, we really appreciate it.

I have already reported this to the MQX team.

Best regards,

Carlos

[Carlos_Musich]

Hi PMT,

What address are you using for the external memory? Have you tried non cacheable address range such as 0xA0000000?

Carlos

[pmt]

I moved the external memory to non-cachable region (0xA0000000) and everything works like a champ.  However, this is not a good solution going forward.  That significantly sacrifices system performance. 

Seems like the Ethernet buffers need to handle working from internal or external memory (with cache line flushes), or that they should be allocated statically so they can be tagged by the linker for internal memory (much like the USB buffers are allocated).  I know static allocation is not a good general solution since some systems will undoubtedly have multiple Ethernet devices, however it is a safe bet given that this Ethernet controller is captive to the K60.

PMT

[Carlos_Musich]

Hi PMT,

I haver never made this test, but if you want to use the 0x60000000 cacheable area I would try flushing the cache before and after calling ipcfg_init_device().

Carlos

[pmt]

Carlos,

I don't think this will do it.  The cache coherency is an issue not with the ARM core, but with DMA bypassing the cache.  So in this case I suspect it is the Ethernet peripheral DMA, which will be a problem on every Ethernet transfer, not just initialization.  To solve the issue (assuming that it is the issue) MQX would have to modify the Ethernet driver buffers to use statically allocated internal memory.

See the MQX USB driver example.  There is a special section defined in the MQX provided linker file (USB_BDT) to allocate static non-cachable memory.  I think the Ethernet driver needs the same type of treatment.

Can you confirm that the Ethernet driver was tested with cache-able memory regions?

PMT

[pmt]

Carlos,

In MACNET_initialize there is a bit of code:

   // TODO remake to using alloc_align fn

#if BSPCFG_HAS_SRAM_POOL && BSPCFG_ENET_SRAM_BUF

   bd_ptr = (VENET_BD_STRUCT_PTR)_mem_alloc_system_zero_from(_BSP_sram_pool, (sizeof(ENET_BD_STRUCT)*(macnet_context_ptr->NumRxBDs+macnet_context_ptr->NumTxBDs))+MACNET_BD_ALIGNMENT);

#else

   bd_ptr = (VENET_BD_STRUCT_PTR)_mem_alloc_system_zero_uncached((sizeof(ENET_BD_STRUCT)*(macnet_context_ptr->NumRxBDs+macnet_context_ptr->NumTxBDs))+MACNET_BD_ALIGNMENT);

#endif

that is supposed to allocate noncached memory.  This is not working for me, and probably why the Ethernet is failing.  Apparently this was not implemented in the K60 Tower BSP (since it doesn't have external memory by default), but does exist in a K70 version.  I'll try to get it this working the way it should and I think it will fix my issue.

PMT

[pmt]

I am in the cache-able area: 0x60000000.  I will give the non-cache-able area a try for a data point, but ultimately I don't want to leave it that way.

I have narrowed it down.  If I call ipcfg_init_device() after extended the memory pool then ICMP and TCP don't work.  I think specifically the Ethernet buffer allocated under ENET_initialize_ex() are probably the culprit.  I think this a DMA cache coherency issue with the Ethernet Buffers.  Once these are allocated in external memory seems things stop working as they should.  Don't know how the DHCP client is managing to work however.

PMT