Dual Ethernet application with LWIP on RT1064

[kvleonhardt]

Hi Guys

We have developed a custom board with two ethernet ports for redundancy. The board holds two pcs of DP83640 phy's one on each ETH peripheral.

I have modified the DHCP example (the freRTOS one) to adapt our custom board. One version works perfectly on the ETH0 phy, and another version works on ETH2.

The problem is that the DHCP examples (and any other MCUxpresso examples) uses a LWIP-port-API, which busy loops while waiting for link. And the port does not allow to do that on both ports at the same time.

Therefore I have initially set ENET_ATONEGOTIATION_TIMEOUT to a relatively low number e.g. 10, allowing my two network tasks to handle one Ethernet port each. Furthermore I have modified the LWIP-port-API to deliver the status code from the phy-initialization back to the application code, to allow it to do it again, if failed. I have also enable some of the LWIP debugging messages which is routed through the semihost-debug-console.

My first attempt is to use static IP-address. But I have two odd issues. And I need a clue to fix it:

* The link flag on the phy is not set, unless I set a breakpoint in the task, and resume it again AFAP. It seems that the phy needs the MCU to take a break after the PHY is SW-reset.

* After the link is found OK, I can be lucky to see ARP and ICMP to work a single time, it seems that I only is able to receive the first 4 or 5 packages, and then I see no more interrupts from the ETH peripheral.

Have anyone seen something like this? 

Have anyone seen a dual port example with LWIP and FreeRTOS? Preferable with MCUxpresso IDE/SDK.

Thanks in advance.

[kvleonhardt]

I am one tiny step further in this case.

First of all, by reducing ENET_ATONEGOTIATION_TIMEOUT to 10, the phy-init does not allow the link to be established before the phy is reset once again. Therefore I have removed the link-detection from the phy initialization. Which in any case should be fine, as the network cable not always is plugged from the beginning.

The other problem with the receiver apparently is turned off is still a mystery. Compared to the DHCP-exampels the only difference I can find is the RDAR register (Receive descriptor active) is 0, but why? I have tried to set it 1 one, but it is stuck at 0. I think I am close, but need some advice to fix it. By the way, just after link is established, I still receive a few packages before the receiver goes down.

Thanks in advance,

Kasper

[kerryzhou]

Hi @kvleonhardt ，

   I have a question about your situation, if you don't use two Ethernet together, just test one by one, whether both the Ethernet works or not? Or just two works together have the issues?

Best Regards,

Kerry 

[kvleonhardt]

I actually test the Ethernet ports one by one. I have one task for each port, and only the one for ETH0 is actually started. The strange thing is that the behavior should be identically to the original examples, except for the changes I have described.

I have to verify the Clock and REF_CLK configuration and performance once again. It could explain why some packages is actually send/received now and then but the majority is vaporized.

One big difference from the eval kit design is, that our HW designer has chosen to ad a 25MHz oscillator on the phy's, instead of using the RT1064 for REF_CLK generation.

[kerryzhou]

Hi @kvleonhardt ,

  This is the 3rd part dual Ethernet circuit:

To your ETH1, do you have more board, and modify it, use the RT1064 REF_CLK , whether that method works or not?

 It should also related to the PHY configuration with different clock.

Best Regard,

Kerry

[kvleonhardt]

Hi Kerry

Thanks. Our circuit is a bit different, as we are using a 50MHz oscillator to driving the REF_CLK for both the MCU and the phy.

One thing I do not understand is that the design works fine with the dhcp-demo example from the SDK.

One thing I'm not totally sure about is how to define the direction on the REF_CLK on the MCU. I have inserted following in my code, but it seems to have no effect:

IOMUXC_GPR->GPR1 |= IOMUXC_GPR_GPR1_ENET1_CLK_SEL_MASK;
IOMUXC_GPR->GPR1 |= IOMUXC_GPR_GPR1_ENET2_CLK_SEL_MASK;

I normally prefer to use the config tool, but I believe that it is no help for the clock-configuration, when using an external oscillator. Is there any other registers I should be aware of?

And those lines is not in the dhcp-demo example. And why the dhcp-demo example works on my board is also a mystery (I have only remapped some pins in the MUX to get it work), as I have done nothing to reverse the REF_CLK, as the RT1064 eval-board uses the MCU for REF_CLK generation.

And to repeat my self, some packages is passing through the phy, as I have seen it answering on an ARP-request. And often I see a trace message (the record is 4 received packages) like this just after MCU-restart and network initialization:

ethernet_input: dest:ff:ff:ff:ff:ff:ff, src:2c:f4:32:cb:0a:f7, type:800 (Which is an Sony TV on the network broadcasting an UDP message)

[kerryzhou]

Hi @kvleonhardt ,

  So, to ENET2, DHCP works, and other lwip code can't work.   

  And, when the code is not work, do you test the PHY oscillator, whether it is oscillating or not?

  Except the DHCP code, which detail other code you can't work? What about lwip_ping?

Best Regards,

Kerry

[kvleonhardt]

All demonstration examples works.

By combining different projects wit different code-snippets, I have discovered that it is located in the basic MCU configuration/initialization.

Could it be the data-cache that gives me problems. As I wrote before, The first few packages is transmitted correctly.

The buffer init is done like this (Un-touched from examples):

err_t ethernetif0_init(struct netif *netif)
{
static struct ethernetif ethernetif_0;
AT_NONCACHEABLE_SECTION_ALIGN(static enet_rx_bd_struct_t rxBuffDescrip_0[ENET_RXBD_NUM], FSL_ENET_BUFF_ALIGNMENT);
AT_NONCACHEABLE_SECTION_ALIGN(static enet_tx_bd_struct_t txBuffDescrip_0[ENET_TXBD_NUM], FSL_ENET_BUFF_ALIGNMENT);
SDK_ALIGN(static rx_buffer_t rxDataBuff_0[ENET_RXBUFF_NUM], FSL_ENET_BUFF_ALIGNMENT);
SDK_ALIGN(static tx_buffer_t txDataBuff_0[ENET_TXBD_NUM], FSL_ENET_BUFF_ALIGNMENT);

I have tried to define a special area in the OC-RAM that is setup as non-cacheble/non-bufferable in the MPU (Just like the examples):

The ITC/DTC/OC memory banks is rearrange in SystemInitHook():

Snippet...
__asm volatile ("LDR R0, =0x400AC044");
__asm volatile ("LDR R1, =0xAAAEAAAA"); /* 1 bank ITC, 15 banks DTC */
Snippet...

But the linker does not place the eth-data-buffers in this area. Form MAP-file:

NonCacheable 0x200252c0 0xd8
NonCacheable 0x200252c0 0xd8 ./lwip/port/enet_ethernetif_kinetis.o

Question:

Should I do any concerns regarding the cache? I tried to remove the SCB_EnableDCache() with no luck.

AT_NONCACHEABLE_SECTION_ALIGN should force the linker to place it in NCACHE_REGION, right?

Should'n the tx/rxDataBuff_0/1 buffers also be "non-cached" in the ethernetif0_init().

I still hope that you can help. Close to giving up.

[kerryzhou · Accepted solution]

Hi @kvleonhardt ,

  Yes, it's related to the cache.

   I checked the SDK lwip code, the UDP put the nonCacheable in the DTCM:

About the cache, you also can read this document:

https://www.nxp.com/docs/en/application-note/AN12042.pdf

So, I think you can use the same as the SDK code.

If you want to put the noncache in the SDRAM, you can check the linker file, whether you really put the noncache section in the sdram, from your test result, your related data still in the DTCM.

Answer your questions :

Should I do any concerns regarding the cache? I tried to remove the SCB_EnableDCache() with no luck.

Answer: Yes, you need to concerns cache. Check the above AN.

AT_NONCACHEABLE_SECTION_ALIGN should force the linker to place it in NCACHE_REGION, right?

Answer: it is related to your settings, eg, in my above picture, it is related to the noncache area setting, you also can check the ld file to check the details.

Should'n the tx/rxDataBuff_0/1 buffers also be "non-cached" in the ethernetif0_init().

Answer: yes, need to put in the non-cached area.

Just refer to the SDK code.

Best Regards,

Kerry

[kvleonhardt]

Hi,

I think my main problem was that I was not able to disable cache for a part og the OCRAM. I moved to the DTCRAM, and the linker did as it should. I think there might be a problem with the linker, as it did not accept to directives to move the buffers to a section in the OCRAM.

And by the way I tried to move the buffers (in addition to the buffer descriptors) to the non cached section, but it failed. It resulted in an unaligned access hard fault. But it forks fine as the original code:

Original:

AT_NONCACHEABLE_SECTION_ALIGN(static enet_rx_bd_struct_t rxBuffDescrip_0[ENET_RXBD_NUM], FSL_ENET_BUFF_ALIGNMENT);
AT_NONCACHEABLE_SECTION_ALIGN(static enet_tx_bd_struct_t txBuffDescrip_0[ENET_TXBD_NUM], FSL_ENET_BUFF_ALIGNMENT);
SDK_ALIGN(static rx_buffer_t rxDataBuff_0[ENET_RXBUFF_NUM], FSL_ENET_BUFF_ALIGNMENT);
SDK_ALIGN(static tx_buffer_t txDataBuff_0[ENET_TXBD_NUM], FSL_ENET_BUFF_ALIGNMENT);

 Faulty attempt:

AT_NONCACHEABLE_SECTION_ALIGN(static enet_rx_bd_struct_t rxBuffDescrip_0[ENET_RXBD_NUM], FSL_ENET_BUFF_ALIGNMENT);
AT_NONCACHEABLE_SECTION_ALIGN(static enet_tx_bd_struct_t txBuffDescrip_0[ENET_TXBD_NUM], FSL_ENET_BUFF_ALIGNMENT);
AT_NONCACHEABLE_SECTION_ALIGN(static rx_buffer_t rxDataBuff_0[ENET_RXBUFF_NUM], FSL_ENET_BUFF_ALIGNMENT);
AT_NONCACHEABLE_SECTION_ALIGN(static tx_buffer_t txDataBuff_0[ENET_TXBD_NUM], FSL_ENET_BUFF_ALIGNMENT);

[kerryzhou]

Hi @kvleonhardt 

  We don't have the directly dual Ethernet code, but I find some 3rd part provide the dual Ethernet code.

  Now, I find the bm for lwip_ping_enet1_enet2, lwip_tcpecho_enet1_enet2,lwip_udpecho_enet1_enet2.

  So, I attach the related main.c code for your reference, you can check it on your side, whether it is useful to you.

  Wish it helps you!

Best Regards,

Kerry

[kvleonhardt]

Hi Kerry

Thank you for the examples, I am afraid that they have the same problem as the examples in the SDK, they require a link on ETH0 before ETH2 is serviced.

Any one out there that have some experience in dual Ethernet on RT1064 or simular?

Kasper

[phantomgz]

@kvleonhardt 

Try this, may be helpful

https://community.nxp.com/t5/MCUXpresso-SDK/fsl-enet-for-i-MX-RT106x-not-able-to-handle-ENET2/m-p/15...

Cai.