FreeRTOS LwIP SDK v2.0.0 tpcipecho demo crash

[wilcovm]

Dear all,

I'm using the K64 Tower board and downloaded the SDK2.0.0. Then I started to run the tcpipecho server demo which uses FreeRTOS and LwIP in KDS. It runs fine for some time (something like a few minutes). I can connect/disconnect to it and echo some text. But after that it crashes (HardFault) (most of the time LwIP's memp_alloc() cannot allocate memory).

I tried a lot to figure out what could cause it. Enabled memory check, increased pbuf size and chain, tried C malloc, increased FreeRTOS heap moved it to SRAM-H .... But nothing helps.

Then I started to download the SDK again but for IAR and downloaded the IAR demo version. Still the same, after a some time it crashes.

- Does someone has the same experience and knows what the problem is?

- Is there a very stable working example for FreeRTOS in combination with LwIP (preferable a K64 client demo)?

- Does someone has tips to let it work?

[paulvanoppen]

Note that PBUF_POOL_BUFSIZE depends on TCP_MSS.

Increasing TCP Maximum segment should be considered rather than just increasing PBUF_POOL_BUFSIZE.

From opt.h:

/**

* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. The default is

* designed to accommodate single full size TCP frame in one pbuf, including

* TCP_MSS, IP header, and link header.

*/

#if !defined PBUF_POOL_BUFSIZE || defined __DOXYGEN__

#define PBUF_POOL_BUFSIZE LWIP_MEM_ALIGN_SIZE(TCP_MSS+40+PBUF_LINK_ENCAPSULATION_HLEN+PBUF_LINK_HLEN)

#endif

[gewesp]

We're still observing crashes of the vanilla lwip_freertos_tcpecho from LPCXpresso 8.2.2.

We traced one problem down to wrong configuration of the DMA buffers.

But there's still a reproducible crash when we execute sudo nmap -sn 192.168.1.0/24 on the subnet containing the embedded device (you may have to adapt the network address to your setup).

I'm interested to hear from anybody who tried any of the fixes suggested in this thread:  Is your example robust against this nmap command?

Best regards

--Gerhard Wesp

KISS Technologies

[danielchen]

Hi Wilco:

I have tried this demo with TWR- K64 for about 20 minutes, and not found this issue. Did you modify the source code?

SDK_2.0_TWR-K64F120M\boards\twrk64f120m\demo_apps\lwip\lwip_tcpecho\freertos\iar

Could you  help me how to reproduce your issue in my side?

Regards

Daniel

[manfredschnell]

Hello Daniel and Wilco,

I have the same problem. I use K64 with KSDK2.0 + LWIP and IAR Workbench.

I use the internal buffer in "pbuf.c" with pbuf_alloc().

The size of PBUF is set to 256, so I get the error quick.

#define PBUF_POOL_BUFSIZE       256

I investigated the problem.

--> the memory corruption happens every time when a incomming packet is greater than PBUF_POOL_BUFSIZE.

The bug is in KSDK_2.0\middleware\lwip_1.4.1\port\ethernetif.c.

In function ethernetif_Input() the buffer is allocated from PBUF_POOL with length. Every time the length is greater than one single buffer size, you get back a chain of buffers. --> this memory area is not continuous!!!!

Then ENET_ReadFrame() is called with the first buffer and the full length..... there happens the shit.

--> data is copied across the buffer boundary. --> you know what that means....

Can someone please have a look on this things... and confirm, that I'm right.

For a quick workaround I changed the pool size to maximum incomming packet size. So the allocated buffer memory is always continous.

#define PBUF_POOL_BUFSIZE       1518U /*=ENET_FRAME_MAX_FRAMELEN*/

This works. But much memory is wasted and it ist slow.

The function ethernetif_Input() should be reviewed and work with chained buffers.

Best regards

Manfred

[danielchen]

Hi Manfred:

Thanks for your effort  to locate the root cause.  By increasing PBUF_POOL_BUFSIZE to contain the whole Ethernet frame is the fix.  This issue is present for KSDK 2.0 Rel1(or older). Not for current. For K64, Rel1 is the latest.

If you need good performance and not waste much memory, I think you can keep the PBUF_POOL_BUFSIZE as original, and  discard the long-size incoming packet in ethernet driver (will lose long size packet), it depends on your requirement.

Regards

Daniel

[manfredschnell]

Hi Daniel,

thank you for confirming the fix.

It's not an option to lose long packets.

In the meantime I revised the ethernetif_Input() function.

Now it works with chained buffers, when necessary. And updates the LINK_STATS of lwip.

Best regards.

Manfred

[wesleyhunter]

I have the exact same problem. I'm setup to receive udp broadcast messages and some of the messages are to large causing this same memory issue. I temporarily fixed it increasing the PBUF_POOL_BUFSIZE but I am curious what changes you made to ethernetif_Input() Manfred, you ok to share your changes? I'm also curious when we will see the fix for download available for K64 Daniel?

[manfredschnell]

Hi Wesley,

I think I can share the revised ethernetif_input(). But I do this without any warranty.

This job should be done by NXP...

Take care...  this function sometime needs a big local buffer with size  ENET_FRAME_MAX_FRAMELEN !!!

Best regards

Manfred

void ethernetif_input(enet_handle_t *handle, void *param)
{
    struct eth_hdr *ethhdr;
    struct pbuf *packetTempBuffer;
    status_t status;
    uint32_t length = 0;
    struct netif *netif = param;
    static enet_data_error_stats_t eErrStatic; // to get the error statically
    // by MS enet_data_error_stats_t eErrStatic;
    uint32_t localLen;
    uint32_t workedBytes;
    uint32_t lengthToCopy;
    static uint16_t type;

    /* Check input parameter */
    if (netif == NULL)
    {
        LWIP_ASSERT("Input param is NULL", 0);
        /* Should not reach this statement if error occurs */
        return;
    }
    /* Read all data from ring buffer and send to uper layer */
    do
    {
        /* Get the Frame size */
        status = ENET_GetRxFrameSize(handle, &length);

        /* Call ENET_ReadFrame when there is a received frame. */
        if (length != 0)
        {
            uint8_t *receivedDataBuffer;
            struct pbuf *packetBuffer;

            packetBuffer = pbuf_alloc(PBUF_RAW, length, PBUF_POOL);

            /* Received valid frame. Deliver the rx buffer with the size equal to length. */
            if ((packetBuffer == NULL) || (packetBuffer->payload == NULL))
            {
                pbuf_free(packetBuffer);
                packetBuffer = NULL;
                LWIP_ASSERT("Fail to allocate new memory space", 0);
                LINK_STATS_INC(link.memerr);
                /* Should not reach this statement if error occurs */
                return;
            }
            if ( length <= PBUF_POOL_BUFSIZE )   {
              // target buffer
              receivedDataBuffer = packetBuffer->payload;
              packetBuffer->len = (u16_t)length;
              // read direct to one buffer
              ENET_ReadFrame(ENET, handle, receivedDataBuffer, packetBuffer->len);
              LINK_STATS_INC(link.recv);
            }
            else {
              // read to one temp buffer and copy then to allocated memory
              unsigned char ucBuffer[ENET_FRAME_MAX_FRAMELEN];

              if ( length > ENET_FRAME_MAX_FRAMELEN )  {
                  pbuf_free(packetBuffer);
                  packetBuffer = NULL;
                  LWIP_ASSERT("Fail length > ENET_FRAME_MAX_FRAMELEN", 0);
                  LINK_STATS_INC(link.lenerr);
                  /* Should not reach this statement if error occurs */
                  return;
              }

              // read local
              ENET_ReadFrame(ENET, handle, ucBuffer, length);
              LINK_STATS_INC(link.recv);

              // take the total length
              localLen = length; 
              workedBytes = 0;

              packetTempBuffer = packetBuffer;
              while ( (localLen) && (packetTempBuffer != NULL) )  {

                lengthToCopy = localLen;
                if ( lengthToCopy > packetTempBuffer->len ) {
                  lengthToCopy = packetTempBuffer->len;
                }

                // copy one part
                memcpy(packetTempBuffer->payload, &ucBuffer[workedBytes], lengthToCopy);

                // next buffer in chain
                packetTempBuffer = packetTempBuffer->next;

                localLen -= lengthToCopy;
                workedBytes += lengthToCopy; 
              };

            }
            /* points to packet payload, which starts with an Ethernet header */
            ethhdr = packetBuffer->payload;
            type = htons(ethhdr->type);
            switch (type)
            {
                /* IP or ARP packet? */
                case ETHTYPE_IP:
                case ETHTYPE_ARP:
#if PPPOE_SUPPORT
                /* PPPoE packet? */
                case ETHTYPE_PPPOEDISC:
                case ETHTYPE_PPPOE:
#endif /* PPPOE_SUPPORT */
                    /* full packet send to tcpip_thread to process */
                    if (netif->input(packetBuffer, netif) != ERR_OK)
                    {
                        LWIP_DEBUGF(NETIF_DEBUG, ("ethernetif_input: IP input error\n"));
                        pbuf_free(packetBuffer);
                        packetBuffer = NULL;
                    }
                    break;
                default:
                    pbuf_free(packetBuffer);
                    packetBuffer = NULL;
                    LINK_STATS_INC(link.drop);     
                    break;
            }
        }
        else
        {
            /* Update the received buffer when error happened. */
            if ( (status != kStatus_Success) && (status != kStatus_ENET_RxFrameEmpty) )
            {
                /* Get the error information of the received g_frame. */
                ENET_GetRxErrBeforeReadFrame(handle, &eErrStatic);
                /* update the receive buffer. */
                ENET_ReadFrame(ENET, handle, NULL, 0);
                LINK_STATS_INC(link.err);     
            }
        }
    } while (kStatus_ENET_RxFrameEmpty != status);
}

[rojer]

this solution requires 1.5K of stack memory. as suggested in my report of the same issue, the easiest thing to do is to allocate a PBUF_RAM buffer if the packet doesn't fit into the allocated PBUF_POOL buffer.

increasing PBUF_POOL_BUFSIZE to 1.5K is actually a pretty inefficient solution and will waste a lot of memory - lwip is written to deal with pbuf chains, and this allows for more efficient memory management.

the real solution is to make ENET_ReadFrame do partial reads.

[manfredschnell]

Hi Deomid,

my solution needs 1.5K stack, that's true. I prefer a static allocated buffer, instead of a dynamic allocated memory from heap. That may lead to memory defragmentation dependent on network load.

increasing PBUF_POOL_BUFSIZE to 1.5K  --> is just a cheap workaround with many disatvantages.

"the real solution is to make ENET_ReadFrame do partial reads." --> yes ENET_ReadFrame should read direct to chained buffers. --> this API should be revised again by NXP.

I'm curious, when we will get a reworked KSDK2.x.... There is enough to improve for NXP.....

Best regards

Manfred

[rojer]

ENET_ReadFrame should read direct to chained buffers.

ENET_ReadFrame is not lwip-specific and making it aware of pbuf chains would be wrong. however, there is a simpler solution: right now ENET_ReadFrame it assumes that length is enough to accommodate entire frame. instead, it just needs to read as much of the frame as the passed length parameter allows and leave the rest in the buffer(s). ethernetif_input should then loop over the allocate pbuf chain and call ENET_ReadFrame repeatedly until the frame is fully copied.

[daweiyou]

Hi:

Yes, I think this API should be revised again.

[wilcovm]

Hi Daniel,

Thanks for your reaction!

Yes, that's the demo I use. I didn't modify the code (except for a different IP address). But we have a heavily used network at work. E.g we have one device on the network (network with more than 50 devices) which sends quite often a broadcast with one package length of 682 bytes or more (seen with Wireshark). I'm not sending anything to the demo myself, I only connect to it.

At my home (very small hardly used network) I don't have this problem of the crash. There it works fine for more than 30 min.

That's making me scared because the final device should work very reliable on every network.

Maybe you can reproduce by sending a broadcast which has a package with a very large package length (say >682 bytes).

Regards,

Wilco