topic Re: LwIP sample program failure in LPC Microcontrollers

LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:46:55 GMT

Content originally posted in LPCWare by tamirmichael on Mon Jun 11 10:59:57 MST 2012
Hello,

I have downloaded the port of lwIP for LPC17xx and encountered a program failure after some time the controller (LPC1788) handes bus events
(I ran the TCP echo sample, but I suspect that any will do to demonstrate this):

/* all pbufs in a chain are referenced at least once */
LWIP_ASSERT("pbuf_free: p->ref > 0", p->ref > 0);

Which can be found at pbuf.c, Line 627 of version 1.4.0 (the function "pbuf_free"). The failure occurrs both with dynamic and staticaly allocated heap after at most several minutes of running, being pinged or not.
How can this happen? Did you encounter this too? What can be done about it?

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:46:56 GMT

Content originally posted in LPCWare by wellsk on Mon Jun 11 11:56:11 MST 2012

I ran flood ping tests with tcpecho for a while (easily more than 2 minutes). but didn't see this issue. But I have seen this trap get hit before in specific cases (all of which shouldn't be present except for possible one...). larsjep (another developer) recently discovered an issue with the way chained pbufs are de-allocated in the 18xx/43xx port that may be contributing to this issue. (It's an 18xx/43xx patch, but the same logic should apply to the 17xx port).

The patch is here:
http://www.lpcware.com/content/bugtrackerissue/lwiplpc-lpc18xx43xx-driver-pbuf-leak-during-transmit

I've noticed problems with the reference counters getting messed up when under load with an RTOS when SYS_LIGHTWEIGHT_PROT is not enabled (SYS_LIGHTWEIGHT_PROT = 0 and NO_SYS = 0). Are you currently using one of the examples without any changes? If you are using a FreeRTOS based example (NO_SYS = 0), can you verify that the SYS_LIGHTWEIGHT_PROT option is set to 1?

Do you have a stack trace available, or know if the issue occurred from the TX or RX handler side?

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:46:57 GMT

Content originally posted in LPCWare by tamirmichael on Mon Jun 11 12:19:28 MST 2012
Thanks a lot for your reply. I will try to patch my sample/project tomorrow morning!
Yes, indeed, I have been using the sample without my change (after not understanding why my official project fails...).
I do not have stack trace available yet, but I am quite sure the failure occurs while processing a received message.
I will report more details tomorrow.
Thanks again!

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:46:57 GMT

Content originally posted in LPCWare by tamirmichael on Tue Jun 12 00:20:41 MST 2012
Kevin,

I have merged this change into the my project and it seems to have been working fine for a while now (pinged for about 30 minutes plus).
Thanks again for sharing this patch.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:46:58 GMT

Content originally posted in LPCWare by tamirmichael on Tue Jun 12 02:17:48 MST 2012
A small update: I did see my program fail again after about an hour of operation. I cannot provide more details as my debugger was not connected.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:46:58 GMT

Content originally posted in LPCWare by wellsk on Tue Jun 12 09:00:33 MST 2012
Do what you can to identify the area of the problem and we'll try to repeat the issue here. Some type of case where the issue occurs would be very helpful (ie, the stack trace when the error occurs to help trace the failure path). The patch fixes a potential issue with TX pbufs only, but may or may not help at all. Are you using a standalone configuration or with an RTOS?

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:46:59 GMT

Content originally posted in LPCWare by tamirmichael on Tue Jun 12 23:50:01 MST 2012
In order to reproduce the problem, all you need to do is use the embedded artists LPC1788 evaluation board and program it with this sample program:

"nxpcommon\examples\lpc177x_8x\ea1788\ea1788_tcpecho_freertos"

and let it run being pinged or not. After some time, even with the patch posted above, it will hang at the same place indicated above.
I am going to start another test with a debugger - will provide a stack trace once failure occurs.
Thanks for your help.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:00 GMT

Content originally posted in LPCWare by tamirmichael on Wed Jun 13 00:05:20 MST 2012
Without the patch above, the failure seems to occur much faster, producing this stack trace:

pbuf_free (called by ip.c, line 433)
ip_input (called by etharp.c, line 1282)
ethernet_input (called by lpc17_emac.c, line 468)
lpc_enetif_input (called by lpc17_emac.c, line 787)
vPacketReceive_Task

this appiles to version 1.4.0 of lwIP.

I will now attempt to reproduce the failure with the patch.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:00 GMT

Content originally posted in LPCWare by tamirmichael on Wed Jun 13 01:06:40 MST 2012
I managed to fail the program with the patch - here is the stack trace:

pbuf_free (called by lpc17_emac.c, line 477)
lpc_enetif_input (called by lpc17_emac.c, line 787)
vPacketReceiveTask

It took me something like 45 minutes of continuous pinging (I stopped and resumed shortly before the failure) to cause this.
I hope this helps.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:01 GMT

Content originally posted in LPCWare by wellsk on Wed Jun 13 08:17:43 MST 2012
It looks like you are using FreeRTOS and the failure occurred on the input path. I'd keep the TX pbuf patch in, but it looks like I might have another problem with the example code itself using the wrong input path for an RTOS. (Sorry :()

When used with FreeRTOS, the 'ethernet_input' function should be replaced with the 'tcpip_input' function. This is correct for the LPC18xx/43xx examples, but it seems I got a little to comfortable with cut-and-paste.

In the example code, can you make the change indicated below and give it a try?

<code>
/* Add netif interface for lpc17xx_8x */
memset(lpc_netif, 0, sizeof(lpc_netif));
// if (!netif_add(&lpc_netif, &ipaddr, &netmask, &gw, NULL, lpc_enetif_init, ethernet_input))
if (!netif_add(&lpc_netif, &ipaddr, &netmask, &gw, NULL, lpc_enetif_init, tcpip_input))
LWIP_ASSERT("Net interface failed to initialize\r\n", 0);
</code>

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:01 GMT

Content originally posted in LPCWare by wellsk on Wed Jun 13 09:11:51 MST 2012
I've entered a bug tracker issue for this case.
http://www.lpcware.com/content/bugtrackerissue/lwip-17xx-freertos-examples-use-wrong-input-function

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:02 GMT

Content originally posted in LPCWare by tamirmichael on Thu Jun 14 00:46:05 MST 2012
I will try this fix ASAP but I must deal with some bush fires first...
Thanks for the efforts, I will report the results once I have something.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:03 GMT

Content originally posted in LPCWare by SeleneSW on Tue Jun 19 02:54:43 MST 2012
I have the same problem with the "ea1788_http_freertos" demo.
I tried with the patches described above but with no success.

Any news about this thread?

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:03 GMT

Content originally posted in LPCWare by wellsk on Wed Jun 20 14:35:48 MST 2012
I have uploaded the LPC17xx and LCP43xx ports with larsjep's updates and several other fixes for both the 17xx and 43xx platforms.
You can pull the changes from the GIT repo at: http://sw.lpcware.com/?p=lwip_lpc.git&a=summary

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:04 GMT

Content originally posted in LPCWare by tamirmichael on Thu Jun 21 03:58:50 MST 2012
Thanks for the effort. I have downloaded the corrected program and just started what I hope will be a very long test. I will report back.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:04 GMT

Content originally posted in LPCWare by tamirmichael on Thu Jun 21 04:58:13 MST 2012
Test failed (ping reply no longer generated). I did not have a debugger connected to the target.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:05 GMT

Content originally posted in LPCWare by tamirmichael on Thu Jun 21 05:29:32 MST 2012
I am conducting an additional test to make sure what I have seen is indeed a legitimate failure.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:05 GMT

Content originally posted in LPCWare by tamirmichael on Thu Jun 21 06:10:44 MST 2012
Test failed again. I did not have a debugger connected to the target but it is easy to reproduce - just let any of the test programs run pings (or not) just like above.

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:06 GMT

Content originally posted in LPCWare by wellsk on Thu Jun 21 11:19:32 MST 2012
>Test failed again. I did not have a debugger connected
I wasn't able to repeat the failure here with the current FLASH variant of the ea1788_tcpecho_freertos project.
I opened multiple (3+) flood ping sessions to the EA1788 board and ran them for an hour - the board never lost ping response, although it did occasionally lose some packets or get increased latency times due to loading. (This is with only 3 TX and RX descriptors being used for the driver, so it's to be expected).

The FLASH example is very memory limited with the following configuration:
From lwipopts.h...
<code>
#define PBUF_POOL_SIZE 6
#define MEM_SIZE(12*1024)
</code>
From lpc_emac_config.h...
<code>
#define LPC_NUM_BUFF_RXDESCS 3
#define LPC_NUM_BUFF_TXDESCS 3
</code>

It -is- very possible to get LWIP into a state where it's internally 'out of memory' and no more pbufs can be allocated for the receive descriptor queue. If this happens, the code will continue to run, but the MAC controller will accept no more packets as no more descriptors are queued due to no buffers being allocatable! Packet fragmentation will definitely do this (ping -f -s 5000 <IP address>). When LWIP allocates space for copied packet fragments, it uses the same memory for normal buffers - and the pbuf allocation will eventually fail. Only adding more memory will fix this.

The FLASH variant example uses the internal RAM only. Ethernet buffers and descriptors can only be located in peripheral RAM on the 17xx, which is only 32K. This isn't a lot for LWIP. A DRAM variant is also available that uses DRAM for buffer/pbuf/descriptor allocation from a huge memory pool.
Here are the DRAM variant settings, this also boots from FLASH...
<code>
#define PBUF_POOL_SIZE 256
#define MEM_SIZE(4096*1024)
#define LPC_NUM_BUFF_RXDESCS 128
#define LPC_NUM_BUFF_TXDESCS 128
</code>

I've attached both versions (FLASH/IRAM and FLASH/DRAM) of the example here for the EA1788 board. I've also attached a debug version of the DRAM version here that should output messages to the UART port similar to the following. If LWIP fails, you should see an error message you can re-post here.
<code>
lpc_low_level_input: Packet received: a0027110, size 102 (index=109)
pbuf_alloc(length=1536)
pbuf_alloc(length=1536) == a0026af4
lpc_rxqueue_pbuf: pbuf packet queued: a0026af4 (free desc=0)
pbuf_header: old a0027120 new a002712e (-14)
pbuf_free(a0027110)
pbuf_free: deallocating a0027110
lpc_low_level_input: Packet received: a002772c, size 102 (index=110)
pbuf_alloc(length=1536)
pbuf_alloc(length=1536) == a0027110
lpc_rxqueue_pbuf: pbuf packet queued: a0027110 (free desc=0)
pbuf_header: old a002773c new a002774a (-14)
pbuf_free(a002772c)
pbuf_free: deallocating a002772c
lpc_low_level_input: Packet received: a0027d48, size 102 (index=111)
</code>

Re: LwIP sample program failure

lpcware — Wed, 15 Jun 2016 17:47:07 GMT

Content originally posted in LPCWare by tamirmichael on Fri Jun 22 01:02:15 MST 2012
Thanks. I have downloaded "dram_uart_example.hex". I will start a test right away.