lwIP hardfault

[lpcware]

Content originally posted in LPCWare by tjoAG on Mon Sep 10 04:49:45 MST 2012
Dear All

I sometimes have some hard fault in the lwIP stack.
I have tried debugging it and have seen this

The hard-fault error happen in the lpc_low_level_input function line 414

<code>
/* Free pbuf from desriptor */
lpc_enetif->rxb[idx] = NULL; <-- Hard fault on this!!!
lpc_enetif->rx_free_descs++;
</code>

This only happens, if the previous call to lpc_rx_queue failed with a pbuf_alloc error

In the case the pbuf_alloc fails, no pbuf is queued. Thereby the
lpc_enetif->prxd[idx].control, lpc_enetif->rxb[idx] etc isn't set.

when the lpc_low_level_input is called again the lpc_enetif is not ready for further processing. The Ethernet frame was received but never queued and cleared?
Could that be the cause?

Should the 'lpc_low_level_input' not react to the return value of lpc_rx_queue if the frame was queued or not?

Why does pbuf_alloc fail? Does I receive to many packets to fast? It is on the local network.

I'm running in FreeRTOS

used buffers:

#define LPC_NUM_BUFF_RXDESCS 3
#define LPC_NUM_BUFF_TXDESCS 3
#define MEM_SIZE(12*1024)

Debug:

lpc_low_level_input: Packet received: 20000630, size 86 (index=1)
lpc_rxqueue_pbuf: pbuf packet queued: 20001260 (free desc=0)
lpc_low_level_input: Packet received: 20000c48, size 64 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20000630 (free desc=0)
......
......
lpc_rxqueue_pbuf: pbuf packet queued: 200019a8 (free desc=0)
lpc_low_level_input: Packet received: 20001390, size 64 (index=1)
lpc_rxqueue_pbuf: pbuf packet queued: 20001fc0 (free desc=0)
lpc_low_level_input: Packet received: 20000d78, size 64 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20001390 (free desc=0)
lpc_low_level_input: Packet received: 200019a8, size 64 (index=0)
lpc_rxqueue_pbuf: pbuf packet queued: 20000d78 (free desc=0)
ow_level_output: pbuf packet(200008aa) sent, chain#=0, size = 68 (index=0)
lpc_low_level_input: Packet received: 200019a8, size 72 (index=1)
lpc_rxqueue_pbuf: pbuf packet queued: 20000d78 (free lpc_low_level_input: Packet received: 200019a8, size 72 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20000d78 (free desc=0)
lpc_low_level_output: pbuf packet(20000a8a) sent, ize 72 (index=0)
lpc_rxqueue_pbuf: pbuf packet queued: 20000d78 (free desc=0)
lpc_low_level_output: pbuf packet(20000c0a) sent, chain#=0, size = 68 (index=0)
lpc_low_levw_level_input: Packet received: 20001fc0, size 72 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20001450 (free desc=0)
lpc_low_level_input: Packet received: 20001ac8, size 64 (index=0)
lpc_rxqueue_pbuf: pbuf packet queued: 20001450 (free desc=0)
lpc_low_level_input: Packet received: 20002140, size 64 (index=1)
lpc_rxqueue_pbuf: pbuf packet queued: 20001ac8 (free desc=0)
lpc_low_level_input: Packet received: 20002758, size 90 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20002140 (free desc=0)
lpc_low_level_input: Packet received: 20001450, size 64 (index=0)
lpc_rxqueue_pbuf: pbuf packet queued: 20002758 (free desc=0)
lpc_low_level_input: Packet received: 20001ac8, size 86 (index=1)
lpc_rxqueue_pbuf: pbuf packet queued: 20001450 (free desc=0)
lpc_low_level_input: Packet received: 20002140, size 64 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20001ac8 (free desc=0)
lpc_low_level_input: Packet received: 20002758, size 64 (index=0)
lpc_rxqueue_pbuf: pbuf packet queued: 20002140 (free desc=0)
lpc_low_level_input: Packet received: 20001450, size 256 (index=1)
lpc_rxqueue_pbuf: pbuf packet queued: 20002758 (free desc=0)
lpc_low_level_input: Packet received: 20001ac8, size 64 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20001450 (free desc=0)
lpc_low_level_input: Packet received: 20002140, size 64 (index=0)
lpc_rxqueue_pbuf: pbuf packet queued: 20001ac8 (free desc=0)
lpc_low_level_input: Packet received: 20002758, size 86 (index=1)
lpc_rxqueue_pbuf: pbuf packet queued: 20002140 (free desc=0)
lpc_low_level_input: Packet received: 20001450, size 64 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20002758 (free desc=0)
lpc_low_level_input: Packet received: 20001ac8, size 86 (index=0)
lpc_rxqueue_pbuf: pbuf packet queued: 20001450 (free desc=0)
lpc_low_level_input: Packet received: 20002140, size 72 (index=1)
lpc_rxqueue_pbuf: pbuf packet queued: 20001ac8 (free desc=0)
lpc_low_level_output: pbuf received: 20002140, size 72 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20001ac8 (free desc=0)
lpc_low_level_output: pbuf packet(2000102a) sent, chain#=0, size = 6 pbuf packet queued: 20001ac8 (free desc=0)
lpc_low_level_output: pbuf packet(200011aa) sent, chain#=0, size = 68 (index=1)
lpc_low_level_input: Packet received: 200lpc_low_level_output: pbuf packet(2000132a) sent, chain#=0, size = 68 (index=2)
lpc_low_level_input: Packet received: 20002758, size 72 (index=2)
lpc_rxqueue_pbuf: pbuf packet queued: 20002140 (free desc=0)
lpc_low_level_output: pbuf packet(2000138a) sent, chain#=0, size = 68 (index=0)
lpc_low_level_input: Packet received: 20001450, size 72 (index=0)
lpc_rxqueue_pbuf: pbuf packet queued: 20002758 (free desc=0)
lpc_low_level_output: pbuf packet(20001462) sent, chain#=0, size = 68 (index=1)
lpc_low_level_input: Packet received: 20001ac8, size 72 (index=1)
pc_rx_queue: could not allocate RX pbuf (free desc=1)
END OF EXECUTION: Hard fault

Next time hard fault error on:
lpc_enetif_input -> lpc_low_level_input -> lpc_enetif->rxb[idx] = NULL; (Line 414)

Thomas

[lpcware]

Content originally posted in LPCWare by wellsk on Tue Sep 11 08:51:00 MST 2012
>The task stack size set by the lwIP is configured to 1024 as the default size and a total of 3 task are created. Is the stacks size of 1024 a minimum or can it be reduced?
The RX and TX pbuf handling threads in the driver can use a smaller stack, maybe 512 bytes.
The TCPIP thread, part of LWIP, needs more than 512 bytes but less than 1024 bytes (at least during my testing).
You might enable the stack checking feature of FreeRTOS and run the code for a while to see if your pushing your stack size.

>I have plenty SDRAM in my system, so maybe it can be moved to using external RAM?
You can place pbufs, Ethernet descriptors, and buffers in SDRAM and have (probably) more Ethernet packet memory than you'll ever need.
The important data regions to place in DRAM are as follows: (scatter file)
<code>
  ; DMA buffers can be located in DRAM
  PIRAM 0xA0000000 0x10000000 {
    lpc17_emac.o (+RW, +ZI)
mem.o (+RW, +ZI)
memp.o (+RW, +ZI)
  }
</code>

Descriptors are located in the lpc17_emac.o file. pbufs and other allocations are in mem.o and memp.o

If you do place these in DRAM, you can increase your available memory as follows:
<code>
/* pbuf buffers in pool. In zero-copy mode, these buffers are
   located in peripheral RAM. In copied mode, they are located in
   internal IRAM */
#define PBUF_POOL_SIZE                  256

/**
* MEM_SIZE: the size of the heap memory. If the application will send
* a lot of data that needs to be copied, this should be set high.
*/
#define MEM_SIZE                                                (4096*1024)

/** \brief  Defines the number of descriptors used for RX. This
*          must be a minimum value of 2.
*/
#define LPC_NUM_BUFF_RXDESCS 128

/** \brief  Defines the number of descriptors used for TX. Must
*          be a minimum value of 2.
*/
#define LPC_NUM_BUFF_TXDESCS 128
</code>

LWIP has it's own memory allocation (heap), while FreeRTOS uses the C runtime environment's heap when using heap3.c. The amount of memory uses for LWIP is set by MEM_SIZE, while the amount of memory used for FreeRTOS is set in the startup code.
<code>
; <h> Heap Configuration
;   <o>  Heap Size (in Bytes) <0x0-0xFFFFFFFF:8>
; </h>

                                IF      :DEF:USE_FREERTOS
Heap_Size       EQU     0x00008000
                                ELSE
Heap_Size       EQU     0x00000000
                                ENDIF
</code>

[lpcware]

Content originally posted in LPCWare by tjoAG on Tue Sep 11 01:02:59 MST 2012
Hi

Well, increasing the Heap_Size from 0x0000 8000 to 0x0000 a000 does the job also.

Setting it to zero is probably not a good idea when using C lib malloc function. :-)

[lpcware]

Content originally posted in LPCWare by tjoAG on Tue Sep 11 00:18:46 MST 2012
Hi,

I do have the suggested change in the code. So I guess fairly update.

You where right on the last part. The lpc_tx_reclaim() function never did get called.

The last thread creation for the txclean_thread task did fail due to malloc fail. I have used all the available heap for other task, so the last creation for the txclean_thread did fail.

Maybe the lpc_enetif_init() function should check the return value and then assert if task creation did fail?

I'm using the heap_3.c for FreeRTOS heap management. Just in the lwIP FreeRTOS sample.

If I set the Stack_Size and Heap_Size in the startup.s file to zero I have enough memory for task creation.?

But I still need to add some task to my code before it is finished. So I could end up not having enough memory again.

The task stack size set by the lwIP is configured to 1024 as the default size and a total of 3 task are created. Is the stacks size of 1024 a minimum or can it be reduced?

Do you know where the total heap size i set, when using the C library malloc function? (Not in the startup file it seems)

I also could use the heap_2.c management for FreeRTOS where the total heap size can be set.

I have plenty SDRAM in my system, so maybe it can be moved to using external RAM?

Thomas

[lpcware]

Content originally posted in LPCWare by wellsk on Mon Sep 10 09:29:00 MST 2012
Hi,

Which version of LWIP are you using? Currently, the latest released version of lwip_lpc is v1.10, *but* there are a few 'critical' updates that have been made after that release.
At least one of these updates fixed a pbuf 'NULL' issue for the 17xx driver. This sounds like it might be that issue.

The specific 17xx change is here:
http://sw.lpcware.com/?p=lwip_lpc.git&a=blobdiff&h=c46e55d351ff4057d6d83e22ccd359e55b31becf&hp=9fb72...

You can pull the latest LWIP code base from the GIT repo here: http://sw.lpcware.com/?p=lwip_lpc.git&a=summary

>pc_rx_queue: could not allocate RX pbuf (free desc=1)
This may not indicate a real failure condition, it only indicates that the pbuf couldn't be allocated due to low memory availability in LWIP. As long as pbufs and descriptors are queued for the RX side, you will receive and process packets. When memory becomes available (later), the RX pbuf(s) will attempt to be reallocated and queued.

>when the lpc_low_level_input is called again the lpc_enetif is not ready for further processing. The Ethernet frame was received but never queued and cleared?
>Could that be the cause?
lpc_low_level_input should be called (in FreeRTOS) whenever a packet is ready for processing. Extra calls to it are ok and won't do anything (it returns NULL indicating no buffer has been received).
"The Ethernet frame was received but never queued and cleared?" <-- I'm not sure what you mean by this. When a packet is received in a pbuf, the descriptor associated with the pbuf is used again by allocating a new pbuf to it. If a pbuf can't be allocated, the descriptor is not used and a packet cannot be received for that descriptor. If no descriptors are used, no packets can be received. The original pbuf for the descriptor is passed up to LWIP and it deallocates it when done.

>In the case the pbuf_alloc fails, no pbuf is queued. Thereby the
>lpc_enetif->prxd[idx].control, lpc_enetif->rxb[idx] etc isn't set.
It's ok that these aren't set or reset as long as the producer index is not incremented (17xx only, this does matter for the 18xx/43xx).

>Why does pbuf_alloc fail? Does I receive to many packets to fast? It is on the local network.
It will fail if a pbuf can't be allocated. If you MEM_SIZE is only 12K, and a receive pbuf uses 1.5K, there isn't a lot of memory to work with before memory runs out.

I do see something odd in the log. Transmit pbufs seem to be going out, but I never see the pbufs for them being reclaimed.
Once a packet has been sent, the transmit interrupt should reclaim the pbuf(s) for that packet in the lpc_tx_reclaim() function.
I'd expect to see something like "lpc_tx_reclaim_st: Freeing packet xxxxx (index x)" in the log.
It might be that the transmit pbufs are not getting freed up and is using all of memory.
Is this something you can verify?

[lpcware]

Content originally posted in LPCWare by tjoAG on Mon Sep 10 05:30:59 MST 2012
I have added pbuf debug and it seems that the de-allocation of buffers isn't handle fast enough?

....
....
...
lpc_low_level_input: Packet received: 20001260, size 64 (index=3)
pbuf_alloc(length=1536)
pbuf_alloc(length=1536) == 20000c48                      <-- Allocated buffer for index 3
lpc_rxqueue_pbuf: pbuf packet queued: 20000c48 (free desc=0)
....
....
lpc_low_level_input: Packet received: 20000c48, size 64 (index=3)
pbuf_alloc(length=1536)
lpc_rx_queue: could not allocate RX pbuf (free desc=1)  <-- Could no allocated
pbuf_free(20000c48)                                     <- Previous allocated first freed here!!!
pbuf_free: deallocating 20000c48


Or is there something else wrong?

Adding more RX buffers (From 3 to 6) only makes it fail more often