Problem with lwip and FreeRTOS on FRDM-K64F board

[gaute_nilsson]

Hello, I have followed Erich Stygers tutorial: lwip with FreeRTOS and the Freescale FRDM-K64F Board​

This works OK, until I load the network interface down.

I added some code to run an iperf server on the board.

After just a few seconds running iperf towards the FRDM-K64F board the code stops on a configASSERT on line 1376 in FreeRTOS queue.c:

#if ( ( INCLUDE_xTaskGetSchedulerState == 1 ) || ( configUSE_TIMERS == 1 ) )

{

    configASSERT( !( ( xTaskGetSchedulerState() == taskSCHEDULER_SUSPENDED ) && ( xTicksToWait != 0 ) ) );

}

The stacktrace:

Thread #1 <main> (Suspended : Signal : SIGINT:Interrupt)

    xQueueGenericReceive() at queue.c:1,376 0x10042

    OSA_MutexLock() at fsl_os_abstraction_free_rtos.c:226 0xb9e4

    sys_mutex_lock() at sys_arch.c:323 0xa77a

    mem_malloc() at mem.c:521 0x43c4

    pbuf_alloc() at pbuf.c:308 0x4afc

    tcp_output_alloc_header() at tcp_out.c:98 0x830c

    tcp_send_empty_ack() at tcp_out.c:854 0x8e46

    tcp_output() at tcp_out.c:931 0x8f28

    tcp_input() at tcp_in.c:383 0x6dec

    ip_input() at ip.c:574 0x1cf8

    ethernet_input() at etharp.c:1,363 0x1488

    ethernetif_input() at ethernetif.c:633 0xa582

    ENET_DRV_ReceiveData() at fsl_enet_driver.c:1,516 0xed4e

    ENET_DRV_RxIRQHandler() at fsl_enet_driver.c:1,927 0xefea

    ENET_Receive_IRQHandler() at lwip_fsl_irq.c:70 0xa61e

    <signal handler called>() at 0xfffffffd

    vPortStartFirstTask() at port.c:965 0xfa8e

    xPortStartScheduler() at port.c:812 0xf9ca

    0x20001b8c

Does anyone know what the problem can be?

I'm using:

• KSDK 1.3.0
• KDS 3.0.0
• ARM Toolchain 4.9.3 20150529 (release)

[adrian_nicolau]

I have a similar stacktrace on KSDK 2.2.0, but I can't seem to find anything related to ENET_RECEIVE_ALL_INTERRUPT. The fsl_enet_driver.h file is no longer present, I assume it's been replaced by fsl_enet.h.

Is there another define that controls this behavior, or has it been removed at all from newer KSDKs?

[adrian_nicolau]

Problem solved by changing the ingress packets callback function from ethernet_input to tcpip_input:

OLD: netif_add(&netif0, &netif0_ipaddr, &netif0_netmask, &netif0_gw, NULL, ethernetif_init, ethernet_input);

NEW: netif_add(&netif0, &netif0_ipaddr, &netif0_netmask, &netif0_gw, NULL, ethernetif_init, tcpip_input);

On incoming packets, this new callback sends a message (by sys_mbox_trypost) that is to be handled by the LwIP task in tcpip_thread (by TCPIP_MBOX_FETCH).

[dleach]

Have you found a solution to this problem yet? If so what did you do?

We found a problem with the FSL examples that exhibited a similar problem you are reporting. I think the problem is that the FSL examples are setup to process the Ethernet interrupt all the way through in the interrupt. It looks like the way to get around this is to change the following in \KSDK\platform\drivers\inc\fsl_enet_driver.h to be zero:

#ifndef ENET_RECEIVE_ALL_INTERRUPT

#define ENET_RECEIVE_ALL_INTERRUPT  1

#endif

The way it is configured to operate with the value of "1" is process the handling of the packet receive and passing the received frame up the protocol stack, in your case TCP/IP, in an interrupt context. The 'assert' is just trying to make sure you didn't reach the function in interrupt space... which will eventually mess up memory since LWIP knows that some of the memory functions are not reentrant. It will work some of the time because the likely the RTOS is in an idle state "most" of the time but occasionally it isn't...

Setting this to "0" will change the behavior where the Ethernet interrupt code will just set an event and return. The FreeRTOS scheduler will see the event and pass control over to a Ethernet receiver thread which processes all outstanding Ethernet receives in non-interrupt mode avoiding any reentrant conditions.

[BlackNight]

As the assert() indicates, either your scheduler has not been started yet or your ticksToWait are zero.

If you check your variables/state at that point, you should hopefully know which one.

Given the call stack, it seems your scheduler has not been started yet. Could it be that you have the ENET interrupts enabled before you have started the scheduler? Make sure you enable the ENET interrupt e.g. from a task, and not before the schedeler (and the RTOS) is running.

I hope this helps,

Erich

[gaute_nilsson]

The scheduler has definitely been started. I'm able to ping the lwip stack before applying more heavy network traffic. As soon as I start to do some more network intensive work, the code stops executing on this assertion.

xTicksToWait is 0xffffffff when this happens.

This is the complete main() function (comments removed):

int main(void)

{

    PE_low_level_init();

    MPU_BWR_CESR_VLD(MPU, 0);

    xTaskCreate(LwipInitTask, "LwipInitTask", configMINIMAL_STACK_SIZE * 4, (void*)NULL, tskIDLE_PRIORITY, (xTaskHandle*)NULL);

#ifdef PEX_RTOS_START

    PEX_RTOS_START();

#endif

    for(;;) {}

}

The rest is copy-paste from the tutorial.

This topic http://www.freertos.org/FreeRTOS_Support_Forum_Archive/April_2015/freertos_vTaskSuspendAll_is_causin... is discussing a similar issue.

The problem seems to be with calling non-ISR safe code from an interrupt.

From my call stack, it seems the lwip port is trying to do a malloc in the ENET receive interrupt, is this OK?

Also, the problem seems to dissapear (masked?) when i change optimization level from -O0 to -O2

[BlackNight]

What does xTaskGetSchedulerState() return at the assertion point? I believe the scheduler might be supspended.
The problem could be as well that the mutex is aquired from inside ISR, but not using the *FromISR() API call?

Erich

[gaute_nilsson]

Yes, you are correct. xTaskGetSchedulerState() returns 0, that is: taskSCHEDULER_SUSPENDED.

The lwip function mem_malloc() does sys_mutex_lock() to prevent the heap from concurrent access. The lwip function sys_mutex_loc() then calls xSemaphoreTake() which is not IRQ safe. I believe this is the problem.

The lwip port has it's own heap handling code (custom written mem_malloc())

I have tried disabling this code and using FreeRTOS code directly by adding:

#define MEM_LIBC_MALLOC         1

#define mem_malloc pvPortMalloc

#define mem_free vPortFree

to the lwipopts.h configuration file, but this did not help, got the same results as before.

Is it generally ok to do a malloc in an interrupt?