assert fault in ethernetif_drop_frame

[creatorwonny]

Hi all,

I have had a assert fault issue in ethernetif_drop_frame. It was very intermittent but it occurred once while debugging with J-Link connected so I got some information. This happened because  ENET_GetRxBuffer() returned kStatus_ENET_RxFrameFail so that assert occurred at the line of LWIP_ASSERT("ENET_GetRxBuffer() status != kStatus_Success", status == kStatus_Success).

static void ethernetif_drop_frame(struct ethernetif *ethernetif)
{
    ...
    do
    {
#if 0 /* Error statisctics */
        enet_data_error_stats_t eErrStatic;
        /* Get the error information of the received g_frame. */
        ENET_GetRxErrBeforeReadFrame(&ethernetif->handle, &eErrStatic);
#endif
        status = ENET_GetRxBuffer(ethernetif->base, &ethernetif->handle, &buffer, &len, 0, &isLastBuff, &ts);
        LWIP_UNUSED_ARG(status); /* for LWIP_NOASSERT */
        LWIP_ASSERT("ENET_GetRxBuffer() status != kStatus_Success", status == kStatus_Success);
        ENET_ReleaseRxBuffer(ethernetif->base, &ethernetif->handle, buffer, 0);
    } while (!isLastBuff);
    ...
}
status_t ENET_GetRxBuffer(ENET_Type *base,
                          enet_handle_t *handle,
                          void **buffer,
                          uint32_t *length,
                          uint8_t ringId,
                          bool *isLastBuff,
                          uint32_t *ts)
{
    enet_rx_bd_ring_t *rxBdRing                  = &handle->rxBdRing[ringId];
    volatile enet_rx_bd_struct_t *curBuffDescrip = rxBdRing->rxBdBase + rxBdRing->rxGenIdx;
    uint32_t address;

    /* Check if current rx BD is under usage by certain application */
    /* Buffer owner flag, 1: owned by application, 0: owned by driver */
    if ((curBuffDescrip->control & ENET_BUFFDESCRIPTOR_RX_SOFTOWNER1_MASK) == 0U)
    {
        curBuffDescrip->control |= ENET_BUFFDESCRIPTOR_RX_SOFTOWNER1_MASK;
    }
    else
    {
        return kStatus_ENET_RxFrameFail;
    }

A picture below shows what functions had been called when the assert occurred.

I don't understand why ENET_GetRxFrameSize() returned kStatus_ENET_RxFrameError so  ethernetif_drop_frame function was called.

struct pbuf *ethernetif_linkinput(struct netif *netif)
{
    ...
    /* Obtain the size of the packet and put it into the "len" variable. */
    status = ENET_GetRxFrameSize(&ethernetif->handle, &len, 0);
    if (status == kStatus_Success)
    {
        /* Read frame. */
        p = ethernetif_read_frame(ethernetif, len);
    }
    else if (status != kStatus_ENET_RxFrameEmpty)
    {
        /* Drop the frame when error happened. */
        ethernetif_drop_frame(ethernetif);
    }
    return p;
}

According to the pictures below, rxBdBase is 0x20000000 (DTC).

I captured the memory data from 0x20000000 as following.

ENET_GetRxFrameSize returns kStatus_ENET_RxFrameError when descriptor's length is 0 or last buffer descriptor's frame has an error. Looking at the memory data above, there shouldn't be the case.

#define ENET_RING_NUM   (1U)
#define ENET_RXBD_NUM   (10U)
#define ENET_TXBD_NUM   (6U)

status_t ENET_GetRxFrameSize(enet_handle_t *handle, uint32_t *length, uint8_t ringId)
{
    /* Reset the length to zero. */
    *length = 0;

    uint16_t validLastMask                       = ENET_BUFFDESCRIPTOR_RX_LAST_MASK | ENET_BUFFDESCRIPTOR_RX_EMPTY_MASK;
    enet_rx_bd_ring_t *rxBdRing                  = &handle->rxBdRing[ringId];
    volatile enet_rx_bd_struct_t *curBuffDescrip = rxBdRing->rxBdBase + rxBdRing->rxGenIdx;
    uint16_t index                               = rxBdRing->rxGenIdx;
    bool isReturn                                = false;
    status_t result                              = kStatus_Success;

    /* Check the current buffer descriptor's empty flag. If empty means there is no frame received. */
    /* If this buffer descriptor is owned by application, return empty. Only need to check the first BD's owner if one
     * frame in mutiple BDs. */
    if (0U != (curBuffDescrip->control & (ENET_BUFFDESCRIPTOR_RX_EMPTY_MASK | ENET_BUFFDESCRIPTOR_RX_SOFTOWNER1_MASK)))
    {
        isReturn = true;
        result   = kStatus_ENET_RxFrameEmpty;
    }
    else
    {
        do
        {
            /* Add check for abnormal case. */
            if (curBuffDescrip->length == 0U)
            {
                isReturn = true;
                result   = kStatus_ENET_RxFrameError;
                break;
            }

            /* Find the last buffer descriptor. */
            if ((curBuffDescrip->control & validLastMask) == ENET_BUFFDESCRIPTOR_RX_LAST_MASK)
            {
                isReturn = true;
                /* The last buffer descriptor in the frame check the status of the received frame. */
                if (0U != (curBuffDescrip->control & ENET_BUFFDESCRIPTOR_RX_ERR_MASK))
                {
                    result = kStatus_ENET_RxFrameError;
                    break;
                }

Can I know why this happen and how to solve this assert fault issue?

[creatorwonny · Accepted solution]

This issue was resolved after updating nxp driver and lwip port codes below to the SDK 2.11.1.

• drivers/fsl_enet.c
• drivers/fsl_enet.h
• lwip/port/enet_ethernetif.c
• lwip/port/enet_ethernetif.h
• lwip/port/enet_ethernetif_kinetis.c
• lwip/port/sys_arch.h
• lwip/port/arch/cc.h
• lwip/port/arch/sys_arch.h

[Hui_Ma]

Hi,

Thank you for the info.

Plese help to provide which RT product you are using?

Whats the test environments info (IDE/SDK version/EVK board version)?

And please help to guide us how to regenerate this issue. Thanks.

Mike

[creatorwonny]

Hi

Sorry, I thought I had put these information in the post. 

[System Information]

• MCUxpresso IDE v11.5.1
• CPU: IMXRT1062
• SDK: 2.9.3
• OS: FreeRTOS

I tried to make this issue reproducible but couldn't. This issue is very intermittent and take a couple of days or more. What I can say for sure is that It occurs when there is a lot of connection and disconnection between Modbus Server on our controller and Modbus Client on PC like picture below.

In addition, I can see following messages in our logging.

tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 1619121200.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 2001380479.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 3118458873.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 1190114352.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 1427156108.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 3511738084.

I posted another question of watchdog fault issue which is related to this condition, but type of fault is different.

I think that some setting for lwip might have been wrong, however I don't know which part I have to look at.

[creatorwonny]

This assert occurs a little less if increase the ENET_RXBD_NUM value. It was set to 10 and changed it to 20. I think increasing this value whenever such this problem occurs is not a right solution. What other things can I try to solve this issue?

This is a python script. It creates a specified number of threads and each thread create a Modbus client. The Modbus client sends a variable length packet at random cycle time and it automatically connects and disconnects to the server every packet.

number_of_threads = 50

def work(work_id, host, port, unit_id):
    client = ModbusClient(host=host, port=port, unit_id=unit_id, auto_open=True, auto_close=True)
    for index in range(100000):
        number_of_variables = random.randint(1, 50)
        cycle_time = random.randint(1, 500)
        regs = client.read_holding_registers(0, number_of_variables)
        if regs:
            print("id:%d:%s" % (work_id, regs))
        else:
            print("id:%d:read error" % work_id)
        time.sleep(0.001 * cycle_time)

# Press the green button in the gutter to run the script.
if __name__ == '__main__':
    thread_id = 0
    for loop in range(number_of_threads):
        thread_id = thread_id + 1
        t = Thread(target=work, args=(thread_id, "192.168.3.102", 1502, 1))
        t.start()

    while 1:
        time.sleep(1)