watchdog fault in tcpip_thread task by software hanging from tcpip_timeouts_mbox_fetch in lwip

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

watchdog fault in tcpip_thread task by software hanging from tcpip_timeouts_mbox_fetch in lwip

1,197 Views
creatorwonny
Contributor III

[System Information]

  • MCUxpresso IDE v11.5.1
  • CPU: IMXRT1062
  • SDK: 2.9.3
  • OS: FreeRTOS

I had a watchdog fault in tcp_thread task. I looked at the stack data of the task and it seemed the system hanging had occurred in tcpip_timeouts_mbox_fetch funciton or functions called inside it.

Stack[032]: 8009bfe0: xQueueReceive() at queue.c:1382 – portYIELD_WITHIN_API();
Stack[051]: 8009bfcf: xQueueReceive() at queue.c:1380 - if( xTaskResumeAll() == pdFALSE )
Stack[058]: 80099c39: lwip_cyclic_timer() at timeouts.c:236 – void lwip_cyclic_timer(void *arg)
Stack[065]: 800fd1ed: sys_arch_mbox_fetch() at sys_arch.c:276 - if( pdTRUE == xQueueReceive( *pxMailBox, &( *ppvBuffer ), ulTimeOut/ portTICK_PERIOD_MS ) )
Stack[069]: 80415540: tcpip_mbox: push {r3,r7}
Stack[077]: 800fddb7: tcpip_timeouts_mbox_fetch() at tcpip.c: 107 res = sys_arch_mbox_fetch(mbox, msg, sleeptime);
Stack[079]: 80415540: tcpip_mbox: push {r3,r7}
Stack[083]: 8008c627: tcpip_thread() at tcpip.c: 151 TCPIP_MBOS_FETCH(&tcpip_mbox, (void***)&msg)
Stack[089]: 8009fa2d: pxPortInitialiseStack at port.c: 220 return pxTopOfStack;

 

static void
tcpip_timeouts_mbox_fetch(sys_mbox_t *mbox, void **msg)
{
  u32_t sleeptime, res;
again:
  LWIP_ASSERT_CORE_LOCKED();
  ...
  UNLOCK_TCPIP_CORE();
  res = sys_arch_mbox_fetch(mbox, msg, sleeptime);
  LOCK_TCPIP_CORE();
  if (res == SYS_ARCH_TIMEOUT) {
    /* If a SYS_ARCH_TIMEOUT value is returned, a timeout occurred
       before a message could be fetched. */
    sys_check_timeouts();
    /* We try again to fetch a message from the mbox. */
    goto again;
  }
 }

u32_t sys_arch_mbox_fetch( sys_mbox_t *pxMailBox, void **ppvBuffer, u32_t ulTimeOut )
{
void *pvDummy;
TickType_t xStartTime, xEndTime, xElapsed;
unsigned long ulReturn;

    xStartTime = xTaskGetTickCount();

    if( NULL == ppvBuffer )
    {
        ppvBuffer = &pvDummy;
    }

    if( ulTimeOut != 0UL )
    {
        if( pdTRUE == xQueueReceive( *pxMailBox, &( *ppvBuffer ), ulTimeOut/ portTICK_PERIOD_MS ) )
        {
            xEndTime = xTaskGetTickCount();
            xElapsed = ( xEndTime - xStartTime ) * portTICK_PERIOD_MS;
            ulReturn = xElapsed;
        }
        else
        {
            /* Timed out. */
            *ppvBuffer = NULL;
            ulReturn = SYS_ARCH_TIMEOUT;
        }
    }
    else
    {
        ...
    }
    return ulReturn;
}

 

I don't know if the software was really stuck at xQueueReceive or if it runs infinitely because the sys_arch_mbox_fetch kept returning SYS_ARCH_TIMEOUT.

This issue isn't occurred in normal condition. It only happens occasionally once in a day when there is modbus tcp clients keeps sending a packet by connecting and disconnecting to modbus tcp server running on our software for each packet.

1.png

In addition, I can see following messages a lot when there is a lot of connection and disconnection between modbus tcp client and server.

tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 1619121200.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 2001380479.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 3118458873.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 1190114352.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 1427156108.
tcp_input: no PCB match found, resetting.
tcp_rst: seqno 0 ackno 3511738084.

 Can I get any advice on this issue? I don't know what to check and how to solve it.

0 Kudos
Reply
2 Replies

779 Views
rak14
Contributor III

Hi @creatorwonny ,

Did you able to resolve this issue? If yes, can you explain what was the problem and how did you solve the same?

0 Kudos
Reply

1,163 Views
creatorwonny
Contributor III

There is one more thing I noticed. When there was watchdog fault in tcpip_thr task, other tasks also had the watchdog fault and all of the task were stuck at xTaskResumeAll funciton. Can this help?

WATCHDOG - tcpip_thr task failed

Stack[031]: 8009d9ed: xTaskResumeAll() at tasks.c:2303 – return xAlreadyYielded

Stack[032]: 8009bfe0: xQueueReceive() at queue.c:1382 – portYIELD_WITHIN_API();

Stack[051]: 8009bfcf: xQueueReceive() at queue.c:1380 - if( xTaskResumeAll() == pdFALSE )

Stack[058]: 80099c39: lwip_cyclic_timer() at timeouts.c:236 – void lwip_cyclic_timer(void *arg)

Stack[065]: 800fd1ed: sys_arch_mbox_fetch() at sys_arch.c:276 - if( pdTRUE == xQueueReceive( *pxMailBox, &( *ppvBuffer ), ulTimeOut/ portTICK_PERIOD_MS ) )

Stack[069]: 80415540: tcpip_mbox: push {r3,r7}

Stack[077]: 800fddb7: tcpip_timeouts_mbox_fetch() at tcpip.c: 107 res = sys_arch_mbox_fetch(mbox, msg, sleeptime);

Stack[079]: 80415540: tcpip_mbox: push {r3,r7}

Stack[083]: 8008c627: tcpip_thread() at tcpip.c: 151 TCPIP_MBOS_FETCH(&tcpip_mbox, (void***)&msg)

Stack[089]: 8009fa2d: pxPortInitialiseStack at port.c: 220 return pxTopOfStack;

WATCHDOG - syslog task failed

Stack[031]: 8009d9ed: xTaskResumeAll

Stack[032]: 8009d284: vTaskDelayUntil

Stack[051]: 8009d275: vTaskDelayUntil

Stack[063]: 80085623: vSyslogTask

Stack[071]: 8009fa2d: pxPortInitialiseStack

WATCHDOG - mbtcpsrv task failed

Stack[031]: 8009d9ed: xTaskResumeAll

Stack[032]: 8009c208: xQueueSemaphoreTake

Stack[051]: 8009c1f5: xQueueSemaphoreTake

Stack[067]: 800fd2b5: sys_arch_sem_wait

Stack[075]: 8008af29: lwip_select

Stack[109]: 800768f3: MBTCPSrvTask

Stack[151]: 8009fa2d: pxPortInitialiseStack

WATCHDOG - system_mo task failed

Stack[031]: 8009d9ed: xTaskResumeAll

Stack[032]: 8009d284: vTaskDelayUntil

0 Kudos
Reply