[System Information]
I had a watchdog fault in tcp_thread task. I looked at the stack data of the task and it seemed the system hanging had occurred in tcpip_timeouts_mbox_fetch funciton or functions called inside it.
Stack[032]: 8009bfe0: xQueueReceive() at queue.c:1382 – portYIELD_WITHIN_API();
Stack[051]: 8009bfcf: xQueueReceive() at queue.c:1380 - if( xTaskResumeAll() == pdFALSE )
Stack[058]: 80099c39: lwip_cyclic_timer() at timeouts.c:236 – void lwip_cyclic_timer(void *arg)
Stack[065]: 800fd1ed: sys_arch_mbox_fetch() at sys_arch.c:276 - if( pdTRUE == xQueueReceive( *pxMailBox, &( *ppvBuffer ), ulTimeOut/ portTICK_PERIOD_MS ) )
Stack[069]: 80415540: tcpip_mbox: push {r3,r7}
Stack[077]: 800fddb7: tcpip_timeouts_mbox_fetch() at tcpip.c: 107 res = sys_arch_mbox_fetch(mbox, msg, sleeptime);
Stack[079]: 80415540: tcpip_mbox: push {r3,r7}
Stack[083]: 8008c627: tcpip_thread() at tcpip.c: 151 TCPIP_MBOS_FETCH(&tcpip_mbox, (void***)&msg)
Stack[089]: 8009fa2d: pxPortInitialiseStack at port.c: 220 return pxTopOfStack;
static void
tcpip_timeouts_mbox_fetch(sys_mbox_t *mbox, void **msg)
{
u32_t sleeptime, res;
again:
LWIP_ASSERT_CORE_LOCKED();
...
UNLOCK_TCPIP_CORE();
res = sys_arch_mbox_fetch(mbox, msg, sleeptime);
LOCK_TCPIP_CORE();
if (res == SYS_ARCH_TIMEOUT) {
/* If a SYS_ARCH_TIMEOUT value is returned, a timeout occurred
before a message could be fetched. */
sys_check_timeouts();
/* We try again to fetch a message from the mbox. */
goto again;
}
}
u32_t sys_arch_mbox_fetch( sys_mbox_t *pxMailBox, void **ppvBuffer, u32_t ulTimeOut )
{
void *pvDummy;
TickType_t xStartTime, xEndTime, xElapsed;
unsigned long ulReturn;
xStartTime = xTaskGetTickCount();
if( NULL == ppvBuffer )
{
ppvBuffer = &pvDummy;
}
if( ulTimeOut != 0UL )
{
if( pdTRUE == xQueueReceive( *pxMailBox, &( *ppvBuffer ), ulTimeOut/ portTICK_PERIOD_MS ) )
{
xEndTime = xTaskGetTickCount();
xElapsed = ( xEndTime - xStartTime ) * portTICK_PERIOD_MS;
ulReturn = xElapsed;
}
else
{
/* Timed out. */
*ppvBuffer = NULL;
ulReturn = SYS_ARCH_TIMEOUT;
}
}
else
{
...
}
return ulReturn;
}
I don't know if the software was really stuck at xQueueReceive or if it runs infinitely because the sys_arch_mbox_fetch kept returning SYS_ARCH_TIMEOUT.
This issue isn't occurred in normal condition. It only happens occasionally once in a day when there is modbus tcp clients keeps sending a packet by connecting and disconnecting to modbus tcp server running on our software for each packet.
In addition, I can see following messages a lot when there is a lot of connection and disconnection between modbus tcp client and server.
tcp_input: no PCB match found, resetting. tcp_rst: seqno 0 ackno 1619121200. tcp_input: no PCB match found, resetting. tcp_rst: seqno 0 ackno 2001380479. tcp_input: no PCB match found, resetting. tcp_rst: seqno 0 ackno 3118458873. tcp_input: no PCB match found, resetting. tcp_rst: seqno 0 ackno 1190114352. tcp_input: no PCB match found, resetting. tcp_rst: seqno 0 ackno 1427156108. tcp_input: no PCB match found, resetting. tcp_rst: seqno 0 ackno 3511738084.
Can I get any advice on this issue? I don't know what to check and how to solve it.
Hi @creatorwonny ,
Did you able to resolve this issue? If yes, can you explain what was the problem and how did you solve the same?
There is one more thing I noticed. When there was watchdog fault in tcpip_thr task, other tasks also had the watchdog fault and all of the task were stuck at xTaskResumeAll funciton. Can this help?
WATCHDOG - tcpip_thr task failed
Stack[031]: 8009d9ed: xTaskResumeAll() at tasks.c:2303 – return xAlreadyYielded
Stack[032]: 8009bfe0: xQueueReceive() at queue.c:1382 – portYIELD_WITHIN_API();
Stack[051]: 8009bfcf: xQueueReceive() at queue.c:1380 - if( xTaskResumeAll() == pdFALSE )
Stack[058]: 80099c39: lwip_cyclic_timer() at timeouts.c:236 – void lwip_cyclic_timer(void *arg)
Stack[065]: 800fd1ed: sys_arch_mbox_fetch() at sys_arch.c:276 - if( pdTRUE == xQueueReceive( *pxMailBox, &( *ppvBuffer ), ulTimeOut/ portTICK_PERIOD_MS ) )
Stack[069]: 80415540: tcpip_mbox: push {r3,r7}
Stack[077]: 800fddb7: tcpip_timeouts_mbox_fetch() at tcpip.c: 107 res = sys_arch_mbox_fetch(mbox, msg, sleeptime);
Stack[079]: 80415540: tcpip_mbox: push {r3,r7}
Stack[083]: 8008c627: tcpip_thread() at tcpip.c: 151 TCPIP_MBOS_FETCH(&tcpip_mbox, (void***)&msg)
Stack[089]: 8009fa2d: pxPortInitialiseStack at port.c: 220 return pxTopOfStack;
WATCHDOG - syslog task failed
Stack[031]: 8009d9ed: xTaskResumeAll
Stack[032]: 8009d284: vTaskDelayUntil
Stack[051]: 8009d275: vTaskDelayUntil
Stack[063]: 80085623: vSyslogTask
Stack[071]: 8009fa2d: pxPortInitialiseStack
WATCHDOG - mbtcpsrv task failed
Stack[031]: 8009d9ed: xTaskResumeAll
Stack[032]: 8009c208: xQueueSemaphoreTake
Stack[051]: 8009c1f5: xQueueSemaphoreTake
Stack[067]: 800fd2b5: sys_arch_sem_wait
Stack[075]: 8008af29: lwip_select
Stack[109]: 800768f3: MBTCPSrvTask
Stack[151]: 8009fa2d: pxPortInitialiseStack
WATCHDOG - system_mo task failed
Stack[031]: 8009d9ed: xTaskResumeAll
Stack[032]: 8009d284: vTaskDelayUntil