"Task Queue Blocked" Problem

[VinGanZ]

Hello,

I am having a situation where MQX 4.0 on Kinetis K60 entering "Task Queue Blocked" status.

when looking at TAD output on IAR and does not come out from there for hours together and just hangs there.

Please help me

Vinod

[VinGanZ]

Making task memory stack to 4500 solved the issue.

But the question remains as the max stack utilisation is shown as less than 51% and why should it does not work with 3000 instead of 4500?

[soledad]

Hi Vinod,

Let me give you what I have seen in my experience: If you think that you will need 1K of stack then I will tell you set to 50% more than what you expect to need then you will have to set 1.5K. Remember that you can calculate that by looking the amount of variables or arrays that you are defining in the task. The minimum that you should give to a task for the TaskControlBlock (TCB) is about 300 bytes. That is why you will give another 50% or 30%.

Regards

Sol

[pmt]

Sol,

This is sort of an arm wavy answer and doesn't address the problem.  The stack is either overflowing or not overflowing.  In this case the stack is not overflowing according to the evidence presented.  We're running into the same issue and increasing stack size does not help.  

There seem to be two problems assuming the stack is not being corrupted:

- _io_serial_int_putc_internal() is being called, at least under some circumstances, in the context of and ISR which _taskq_suspend() does not support and the active task is then halted when this code path is executed.

-  _taskq_suspend() is being called with interrupts disabled and they are left disabled.  So at worst the task is halted and the entire system locks up.  At best, interrupts are left disabled far too long.   

-  In addition the ITTY drivers really abuse _int_disable() and _int_enable() for the purposes of mutual exclusion of the TX queue object.  Consider a semaphore instead that would not shut down the entire interrupt subsystem.     

I suspect the ITTY drivers were not adequately tested in a TX overflow condition.  IMO, the problem seems to be with the driver and not with stack allocation, at least at first glance.  If it is a really problem with the stack then I suspect in a TX overflow condition the stack is growing unbound and just increasing the stack size will not ultimately solve the problem.

Seems that a quick fix would be just to force the driver to IO_SERIAL_NON_BLOCKING mode and discard characters on an overflow condition if this can be tolerated. 

Regards,

PMT

[pmt]

I did some more investigation of this problem and I believe it to be a problem with the MQX Serial I/O driver.  The condition that causes a lockup is when XON/XOFF flow control is enabled and the TX queue fills up.  The driver then gets into a situation (perhaps because of a COM error condition) where the task queue never recovers. 

Steps to reproduce:

Use this open configuration:

#define BSP_DEFAULT_IO_OPEN_MODE (void *) (IO_SERIAL_XON_XOFF | IO_SERIAL_TRANSLATION | IO_SERIAL_ECHO)

Have the MQX device print an endless stream to the console

Open a Putty terminal with XON/XOFF flow control enabled

Hit CRTL-S to enable XOFF

Wait long enough for the TX buffer to fill up on the MQX side

Hit CRTL-Q to enable XON

The port is now stuck, transmission never resumes.

PMT

[macl]

Hi,

What is your application? 

Information on using task queues are avaialable in the MQX users guide.

See section 3.6.9 Task Queues

http://cache.freescale.com/files/32bit/doc/user_guide/MQX_User_Guide.pdf

Mac

[VinGanZ]

Hi Mac,

Thank you very much for the reply.

The application is a custom board used for receiving UART Data from a zigbee module.

We have had the code running without any issues previously but recently we had to change the ITTYF Queue Buffer size to 2048 bytes in BSP.

But the "Task Queue Blocked" error is coming from a task that extensively uses PRINTF function and the ITTYB which is used for printf uses 256 bytes Queue Buffer size. The Code works for about 24 hours before it goes to "Task Queue Blocked" state and hangs there at serlint.c which calls taskq.c. I have indicated the line of code where it hangs too

/*FUNCTION****************************************************************

*

* Function Name    : _io_serial_int_putc_internal

* Returned Value   : void

* Comments         :

*   This function writes out the character to the device if the queue

* is empty, or it writes it to the device.  If the queue is full, this

* function will suspend the writing task.

*

*END*********************************************************************/

boolean _io_serial_int_putc_internal

   (

      /* [IN] the interrupt io device information */

      IO_SERIAL_INT_DEVICE_STRUCT_PTR int_io_dev_ptr,

      /* [IN] the character to print out */

      char                     c,

      _mqx_uint                flags

   )

{ /* Body */

   volatile CHARQ_STRUCT _PTR_ out_queue;

   /* Start CR 388 */

#if (PSP_MEMORY_ADDRESSING_CAPABILITY > 8 )

   c &= 0xFF;

#endif

   /* End CR 388 */

   out_queue = int_io_dev_ptr->OUT_QUEUE;

   _int_disable();

   if(flags & IO_SERIAL_NON_BLOCKING) {

      if (_CHARQ_FULL(out_queue)) {

          _int_enable();

          return FALSE;

      } /* Endif */

   } else {

      if(int_io_dev_ptr->HAVE_STOPPED_OUTPUT) {

          _taskq_suspend(int_io_dev_ptr->OUT_WAITING_TASKS);

      } /* Endif */

      while (_CHARQ_FULL(out_queue)) {

         /* Lets wait */

         _taskq_suspend(int_io_dev_ptr->OUT_WAITING_TASKS);  //HANGS HERE

      } /* Endif */

   } /* Endif */

   if (int_io_dev_ptr->OUTPUT_ENABLED || int_io_dev_ptr->HAVE_STOPPED_OUTPUT) {

      _CHARQ_ENQUEUE(out_queue,c);

   } else {

      int_io_dev_ptr->OUTPUT_ENABLED = TRUE;

      (*int_io_dev_ptr->DEV_PUTC)(int_io_dev_ptr, c);

   } /* Endif */

   _int_enable();

   return TRUE;

} /* Endbody */

and in taskq.c it returns

_mqx_uint _taskq_suspend

(

    pointer users_task_queue_ptr

)

{ /* Body */

    register KERNEL_DATA_STRUCT_PTR kernel_data;

    register TD_STRUCT_PTR td_ptr;

    register TASK_QUEUE_STRUCT_PTR task_queue_ptr = (TASK_QUEUE_STRUCT_PTR) users_task_queue_ptr;

    _GET_KERNEL_DATA(kernel_data);

    _KLOGE2(KLOG_taskq_suspend, users_task_queue_ptr);

#if MQX_CHECK_ERRORS

    if (task_queue_ptr == NULL) {

        _KLOGX2(KLOG_taskq_suspend, MQX_INVALID_PARAMETER);

        return(MQX_INVALID_PARAMETER);

    } /* Endif */

    if (kernel_data->IN_ISR) {

        _KLOGX2(KLOG_taskq_suspend, MQX_CANNOT_CALL_FUNCTION_FROM_ISR);

        return(MQX_CANNOT_CALL_FUNCTION_FROM_ISR); // this is returned to function _io_serial_int_putc_internal

    }/* Endif */

#endif

    td_ptr = kernel_data->ACTIVE_PTR;

    _INT_DISABLE();

#if MQX_CHECK_VALIDITY

    if (task_queue_ptr->VALID != TASK_QUEUE_VALID) {

        _int_enable();

        _KLOGX2(KLOG_taskq_suspend, MQX_INVALID_TASK_QUEUE);

        return(MQX_INVALID_TASK_QUEUE);

    } /* Endif */

#endif

    td_ptr->STATE = TASK_QUEUE_BLOCKED;

    _QUEUE_UNLINK(td_ptr); /* Remove task from ready to run queue */

    td_ptr->INFO = (_mqx_uint) & task_queue_ptr->TD_QUEUE;

    if (task_queue_ptr->POLICY & MQX_TASK_QUEUE_BY_PRIORITY) {

        _sched_insert_priorityq_internal(&task_queue_ptr->TD_QUEUE, td_ptr);

    }

    else {

        _QUEUE_ENQUEUE(&task_queue_ptr->TD_QUEUE, td_ptr);

    } /* Endif */

    _sched_execute_scheduler_internal(); /* Let the other tasks run */

    _INT_ENABLE();

    _KLOGX2(KLOG_taskq_suspend, MQX_OK);

    return (MQX_OK);

} /* Endbody */

Vinod

[soledad · Accepted solution]

Hi Vinod,

Could please try increasing the stack size of the task? For example if the task has 1500 of stack, then try using 3000.

Please let me know if this helps.

Have a great day,
Sol

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[VinGanZ]

Hi Soledad,

I have had it done and it worked for about 47 hours before it crashed again. Now i have increased the stack size from 1500 to 4500 and after 48 hours still it has not gone bad. Fingers crossed now to see how long that will work!!

Thanks for your reply and suggestion.

Best Regards

Vinod Ganesh