MQX is blocking task after enabling interrupts

[michaelschwager]

Running MQX 4.1.2 on Kinetis K60, and latest IAR v7 or something.

I have a setup where I'm reading values from a sensor, then displaying them on the UART, but the display eventually locks up at random times while the sensor continues to run.

The sensor's DRDY output interrupts the K60 at around 400Hz (2.5ms period), and the UART updates every 50ms, showing the last value from the sensor.  So there are multiple interrupts from the sensor for every time the display task runs.  This is so not to overload the terminal software.

The tasks, interrupts, and other things are set up as follows:

1. One task which first blocks on a lightweight event, then prints some stuff on stderr UART, then does a time_delay(50), then repeats loop.  I suspect MQX's handling of time_delay().

2. One task which waits on a semaphore, then reads sensor value from SPI port (using MQX 4.1 dma-and-interrupt driver as SPI master), then posts lightweight event for above task, then starts over.

3.  An interrupt routing which triggers on the sensor's DRDY line, then posts semaphore for second task.

What happens is the display runs fine for a while, then stops in the middle of a printf statement.  Ie the text only goes part way, or sometimes it locks up between lines.  The oscilloscope shows the interrupt line and following SPI read is still going at 400Hz.  When I pause the controller and examine tasks summary with IAR, it says the display task is queue-blocked.  It's not event blocked.

The problem goes away if I remove time_delay at the bottom of the first task, but then there is too much data on the UART.  I've commented out much stuff, so I'm pretty sure there is some weird interaction between the interrupt routine and the time_delay() function.

Prior to enabling hardware interrupts to run the sensors, I was using internal MQX lightweight timer to post the semaphore for the second task.  This did not pose a problem.

A quick and incomplete copy of the code is here:

void device_read_task(uint32_t param) {

  // Reads device values on its own time or on the event, whichever is slower

  MQX_FILE_PTR err = fopen("ittyc:", (const char *) IO_SERIAL_TRANSLATION);

  _io_set_handle(IO_STDERR, err);

  fprintf(stderr, "Starting device_read_task\n");

// .. a bunch of setup stuff, variable declarations, etc.

  while(1) {

  result = _lwevent_wait_ticks(ldev->parent->event, ldev->event_mask, TRUE, 0);

  if (result == LWEVENT_WAIT_TIMEOUT) {

  volatile uint8_t a = 5; // test to see whether the problem is due to event not being sent... this is usually not the case

  }

// ... a bunch of stuff to copy existing values from global location to here...

  fprintf(stderr,"% 5d,   %6hd, %6hd, %6hd,    ", counter++, adc_data[0], adc_data[1], adc_data[2]);

  fprintf(stderr,"% 5.6f, % 5.6f, % 5.6f,    ", xval, yval, zval);

  fprintf(stderr,"% 5.6f, % 5.6f, % 5.6f", xvalcal, yvalcal, zvalcal);

  fprintf(stderr,"\n");

  fflush(stderr);

  _time_delay(p->period);  // When this is removed the lockup problem goes away, but the UART goes too fast

  }

}

void device_run_task(uint32_t param) {

//... bunch of setup stuff

  while(1) { // put something here about enabling or disabling this loop.

  // Wait until the next timer or interrupt...

  _lwsem_wait(ldev->period_semaphore);

  MQX_TICK_STRUCT ticks;

  _time_get_ticks(&ticks);

  async_report.timestamp = ticks.TICKS[0];

// Here we run the SPI bus and read the sensor data, and place in some global variable

  // Post the event which triggers the display/uart task

  result = _lwevent_set(ldev->parent->event, ldev->event_mask);

  }

}

[michaelschwager]

Thanks Martin.  I tried installing the ISR at various priorities: 0, 1, 2, and 7, and they all yielded funny results.

You have also answered my next question, which is that I probably cannot post a semaphore (ie via API) from within a kernel ISR and I must use some other global variable to trigger my task if that's the route I want to go.

If I do this, do I need to disable interrupts from within my kernel ISR?  Or does the hardware take care of that?  What about stack, etc?  Is that handled by the M4 core?  I'm writing this in C with IAR.

thanks

[VinGanZ]

Hi,

I am having a similar problem of "Task Queue Blocked" state.

Did you try increasing the PRINTF UART ITTYX Queue size in the BSP?

WBR
Vinod

[VinGanZ]

I got this somewhat corrected using a larger stack size for the task. I have made it work for 48 hours now and seems to work. but previously it was working similar to 47 hours and was going to "task queue blocked" state.

[michaelschwager]

Hi, I noticed yesterday while debugging many variations of my code (ie, where and how I start up the tasks, enable/disable interrupts, etc.) that MQX was running out of memory while creating a task (I have probably 10 tasks running total). I then actually reduced the stack size for all my tasks, from 8192 to 2048. That, plus creating all my tasks before starting the interrupt source seemed to help.

[michaelschwager]

I didn't change the buffer size, but instead reduced the priority of the interrupt from 0 to 2. That seems to fix this specific problem on this one PCB. However there are other weird issues going on now between this board and a similar one (different version of K60), specifically that some other tasks do not start. These are other monitoring tasks that either use uart (which blocks the blinky led task if started first) or led task (which blocks the uart task if started first). These problems only exist after i introduce hardware pin interrupt.

Maybe I need to disable interrupts from within my ISR but I thought mqx already did that in its generic kernel isr which surrounds my isr. I may also try installing my isr as a kernel isr to bypass the mqx scheduling mechanism and see if that works. I'm not sure what'll happen if I post a semaphore from within a kernel isr, I.e. whether tasks waiting on that semaphore will go or not.

[Martin_]

Hi Michael, MQX managed interrupt with hardware priority 0 or 1 is not allowed. If you need 0 or 1, it must be installed as kernel_isr. An interrupt with hardware priority of 0 or 1 is effectively non-maskable for MQX system (it has higher priority than MQX cricital section), and it is bypassing MQX specific prologue/epilogue, MQX API must not be called from custom kernel isr. You have to develop some other mechanism for synchronization, such as using global variable, also with considering race conditions.

Section 3.9 "Handling interrupts and exceptions" in MQX User's Guide really is worth reading, especially section 3.9.3.4.

-MQX managed interrupt shall be installed with _int_install_isr() followed by _bsp_int_init(). Priority must not be lower than 2 (BSP_DEFAULT_MQX_HARDWARE_INTERRUPT_LEVEL_MAX).

-kernel isr shall be installed either directly to vector table in flash (vectors.c file in BSP) or if vector table is in RAM, using _int_install_kernel_isr(). MQX API must not be called in the kernel isr.

I have seen this kind of weird issues several times, the root case was in all cases a custom interrupt with priority higher than MQX critical section, making a call to MQX API.

-Martin