Posting semaphore from SPI IRQ crashes app

[joséantoniomart]

Hi all,

I am trying to develop a transport protocol over SPI to comunnicate a FRDM-KL27Z board (master) with a CC2640 from TI (slave).

I added a callback call at the end of function SPI_DRV_MasterCompleteTransfer() in file fsl_spi_master_driver.c to inform my app when a SPI transfer is completed. The application launches a SPI write operation and then is waiting in a semaphore in its main loop.

Once the SPI transfer is completed, the function SPI_DRV_MasterIRQHandler() is called, and this ends calling my callback function. From it, I call OSA_SemaPost(&app_main_sem), and this call makes my app crash.

I use lwsemaphores in other parts of the program and they are working OK. Also, I have put a breakpoint before the post operation and everything seems ok with the semaphore.

Any ideas or suggestions?

Thank you very much

[Alice_Yang]

Hello Jose,

I think maybe the reason is the lwsemaphores run quickly .

And could you show your code  about this process :

Once the SPI transfer is completed, the function SPI_DRV_MasterIRQHandler() is called, and this ends calling my callback function. From it, I call OSA_SemaPost(&app_main_sem),

And when app crash , what does it show ?

Hope it helps

Alice

[joséantoniomart]

Hi Alice,

Thank you very much for your interest and your quick response.

I'm sorry, but I think I don't understand the explanation "the lwsemaphores run quickly" and how can it affect in my case (I am novice developing in this platform).

On the other way, I can show anything when app crashes: how could I see anything? I can confirm the crash using breakpoints and also switching on/off the board's led (using different colors to debug). Could you suggest me any other tools or procedures to help me find what is happening? Any suggestions are welcome!.

The following are the relevant parts of my code for the named process (I can send you the code files you want, or even a zip with the complete project if you wish). These are the lines I have added in fsl_spi_master_driver.c:

  // declare reference to callback

  spi_transfer_done_cb_t transfer_done_cb = NULL;

  // public function to set callback

  void SPI_DRV_MasterSetTransferDoneCallback(spi_transfer_done_cb_t transfer_done)

  {

    transfer_done_cb = transfer_done;

  }

  static void SPI_DRV_MasterCompleteTransfer(uint32_t instance)

  {

    ...

    ...

    if (spiState->isTransferBlocking)

    {

      /* Signal the synchronous completion object */

      OSA_SemaPost(&spiState->irqSync);

    }

    // I added the following call to callback function at this point

    if (transfer_done_cb)

    {

      transfer_done_cb();

    }

  }

And this is the relevant code of my main file:

  ...

  // SW3 button interrupt used to request a SPI write operation

  static void MQX_PORTBCDE_IRQHandler(void);

  static void npi_app_tl_packet_parser(void);

  static void transfer_done_callback(void);

  static tl_cbs_t npi_app_tl_cbs = {

    npi_app_tl_packet_parser,

  };

  void Main_task(uint32_t param);

  // The lower "Priority" field is, the higher is the actual priority.

  const TASK_TEMPLATE_STRUCT  MQX_template_list[] = {

    /* Task Index,   Function,   Stack,  Priority, Name,     Attributes,          Param, Time Slice */

    { MAIN_TASK,  Main_task, 700,   20,        "main", MQX_AUTO_START_TASK, 0,     0 },

    { 0 }

  };

  void Main_task(uint32_t param) {

    hardware_init();

    OSA_Init();

    // SRDY / SW1 interrupt

    switchPins[0].config.interrupt = kPortIntFallingEdge;

    // SW3 button interrupt

    switchPins[1].config.interrupt = kPortIntFallingEdge;

    GPIO_DRV_Init(switchPins, ledPins);

    GPIO_DRV_WritePinOutput(kGpioCSn, 1);

    configure_spi_pins(SPI0_IDX);

    // SW3 button interrupt

    OSA_InstallIntHandler(PORTBCDE_IRQn, MQX_PORTBCDE_IRQHandler);

    NVIC_SetPriority(PORTBCDE_IRQn, 6);

    osa_status_t status = OSA_SemaCreate(&app_main_sem, 0);

    if (status != kStatus_OSA_Success) {

      printf("Error creating main app semaphore: %d\n", status);

      return;

    }

    tl_init(&npi_app_tl_cbs, TRANSPORT_TX_DONE_EVT,

       TRANSPORT_RX_EVT, SPI_SRDY_EVT);

    SPI_DRV_MasterSetTransferDoneCallback(&transfer_done_callback);

    for(;;) {

      do {

        status = OSA_SemaWait(&app_main_sem, OSA_WAIT_FOREVER);

      } while(status == kStatus_OSA_Idle);

      if (status != kStatus_OSA_Success) {

        printf("Error: syncStatus: %d\n", status);

      }

      tl_handle_ISR_event();

      if (events & SPI_START_WRITE_EVT) {

        events &= ~SPI_START_WRITE_EVT;

        npi_app_write();

      }

    }

  }

  static void MQX_PORTBCDE_IRQHandler(void) {

    PORT_HAL_ClearPortIntFlag(PORTC_BASE_PTR);

    events |= SPI_START_WRITE_EVT;

    OSA_SemaPost(&app_main_sem);

  }

  static void transfer_done_callback(void) {

    // Post the event to the task thread.

    TX_DONE_ISR_EVENT_FLAG = transport_tx_done_event;

    // App crashes when calling this function

    OSA_SemaPost(&app_main_sem);

  }

As you can guess from the code, I start the board, and then I use SW3 button interrupt to force a SPI write. This causes 270 bytes to be writed to the bus, and correctly received in the other side. The, when the SPI transfer is complete the transfer_done_cb() I have added is called.

Regards,

José Antonio Martínez

[davidtietz]

Hi José,

Is this something that happens immediately on the first transfer or the transfer works for a while and then stops responding?

David

[joséantoniomart]

<EDITED>

Hi David,

Thank you for your interest. It happens always after finishing first transfer (note however that our SPI transfer is composed of 270 bytes).

What I have found is that app crashes in _lwsem_post() function (in lwsem.c file), when trying to recover the main task's context from the context of the SPI ISR where this code is launched.

Specifically, it crashes on following line:

    if ((sem_ptr->VALUE >= 0) && (_QUEUE_GET_SIZE(&sem_ptr->TD_QUEUE)))

    {

        _QUEUE_DEQUEUE(&sem_ptr->TD_QUEUE, td_ptr);

        _BACKUP_POINTER(td_ptr, TD_STRUCT, AUX_QUEUE);

        _TIME_DEQUEUE(td_ptr, kernel_data);

        td_ptr->INFO = 0; /* Signal that post is activating the task */

        _TASK_READY(td_ptr, kernel_data);

        _CHECK_RUN_SCHEDULER(); /* Let higher priority task run */ // <<<<<<<<<<<<<<<<<<<<< CRASHES HERE !!!!>>>>>>>>>>>>>>>>>>>>>>

    }

We continue investigating.

Thank you very much!

Regards,

José Antonio Martínez.

[davidtietz]

Hi José,

The reason I asked was I had seen an issue in MQX 4.0 where there appeared to be an unhandled interrupt ending up in the SPI RX ISR. This would cause the semaphore to decrement one extra count each time this happened. Eventually the semaphore would rollover and would lock up the system. This is the thread:

https://community.freescale.com/message/407549#407549

This doesn't sound like your problem though, so I don't know how much help I can be. Have you installed the unexpected ISR to see if you end up there on your crash? Just a thought.

David

[joséantoniomart]

Hi David,

I have taken a look to the thread you suggest (again), but I could not see anything directly related to my case. What I see is not the ISR being called more times than expected (at least when using breakpoints to debug), but the execution fails in any case (while the semaphore seems to be OK, and it (I refer now to the option under MQX menu in KDS) shows an only task waiting on it as expected just before posting).

Please be patient with me, but I think I don't understand what you mean with "installed the unexpected ISR". Could you explain me in more detail?.

By the way, could you give me any clue about a tool or  procedure to "know what is happening" when app seems to crash? I don't know any way to see any kernel messages or similar that inform about the problem.

Thank you very much.

Regards,

José Antonio Martínez.

[Alice_Yang]

Hello David,

When you not use the lwsemaphores , does it can work well ?

Which IDE do you use ? And when it crash,  please screenshot the window of your IDE ?

And after i checked your code , it hard to fund the problem , could you send your project to me , and tell me your IDE and the version.

BR

Alice

[joséantoniomart]

Hi Alice,

In addition to previous responses, I just have send you the following files:

• A zip with my full project (it has some minor modifications so as the code can be tested using only the evaluation board FRDM-KL27Z).
• The source files fsl_spi_master_driver.c and fsl_spi_master_driver.h: they include my minor changes to invoque a callback when SPI transaction finishes.
• Three screen captures: corresponding to the moments before posting semaphore, after posting it (execution doesn't stop in breakpoint as expected), and pausing then execution to see what is executed.

To reproduce the problem, I do the following steps:

1. Launch a debug session.
2. Put a breakpoint in the conflicting OSA_SemaPost() line (that is tl.c, line 67).
3. Pulse SW3 button in evaluation board: this causes a NPI write transaction to be started and so, execution stops in the breakpoint
4. Now, I can put another breakpoints, for example in lwsem.c in lines 569 and 583 (inside implementation of _lwsem_post()): using these breakpoints one can see that the first one is reached, but code never reachs the second one (after _CHECK_RUN_SCHEDULER()).

I will be glad to help you or explain any point that is not clear. Also, I can send you any other file you can need. As for me, I am investigating following David's indications to try to know the ultimate cause of the problem. I will keep you informed.

Thank you very much!

Regards

José Antonio Martínez

[joséantoniomart]

Hi Alice, David, (and everyone),

Some more details while I contine investigating:

1. I have added "_int_install_unexpected_isr()" in main, but it does not help (as far as I can see), nothing different appears on console. Maybe this is the expected result considering the following point:
2. I must clarify that the app does NOT crash. What is happening is that it stucks in _CHECK_RUN_SCHEDULER(). Specifically, when I pause the execution, it is always stopped at this point of dispatch.S file (function _sched_execute_scheduler_internal()) :

               // r1 is 0 -> empty

ASM_LABEL(no_one_to_run)

                // wait for interrupt

                cpsie i

                wfi

                cpsid i

                // TODO check r0, must be kernel data

                ldr r1, [r0, #KD_READY_Q_LIST]      // get first queue from ready list <<<<< EXECUTION ALWAYS STOPS HERE WHEN I PAUSE IT

                b find_noempty_que

I am reading some threads about this problem. I have already test changing task's stack size (from 700 to 7000), and I am trying to clarify if everything is OK with interrupts, priority levels, etc..

Any suggestions are, as always, welcome.

Thanks a lot.

Regards.

José Antonio Martínez

[joséantoniomart]

Hi Alice, David, (and everyone),

I am happy to say I think I have found the problem: reading the doc "Freescale MQX RTOS Introduction" I have realized that semaphores are used to synchronize tasks, and events can be used to synchronize tasks and also a ISR with a task. As you can see in my code, I have only one task, and I am using the semaphore to synchronize the task (there is a wait in its main loop) to the ISR that is called at the end of a SPI transaction. I have replaced the semaphore with an event, an everything seems to work correctly by now. I am going to do some tests more, but I would want  to express my gratitude for all your help and ideas.

Thank you very much.

Regards,

José Antonio Martínez.

[davidtietz]

Hi Jose,

Alice brings up a good point. What exactly happens when your program crashes?

I have not tried any of this in KDS yet so that is my disclaimer. If you install _int_install_unexpected_isr(); in your main routine and if you are getting a hard fault or such during your crash you will end up in the unexpected isr handler. From there you can use this tutorial to trace back to the problem.

Debugging ARM Cortex-M Hard Faults with GDB Custom Command | MCU on Eclipse

This is of course assuming this is what happens. I am really just guessing.

[joséantoniomart]

Hi David,

Thank you very much for your response and your clear explanations. I was not aware of this issue. I am going to test it now and will keep you inform about the results.

Thanks  a lot for your help.

Regards,

José Antonio Martínez.

[davidtietz]

Jose,

There are also some useful registers to look at when the crash occurs.

Take a look at the SystemControl registers (SCB). Some of them can tell you of issues that are causing problems.

[joséantoniomart]

Hi Alice,

I think you are referring to me (when you say "Hello David" by error) isn't it?

Well, these are the details of our environment:

• HW: evaluation board FRDM-KL27Z
• IDE: KDS 3.0.0 (running on Windows 10)
• KSDK 1.2.0: so we use SPI drivers of MQX for KSDK. Specifically, our board act as a master, so we use "fsl_spi_master_driver.c"

In response to your other questions:

• The project works normally if we don't use semaphores, but then, its functionallity changes. We are now investigating if it would work changin semaphores to use lwevents. I will keep you informed. By the way, I have seen this thread: _lwsem_post problem on 32Bit SPI slave ISR It seems a similar problem, and for that case there was no solution reported. However it seems that lwevents worked where semaphores don't.
• I am goint to send you two things (I am preparing them now), as you requested:
  • Screenshot when the program crashes.
  • A zipped file with my project.

     However, note that I can't see anything weird in the IDE. Is a said in my last post I don't know a way to know "what is happening" or the "reason to failure". Everything I know, is that execution doesn't reach code after lwsempost() (for example if I put a breakpoint, or event if a toggle a LED). The IDE says that program is running, but I have loose control over it!.

Thank you very much.

Regards,

José Antonio Martínez