Port interrupt preempting eDMA i2c transfer causes i2c to hang

[yorknh]

I have 3 issues...

1. I have a pin based interrupt whose ISR takes 200us to execute that happens every 2ms.

2. That pin based interrupt is preempting an edma I2C transfer in progress.

3. The I2C bus will eventually hang.

I have an I2C device that I'm polling 1/s and it takes roughly 6ms to complete the transfer. This will run indefinitely if I remove the source of the pin based interrupt. If I enable that source the I2C bus will hang with both SDA and SDL being held low, seemingly by the micro.

The processor is a K64 and I'm running FreeRTOS. I've attempted to make it so that the DMA0 channel (which I'm using for I2C) has a higher priority than the pin-based interrupt by making the calls: 

NVIC_SetPriority(GPIO_B_IRQN, configLIBRARY_LOWEST_INTERRUPT_PRIORITY);  NVIC_SetPriority(DMA0_IRQn, configLIBRARY_LOWEST_INTERRUPT_PRIORITY - 4);

This doesn't seem to have any effect as the pin ISR will get called 2-3 times during the I2C transfer. So I guess my 2 questions are how can I stop the pin based ISR from interrupting the eDMA handler, and why is the I2C bus hanging?

I've attached a scope capture of the last I2C transaction as the bus hangs.

[yorknh]

By switching away from eDMA and the SDK provided drivers to using I2C_RTOS_Transfer() the problem goes away and since there was negligible, if any, negative impacts to overall system performance, I'm just going to stick with that implementation. So I went from not being able to poll a single device for more than 60s until bus lockup, to polling a total of 12 devices spread across 3 I2C busses with no lockups.

This is a solution to my problem, but doesn't solve whatever problem there seems to be in the I2C/eDMA driver approach.

[myke_predko]

@yorknh 

That's great you found a solution.  

If you ever get some time, I'd be curious to see what happens when you take that 3rd party protocol stack our of the interrupt handler and initiate it thorugh a FreeRTOS notification or queue message.  

To be honest, I'd be very surprised if the issue was in the driver rather than the code running 10% of the system cycles in an interrupt handler.  

[yorknh]

The thing is that 200us interrupt still executes. It was changing from the dma based driver that allowed things to work.

[ErichStyger]

Hi @yorknh ,

I'm not really clear what you have in your scope picture, but anyway:

You do this:

NVIC_SetPriority(GPIO_B_IRQN, configLIBRARY_LOWEST_INTERRUPT_PRIORITY);
NVIC_SetPriority(DMA0_IRQn, configLIBRARY_LOWEST_INTERRUPT_PRIORITY - 4);

I assume your configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY is 4 or numerically higher.

You only need to  set the priority to configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY or numerically higher if you are using FreeRTOS API calls such an interrupt handler. In your case you are allowed to used FreeRTOS in GPIO_B_IRQn, but *not* in DMA0_IRQn. I assume you know about this, and this is the case, see my article series in my other reply.

Now keep in mind that this is about interrupts, not the DMA transfer itself. So depending on what you do in your DMA transfer, you need to make sure it is re-entrant.

Or in other words: the DMA (data)transfer on the bus is de-coupled from the core running the interrupts. with your above priority setting you give the DMA IRQ a priority to preempt the GPIO interrupt, as you give it a numerically lower value compared to the GPIO interrupt. But this is only for the interrupt, the DMA transfer still happens in the background.

The other thing you should know about the ARM interrupt system is the following: https://mcuoneclipse.com/2015/10/16/nvic-disabling-interrupts-on-arm-cortex-m-and-the-need-for-a-mem... , just in case you are not aware: this is an ARM Cortex-M silicon issue, and you need to have memory barriers in place as workaround.

I hope this helps,

Erich

[yorknh]

The scope picture just shows an I2C transaction that was in process and was not properly terminated because SDA and SCL are being held low.

configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY  is currently set to 2 though I can't say that was chosen for any particular reason. I'll admit when to use/not use NVIC_SetPriority has been a little unclear. I'll reread the posted articles.

I'm using the fsl_i2c_edma driver that comes with SDK 2.7.0 I presume this is what would have to be reentrant.

I have been including __DSB() as the last line in interrupt handlers on other projects. Should I be adding __ISB() as well?

Thanks

[ErichStyger]

The __DSB() is what is needed to work-around the ARM errata #838869, as it affects data accesses.

[myke_predko]

@yorknh 

Can you explain why your pin ISR takes 200us to execute?  That's really quite a long time and I suspect that while it is executing something that is needed for the combined I2C/EDMA operation is timing out before the IRQ completes.  

Have you tried something like reducing the ISR's code that executes in your ISR to something like "pinInterruptFlag = true;" (as well as the hardware reset) to see if that eliminates the problem?  If it does, then we can work at architecting your application so the required execution that takes place happens in a task rather than an ISR.  

[yorknh]

I understand that 200us is a very long time for an ISR. For the sake of argument it is non-negotiable. It doesn't make use of anything DMA or I2C related. It's basically a bunch of Flexbus transactions and logic. It's part of a protocol stack provided by a 3rd party.

That said I thought I should be able to set the priority of the eDMA irq to be higher than that of the pin interrupt. Is that not the case? Also what could the eDMA/i2c hardware be reliant on that could cause it to fail in a way that it won't release the bus?

I should also add that the micro spends >90% of the time in the idle thread.

[myke_predko]

@yorknh 

I did a bit of looking around and I think the issue is that you are not changing the preempt priority in your "NVIC_SetPriority" method calls - see CMSIS-Core support for Cortex-M processor-based devices - Interrupts and Exceptions (NVIC) 

You might also want to read through A Practical guide to ARM Cortex-M Exception Handling as well as Cutting Through the Confusion with ARM Cortex-M Interrupt Priorities I would have thought @ErichStyger would have written something on this but I couldn't find anything - Erich?

Regardless, I don't think you're really understanding what's going on here with the statement that the "micro spends >90% of the time in the idle thread" as from your 'scope picture it seems pretty clear to me that the problem is when the pin interrupt request is being serviced in the other 10% of the time.   

I think the best solution to this issue is to create a task that executes 3rd party protocol stack when it receives a poke from the pin IRQ handler which can be a four byte message (see xQueueSendFromISR()) or a notification (see xTaskNotifyFromISR()).  The IRQ handler just executes the message send or notify.  

If you're not comfortable using FreeRTOS messages or notifications, then look back at by original post - have the IRQ set a flag and have a task (containing the 3rd party protocol stack) spinning waiting for the flag to be set - if your application is going to spend most of it's time in idle then you might as well have it running a polling loop for you rather than having it run a polling loop for the OS.  

[yorknh]

Hi Myke,

With regards to my 90% comment, it was just to indicate that there is effectively nothing else going on in the system that would potentially be contributing to the problem. In other words the only thing the system is doing is servicing the gpio interrupt and the DMA I2C transfers.

I won't say I'm expert level with regards to messaging but I have used it pretty extensively, so I know what you are driving at but it would require a deep dive into the 3rd party code to implement what you suggest. The system consists of an FPGA and the K64. Part of the stack is implemented on an embedded micro in the FPGA and part on the K64. The two communicate with each other via shared memory that is accessed via the flexbus. The portion running on the FPGA indicates that an exchange between the two processors needs to occur via the gpio interrupt. Even that is a simplification of what is going on. It would be easy enough to defer that to a task if I knew exactly what the fpga side is expecting to be complete on return from the ISR. The problem is I don't, and it is a nontrivial undertaking to be sure I don't introduce some hard to track down failure by messing with the inter-processor communication.

I'll be sure to read the docs you listed.

Thanks for taking the time to help.

[myke_predko]

Hi @yorknh 

You were very clear in your original post that, other than the DMA transfers, nothing was happening in the system except when the pin interrupt handler was executing.  I've found in the past when somebody says something like the "micro spends >90% of the time in the idle thread" they don't have a very good grasp on how a multi-tasker works.  

I think @ErichStyger had a good place for you to start with checking "configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY".  He's always an amazing resource.  

Good luck and keep us updated as to your progress.  

[ErichStyger]

I would have thought @ErichS would have written something on this but I couldn't find anything - Erich?

I have a 3-part series here:

• ARM Cortex-M, Interrupts and FreeRTOS: Part 1
• ARM Cortex-M, Interrupts and FreeRTOS: Part 2
• ARM Cortex-M, Interrupts and FreeRTOS: Part 3

I hope this helps,

Erich