LPC11e6x + FreeRTOS + optimizer = hard fault on _some_ chips

[chrispflieger]

I've never seen something like this. Ever.

This software has been in production for some time, but we've recently had a fallout of a considerable percentage. I've spent the last day trying to figure it out, and I've been able to prove that if I turn off optimization (going to -O0 instead of -Os), it runs. Otherwise, it near immediately hard faults.

I've got a batch of identical production run devices, and the one that fail all have similar coding:

LPC11E67J
23 13
ZSD20
121A

I'm running FreeRTOS 10.0.1

I have one task that does some serious DSP on audio, so I can't live with the RTOS running slow. In fact, to get it to run, I had to disable that task. If I enable it, it hits a hard fault.

[chrispflieger]

I had a few more failures in the last production run, but my CM is telling me they don't have anymore parts with these lot numbers.

So, at this point, I don't have the time to chase this anymore.

[diego_charles]

Hi @chrispflieger 

I just wanted to provide follow up.

Many thanks you for update and your efforts to follow my previous recommendation.

You may also check with your distributor if you are getting more failures with  more lots .However I am not sure how your distributor may work. As far as I know some distributors use  to request swap tests before checking for any possible quality incident. I am not implying that your chips have a quality issue, but just mentioning this to narrow down the issue.

Is there something else where I can help you with?

I am sorry for the delay.

Diego.

[chrispflieger]

We had another production run with chips made in Q4 of 2020, very low fallout on that run.

There's another run today, so I'll be monitoring it.

Right now, I'm considering this a one-off event. Which hopefully it is because I've got four more products in the pipeline based on this chip.

[chrispflieger]

A bit more about that - I can turn on the SPI, so long as I don't actually turn on the DMA that drives it. I have to comment out the last two lines of the init function.

Even if my DMA ISR just dumps the data in the trash, it hard faults - but only on some MCUs. On most boards this is working rock solid.

[chrispflieger]

I upgraded to FreeRTOS v10.4.3 - it runs with full optimization. But, I have to disable the I2S (via SPI & DMA) to prevent a hard fault.

[diego_charles]

Hi @chrispflieger 

That's an interesting problem. 

If I understand you properly , the issue seems to be parts with the part marking that you gave. 

Are you able to provide some demo code were the issue is replicated?

In you FreeRTOS tasks, could you provide more details on how  the SPI and DMA peripherals accessed and configured ? Do you have some kind of synchronization technique for  this?

Thank you in advance.

All the best, 

Diego.

Edith: re-phrasing

[chrispflieger]

I've attached the I2S driver. On a bad board, I need to disable the last line, which then keeps it from running.

I still can't figure out why an older version of FreeRTOS wouldn't run, but a newer one would. I ran it with minimal tasks, and with no devices, or off chip interfaces... I even tried to run code from another project (we use that chip and code base in a lot of production devices), it hard faulted there as well.

Even with I2S turned off, I still think there's occasional hard faults - maybe every ten or twenty minutes. I need to investigate that more.

Debugging hard faults on a CM0 is a real pain, but I can provide whatever data you want.

Note, I do see some problems on ZSD20 331A parts, but never on any made in 2019 or 2018.

[diego_charles]

Hi @chrispflieger 

Thank you for your code and additional information.

Regarding hardware , Are you able to perform the following swap test and let me know your results?

1 Solder  a chip from  2020 into a  2019 or 2018 board.

2 Solder  a chip from  2018 or 2019 into a  2020 board .

Many thanks

Diego.

[chrispflieger]

I failed on the first swap - I'll try another one. My solder rework skills aren't the best.

[diego_charles]

Hi @chrispflieger 

Thank you for your update and help for doing this procedure.

Diego