MPC5744P ADC stop working in the field

[alimeraj]

Hi,

I hope someone can help me on this issue urgently.

We have a product in the field where the ADC suddenly stop working and the value of ADC values are frozen. This issue only occurs in the field and we cannot replicate in test lab. I have consulted the processor Manual but couldn't find any source that could stop the ADC conversion without any warning. The ADC start working properly after the CPU is hard reset.

We are also monitoring internal CPU temperature which shows the same behavior.

Please could you explain what could be the source of this behavior?. is it possible to just restart the ADC module without hard rest to bring the ADC module back to life?.

Thanks,

Kind Regards,

Ali Meraj

[petervlna]

Hi Ali,

From the very brief description you have provided it can be anything from top level tasks handling trough  permission on buses down to the unserved ADC trigger or clock glitch.

Maybe it would be a good start to tell us how do you use ADC (CTu or CPU mode?)

How do you trigger conversion

How do you serve converted data

because it really looks like unserved corner case of your SW design.

Is your SW running correctly on NXP EVB?

regards,

Peter

[alimeraj]

Hi Peter,

thanks for reply.

I am using SCAN mode @80Mhz clock rate.

I start the conversion in the startup task which only execute once.

I collect the converted data from CDATA reg whenever it is need by Application.

we have our own control board which we are using for testing.

This issue occurs when high voltage supply is turned on i.e.900Vdc. so it might be related to EMI but the ADC never recovers and counts reports 1373 counts constantly. This happens only in field as we cant replicate this issue in our HV lab.

below is the code for ADC setup and reading

Thanks,

Ali

ADC_0.MCR.B.OWREN = 1U; /* Enable result overwriting */

ADC_0.MCR.B.CTUEN = 0U; /* CTU mode not enable */

ADC_0.MCR.B.MODE = CDD_ADC_CONTINIOUS_SCAN_CONVERSION;

ADC_0.MCR.B.ADCLKSEL = 1U; /* 80 Mhz mode */

ADC_0.MCR.B.PWDN = 0U; /* Enable ADC0 (release it from power-down mode) */

/*

NSTART Should be the last register bit set

*/

ADC_0.MCR.B.NSTART = 1; /* Start normal mode */    

READING of ADC converted DATA

                                                                      

[petervlna]

ok, since it is only when you turn 900V on, them I suspect it can be EMI issue.

What are the ADC voltages measurement when this happens? Are there any spikes on ADC ref or ADC_HV?

What about measured line voltages?

regards,

Peter

[alimeraj]

Hi Peter,

when this happens the counts are frozen to 1373 counts and never recover so we have to do power cycle. it mostly happens on HV power up but sometimes its random and it happens during the functional operation.

We have tested the boards in HV lab and seen spikes on ADC_HV and ADC_ref when the HV is turned on but ADC doesn't stop working it only happens in field where I am assuming the EMI levels are high. The ADC_HV and ADC_ref voltage settles downs quite quickly after HV is on.

Do you know any particular state that ADC takes itself in case of power failure or noise? does restarting ADC will help?

Kind Regards,

Ali

[petervlna]

Well, all this is hypothetical.

If the ADC power drops or spikes out of LVD/HVD thresholds then the device will perform notification and reset if enabled.

It can cause decalibration of ADC capacitors, or stuck on logic level, or many other things.

What about enabling and disabling ADC?

What is the Status of the ADC when it is stuck?

regards,

Peter

[alimeraj]

Hi Peter,

The main problem is that I cant replicate the fault in the LAB so I am not sure what will be the status of the ADC status register in the field.

I have prepared a test software to perform ADC reset if the ADC values are stuck so I am hoping that might get around this issue.

what i have tried is taking the ADC supply below 3.3V but then MCU resets so that is not what is causing the issue as we don't see the CPU reset in the field. I have also tried changing the ADC ref voltage but this doesn't stop the ADC module from working.

is there anyway I can force any particular issue on ADC hardware so that it gets stuck for example as you mentioned about loss of ADC calibration?

I will check the FCCU for the fault reaction in case of voltage issue which i think is not configured to perform reset.

Thanks,

Ali

[petervlna]

Hmm,

The most important is to see if the ADC is still converting but the result registers are not updated.

Test SW should check the ADC status and store it when ADC conversion is stuck.

As it is in scan mode, I guess only loss of clock can cause this condition.

But first lets see if the ADC is still converting when the issue occurs.

regards,

Peter

[alimeraj]

Hi Peter,

I will try to save the ADC status in the flash when this fault occurs.

if it is a clock issue then do you think that taking ADC into power down and restart will resume the clock?

Regards,

Ali

[petervlna]

Hard to say,

I never tried it this way.

But if the clock is lost you wont be able to read anything from registers on PBRIDGE where also ADC is.

But as you are able to read results, I will suspect the ADC conversion is not in scan mode and it is IDLE.

[alimeraj]

Hi Peter,

I hope you are doing good.

I have an update

The ADC module restart didn't solve the issue as the ADC values remain frozen even after trying to restart ADC module.

I have register snapshots which give more insight of this issue.

When this issue happens the ADC module is working normal and ADC Status is toggling between state  4 and 6 but the CDATA reg is frozen to a fix value

CDR Registers

so they all lock up to same value of 1171 decimal (0x493 Hex) and the Valid bit remains high

The CPU and BandGap shows the same behavior.

Could you please help me on understanding why the CDR reg is locking up?.

Thanks,

Kind Regards,

Ali Meraj

[petervlna]

Hi,

The ADC module restart didn't solve the issue as the ADC values remain frozen even after trying to restart ADC module.

How did you restart ADC module? Did you turn it of in PCTL register and then mode transition to RUNx and back?

Could you please help me on understanding why the CDR reg is locking up?.

It looks like the ADC conversion finished and new one is not started.

(So the ADC is not stuck but staying in defined state and waiting for new conversion trigger, either SW or CTU)

This I do not know as I have only limited information on how do you use ADC.

regards,

Peter

PS: what happen if you start ADC conversion manually via debugger after it is stuck? Does it continue converting?

[alimeraj]

Hi Peter,

I took the ADC to power down by setting power down bit to 1 then wait till the ADC status is changed to power down then restart it. i have attached the code of ADC restart.

I am using SCAN mode for normal conversion not CTU mode.

so at the startup of the code the ADC is started once by setting NSTART bit to 1 and then just read the data from CDATA

i have attached the code of ADC config and ADC read every 1ms

I hope this explain how the ADC is setup.

why would the conversion stop? this is random issue as it mostly happens when the high voltage (800V) supply is turned on. This issue is EMI related but its locking up the CDATA reg. how would i trigger the conversion again as I think in SCAN mode trigger is not required.

the valid bit should be cleared when the data is extracted from CDR data but it doesn't clear when the CPU is in this state.

you mention about PCTL register could you please provide more information on restarting the ADC using PCTL reg.

Please let me know if you need more information.

Regards,

Ali Meraj

[petervlna]

Hi,

PCTLx will cut off the clocks to ADC.

Refer to reference manual:

59.3.52   ADC_0 Peripheral Control Register (MC_ME_PCTL237)

But I have just tested it and it will  not erase the ADC registers. (they are on PBRIDGE) so after restoring the clocks to ADC the registers value it the same.

I guess only way to reset the ADC registers is to trigger reset of micro.

What happen if you shield the ECU from EMI? Does the issue remain?

regards,

Peter

[alimeraj]

Hi Peter,

Thanks for testing.

The problem is that we can reset the Micro as these ADC fault doesnt effect the control but it limits the product functionality.

The products are already with the customer we can modify anything on those but we are doing internal investigation on how to minimize EMI effect on Mirco so that we revise our product. The interim fix would be to restart the ADC somehow to get them back online when the fault occurs on the existing products.

so I think you are suggesting the only solution would be to reset the Mirco and there is no other way to bring the ADC back to life?

would you suggest to first take the ADC into power down then disable the clock and then restart the clock and then restart the ADC do you think that would restart the ADC conversion again?

Thank you very much Peter for your help.

Kind Regards,

Ali Meraj

[petervlna]

Hi,

The problem is that we can reset the Micro as these ADC fault doesnt effect the control but it limits the product functionality.

You can or can't? Because there is only way how to reset ADC module and PBRIDGE(where the registers are located) and that is system reset. This is also outcome from my discussion with design.

Furthermore, if you disturbing micro with EMI (900V switching) I assume that this is critical also for micro as whole concept.

How you can guarantee the safety functions with such heavy disturbance which stops ADC module.

Other modules can be affected as well, modules like FCCU, STCU, or similar which are not in use all the time micro is running.

You just see only ADC issue as it is all the time running.

In case you switch on high voltage in moment micro is processing some IVOR, or FCCU routine, it can have critical impact on system.

These are just a few assumption from my side.

regards,

Peter

[alimeraj]

Hi Peter,

Sorry I wanted to mention that we can't restart.

we have done some high resolution scope measurements and we are finding that when HV is turned ON the ADC input goes above 5V from couple of micro seconds before it is settled again. we are looking into this issue closely in the lab now and the Electronics team is working to mitigate this issue.

I agree with all of your points.

I will update you with some scope plots once I have them from the Lab and probably provide you more information of our findings.

Thank you very much Peter.

Regards,

Ali Meraj

[alimeraj]

Ok Peter, thanks. I will log the ADC status and if i have an update I will get back to you.

thanks,

Ali

[alimeraj]

Hi,

Could someone help me on this issue please?

Regards,

Ali