topic Re: IMX6 Solo intermittent boot-up failures in i.MX Processors

IMX6 Solo intermittent boot-up failures

dvona — Wed, 07 Jan 2015 22:03:41 GMT

BACKGROUND:

We have an existing system (custom designed), that exhibits a bootup failure. The nature of the failure is that the processor appears to not recognize our boot device properly on first time power up, even though our BOOT_MODE and Configuration pins are set properly.

Some notes:

When bootup fails, we notice immediate activity on USB. We believe this indicates that the wrong boot source has been latched. Our system is designed to boot from an eMMC card, but it seems to move immediately to USB boot.
With power on, executing a POR_B causes a proper boot every time. SO we surmise the issue has to do with power sequencing or some other power on anomaly.

SOLUTION 1: Change from ANA_V2P5V to VSNVS for all boot config pullups (much earlier in sequence and away from strange ON-OFF-ON behavior as exhibited in the following figures)

(See Figure 1): ON-OFF-ON LDOs. Our Config Pins appear to pull up during this transition. Boot fails

(See Figure 2): Moved Config Pins to Much early in Boot (VSNVS) out of the ON-OFF-ON Area. Boots reliably

(See Figure 3): Capture showing that the VDDSOC/ARM_CAP ON-OFF-ON is internal, not a result of VDDSOC/ARM_IN.

Notes for Figures 1 - 3:

Documentation implies the boot config is latched at rising edge of POR_B (i.MX6Dual/6Quad Applications Processor Reference Manual Rev 2, 06/2014)
Currently boot resistors are strapped to ANA_V2PV which is safely stable before POR_B rising edge
When we change config resistors to VSNVS rail (80ms earlier than ANA_2P5V) we boot reliably
This seems to behave as if the boot config is being read/latched at this unstable VDDSOC ON/OFF/ON area

USB ACTIVITY: On Our failure boot-up case, we see USB_OTG doing something early BEFORE POR_B. eMMC never executes. If, with power still applied, you perform a second POR_B, then boot from eMMC always happens correctly

(See Figure 4): USB Activity before POR_B failure

(See Figure 5): Proper Boot with eMMC and USB activity as expected

Power-up rails/POR_B sequence:

(See Figure 6): Power-up sequence/POR_B

Fix #2 = We have found that reducing the capacitor on VDDSOC_CAP from 22uF to 10uF to 0uF seems to positively impact the boot (NOTE: A recent test showed one failure with a 10uF capacitor, so this is not seen as reliable). The VDDSOC_CAP signal appears to be the one that contributes to the issue (not VDDARM_CAP).We have tested with 22uF/10uF/0uF on the VDD

(See Figure 7 - VDDSOC_CAP with 22uF Cap Fails)

(See Figure 8 - VDDSOC_CAP with 10uF Cap normally boots, but has failed at least once)

(See Figure 9 - VDDSOC_CAP with 0uF Cap boots reliably)

NOTE: We are fully aware that the MAX capacitor value is 22uF including trace capacitance and have improved our design to be as close as mechanically possible to the _CAP pins. However, even with a 10uF + small trace length we are not booting reliably so we are not convinced this is all of the problem and do not feel this explains the strange on-off-on behavior.

Fix#3 = Burn eFUSES resolves intermittent boot

This will be done, but it does not solve our first time boot problem. And we also do not understand why the boot fuse method would not be effected in the same way as the config pins. We know Freescale recommends using the boot fuses in their documentation. We need to understand WHY this is more reliable.

Answers needed from Freescale:

Q1) What would cause the VDDSOC and VDDARM LDOs to exhibit the ON-OFF-ON at power-up? Is this a known issue or errata?

Q2) Is the ON-OFF-ON situation somehow causing the apparent early latching? And how do we mitigate it in such a way that we guarantee a reliable boot from the config pins?

Q3) Why does moving the config pins to a rail that powers up much earlier help the situation?

Q4) What is different about the latching of the configuration when using the eFUSES versus the pull-ups/pull-downs that it would not suffer the same problem?

In general, our goal is to fully understand the mechanism that is causing this boot failure inside the chip so that we can reliably implement our fix and KNOW that it is going to work.

Re: IMX6 Solo intermittent boot-up failures

igorpadykov — Thu, 08 Jan 2015 07:38:02 GMT

Hi Dan

you can try to prolong POR, say up to 0.8-1 sec. and check if that helps.

In i.MX6S there is no "ANA_V2P5V" signal, what it is this signal connected to ?

In general, SNVS (VDD_SNVS_CAP) power should be powered first and it should

be very quite and well filtered.

Any other power is provided later SNVS, while boot signals belong to SNVS power domain.

So SNVS will see unexpected current (voltage spike) drawn from boot resistors

(when "ANA_V2P5V" start-up), so SNVS power gitch detector can reject boot

if this is secure boot.

Best regards

igor

Re: IMX6 Solo intermittent boot-up failures

sohara — Thu, 22 Jan 2015 21:26:39 GMT

ANA_2P5V rail is generated by the PMIC "VGEN6_OUT", this rail comes up about 72ms after VSNVS and about 14mS before we de-assert POR_B.

We release POR_B about 14mS after ANA_2P5 rail, documentation states in multiple places that boot resisrors get latched at POR_B release.

Can you provide more insight about SNVS getting unexpected spike from ANA_V2P5 pull-resistors? Could it be possible for this glitch to get something in such a bad state that 1st POR_B does not work correctly but 2nd POR_B always works correctly?

Could our problem be related to the VDDSOC_CAP turning ON-OFF-ON? Why is this ON-OFf-ON happening?

Re: IMX6 Solo intermittent boot-up failures

igorpadykov — Fri, 23 Jan 2015 01:59:51 GMT

what PMIC are you using, in attached diagram VDDARM_IN,VDDSOC_IN

provided simultaneously, thus huge current soike may occur -

this may be reason for "turning ON-OFF-ON".

For i.MX6SDL usually MMPF0100 "F0" is used, it gives powering

VDDARM_IN,VDDSOC_IN in different time steps.

Re: IMX6 Solo intermittent boot-up failures

sohara — Fri, 23 Jan 2015 13:27:00 GMT

We are using MMPF0100NPEPR2, we cannot delay 0.8sec without PCB layout change, maximum POR_B delay is 32ms.

All supplies are stable before POR_B release. Something is in latch-up condition and 1st POR_B does not work, 2nd POR_B seems to always work.

We will try test to to separate VDDARM_IN and VDDSOC_IN although that still would not explain why 1st POR_B does not work.

Re: IMX6 Solo intermittent boot-up failures

igorpadykov — Fri, 23 Jan 2015 13:33:28 GMT

1st POR_B may not work due to

large power spikes, since board capacitors

start to charge. Second POR_B resets processor

when all capacitors are charged so there are no power

spikes.

Re: IMX6 Solo intermittent boot-up failures

sohara — Fri, 23 Jan 2015 15:19:57 GMT

1st POR_B is 14mS after this glitch area, all external capacators are fully charged confirmed by scope, you don't think that is enough time internally for some reason?

After glitch do we need to deassert/assert/deassert POR_B? (PMIC is not designed to do that,We cannot do that easily.)

Also we cannot re-program PMIC very easily, we need to program new PMIC and send board out for part change.

Re: IMX6 Solo intermittent boot-up failures

igorpadykov — Sat, 24 Jan 2015 00:51:58 GMT

could you try to prolong POR, say up to 0.8-1 sec. and check if that helps ?

It can be done manually since MMPF0100 RESETBMCU is open drain.

Re: IMX6 Solo intermittent boot-up failures

sohara — Sat, 24 Jan 2015 21:54:00 GMT

We performed this test, we disconnected PMIC reset and ran external reset, 1st POR_B de-assert at about 5 seconds delay still does not work, 2nd POR_B de-assert at second 5 seconds delay works/boots successfully. That's why we ask questions about what could get into such a bad state from VDDSOC_CAP ON/OFF/ON sequence? So bad that 1st simple POR_B de-assert does not correct. Seems some module seems to require POR_B deassert/assert/deassert to get out of the problem scenario.

Re: IMX6 Solo intermittent boot-up failures

igorpadykov — Sun, 25 Jan 2015 12:18:43 GMT

you said that :

"ANA_2P5V rail is generated by the PMIC "VGEN6_OUT""

VGEN6=2.8V on MMPF0100N part, what is real voltage on that rail ?

How connected VDDHIGH_IN, what MMPF0100 power is used for it ?

Also, are you using boot signals for other purposes (such as parallel NOR) as in

Figure 2-1. Boot configuration for development mode IMX6DQ6SDLHDG

One can easily to check if boot signals are sampled correctly by

stopping processor, attaching jtag and checking SRC_SBMR1,SRC_SBMR2

IMX6SDLRM 60.7.2 SRC Boot Mode Register 1 (SRC_SBMR1)

Re: IMX6 Solo intermittent boot-up failures

sohara — Mon, 26 Jan 2015 14:05:50 GMT

VGEN6 is programmed for 2.5V OUT, we do not use default (acceptable range = 1.8 to 3.3V)

Measures ~2.5V

VIN3 = about 3.8 to 4.1V

VGEN3=VDDHIGH_IN1/VDDHIGH_IN2 (connected together)= 2.8V (22uF, 220nF, 220nF) - NO connections to anything else

VDDHIGH_CAP1/VDDHIGH_CAP2(connected together)=program out = 2.8V (22uF, 220nF, 220nF) - NO connections to anything else

Boot signals are all fixed loaded/No loaded for eMMC boot

We cannot check boot signal sampling in failure case, failure case is completely dead behavior

Re: IMX6 Solo intermittent boot-up failures

igorpadykov — Mon, 26 Jan 2015 14:34:04 GMT

so VDDHIGH_IN=VDDHIGH_CAP=2.8V, have you connected them together?

Having VDDHIGH_CAP= 2.8V" violates Table 8. Operating Ranges IMX6SDLCEC :

NVCC_LVDS_2P5, NVCC_MIPI max.= 2.75V

as VDDHIGH_CAP powers NVCC_LVDS_2P5, NVCC_MIPI

VDDHIGH_CAP=VDDHIGH_VPH=GEN_2V5 on SPF-27417

Sabre SDP schematic

"dead behavior" means jtag can not attach to

processor ? Is 24MHz working in that case ?

Re: IMX6 Solo intermittent boot-up failures

sohara — Mon, 26 Jan 2015 16:16:44 GMT

--so VDDHIGH_IN=VDDHIGH_CAP=2.8V, have you connected them together?
NO, VDDHIGH_IN1 connected to VDDHIGH_IN2 (measured = 3.0V)
VDDHIGH_CAP1 connected to VDDHIGH_CAP2 (measured = 2.55V, also connects to NVCC_LVDS2P5 ball and PCIE_VPH ball)

Having VDDHIGH_CAP= 2.8V" violates Table 8. Operating Ranges IMX6SDLCEC :
I made a mistake, we actually have 2.55V on VDDHIGH_CAP
We are using MCIMX6S5DVM10AB
I don't believe we have any violation.

--NVCC_LVDS_2P5, NVCC_MIPI max.= 2.75V
correct PCIE_VPH and NVCC_LVDS2P5 connected to VDDHIGH_CAP which is 2.55V

--"dead behavior" means jtag can not attach to processor ?

-- Is 24MHz working in that case ?
Correct, 24 MHz is working, at glitch time processor is *immediately* stuck trying to boot from USB-OTG for some wrong reason

Re: IMX6 Solo intermittent boot-up failures

sohara — Mon, 26 Jan 2015 19:14:57 GMT

##what PMIC are you using, in attached diagram VDDARM_IN,VDDSOC_IN

##provided simultaneously, thus huge current soike may occur -

##this may be reason for "turning ON-OFF-ON".

##For i.MX6SDL usually MMPF0100 "F0" is used, it gives powering

##VDDARM_IN,VDDSOC_IN in different time steps.

We installed new programmed PMIC with staggered ARM/SOC on time, "F0":

VDD_ARM on 1ms before VDD_SOC

same result, same boot failure

NOTE: VDDARM_CAP and VDD_SOC still ON/OFF/ON same time (no 1ms stagger) somthing inside IMX still produce simultaneous VDDARM_CAP and VDDSOC_CAP ON/OFF/ON.

Re: IMX6 Solo intermittent boot-up failures

igorpadykov — Tue, 27 Jan 2015 02:51:47 GMT

so you can not attach jtag and check SBMR1,2

registers even with 24MHz working, correct ?

Do you see any activity on eMMC, does it tries to read from

SD port or processor immediately goes to Serial Downloader mode ?

How many boards have such behaviour: one, all ?

Re: IMX6 Solo intermittent boot-up failures

sohara — Tue, 27 Jan 2015 14:50:04 GMT

We will try JTAG probing and report results soon.

In failure case we see absolutely NO activity to eMMC, 1st and only activity is USB_OTG.

(If we assert/deassert POR_B a second time manually eMMC boot works correctly)

Last build was 17 out of 25 boards have this problem.

NOTE we had a similar high failure on early prototype boards about 6 months ago.

Re: IMX6 Solo intermittent boot-up failures

igorpadykov — Tue, 27 Jan 2015 15:20:53 GMT

when you are saying "first time boot problem" - this means

boot from unpowered state, correct ? If this is so could you

recheck recommended capacitors values on *_CAP, NVCC_PLL_OUT

nodes given in latest Design Check List,

HW Design Checking List for i.Mx6DQSDL Rev2.8.xlsx

they were decreased (to 10uF) compared with Sabre schematic

https://community.freescale.com/docs/DOC-93819

Re: IMX6 Solo intermittent boot-up failures

sohara — Tue, 27 Jan 2015 15:26:02 GMT

FAILURE Case where board does not boot:

mem read 0x20D8004 (SBMR1)

0x40000000

mem read 0x20D801C (SBMR2)

0x22000001

WORKING Case where board does boot:

mem read 0x20D8004 (SBMR1)

0x40005062

mem read 0x20D801C (SBMR2)

0x22000001

This seems to confrim our suspicion that the boot config resistors were getting latched wrongly at the unexplainable ON/OFF/ON internal IMX rails. And they are not getting re-latched at the good POR_B deassert which is much later in time when the supplies are all stable.

Re: IMX6 Solo intermittent boot-up failures

sohara — Tue, 27 Jan 2015 15:36:41 GMT

We already included this information in webcase, see above:

(See Figure 7 - VDDSOC_CAP with 22uF Cap Fails)

(See Figure 8 - VDDSOC_CAP with 10uF Cap normally boots, but still fails at lower frequency)

(See Figure 9 - VDDSOC_CAP with 0uF Cap boots reliably on 2 out of 2 boards tried)

We have change all _CAP locations from 22uF to 10uf and double checked Freescale latest Design Guidelines, but this does not resolve boot-up failure issue.

0uF on VDDSOC_CAP does resolve issue on 2 out of 2 boards, BUT we do *not* accept 0uF as a resolution to the problem, This is a medical product, we need to understand root-cause of this failure and have 100% certainty in final resolution before production release.

Re: IMX6 Solo intermittent boot-up failures

igorpadykov — Tue, 27 Jan 2015 16:34:21 GMT

how connected boot pins, what resistor values are used,

is smth else connected to EIM_DA8-15 pins (parallel NOR for example ?)

when you are saying :

"Change from ANA_V2P5V to VSNVS for all boot config pullups"

what pins do you mean by "all boot config pullups" , only BOOT_MODE0,1 ?

boot pins EIM_DA8-15 BOOT_CFG2[7:0] have power domain NVCC_EIM, they

are not latched correctly according to your SBMR1 reading

0x40000000 vs 0x40005062

could you check NVCC_EIM power rail by oscilloscope and signals

at these boot pins (EIM_DA8-15) as close to processor as possible.

Additionally please check processor 24MHz and 32 KHz clock - are they stable

and good shape ?

Also is some externally powered device can be connected to board when you

powers board first time ? May be board connected to PC with

UART or USB, HDMI monitor, SATA HDD, camera ? Could you

disconnect all and check ? If there is additional case "ground", remove it.