i.MX6 SabreSD PCI is broken

[Tarek]

Hi All,

I was writing PCI driver on the SabreSD to communicate with Altera FPGA over PCI.

I think I have damaged the PCI port on the chip after unplugging and plugging the FPGA board again. I was very careful to power down both boards before plugging / unplugging but looks like that is not enough.

Now I still can boot the SabreSD board connect UART console and SSH to it but the PCI is not working. I tried a reference PCI Network card and it’s not working.

I tried to investigate the problem and this is my findings:

• From HW side I can see i.MX6 is sending PCI reference clock going but it is not sending training data on the PCI-Tx.
• The software does not show any pci device and dmsg shows:

imx6q-pcie 1ffc000.pcie: phy link never came up

imx6q-pcie 1ffc000.pcie: Failed to bring link up!

imx6q-pcie 1ffc000.pcie: failed to initialize host

imx6q-pcie: probe of 1ffc000.pcie failed with error -22

• Only 2 CPU’s  are running !

# cat /proc/cpu/alignment

User:           0

System:         7

Skipped:        0

Half:           0

Word:           0

DWord:          0

Multi:          7

User faults:    2 (fixup)

root@imx6qsabresd:/sys/devices/system/cpu#

root@imx6qsabresd:/sys/devices/system/cpu# cat /proc/cpuinfo

processor       : 0

model name      : ARMv7 Processor rev 10 (v7l)

BogoMIPS        : 3.00

Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpd32

CPU implementer : 0x41

CPU architecture: 7

CPU variant     : 0x2

CPU part        : 0xc09

CPU revision    : 10

processor       : 1

model name      : ARMv7 Processor rev 10 (v7l)

BogoMIPS        : 3.00

Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpd32

CPU implementer : 0x41

CPU architecture: 7

CPU variant     : 0x2

CPU part        : 0xc09

CPU revision    : 10

Hardware        : Freescale i.MX6 Quad/DualLite (Device Tree)

Revision        : 0000

Serial          : 0000000000000000

Is it possible that I have accidentally disabled the clock “clock gate” some blocks in the Chip? How do I enable it again?

root@imx6qsabresd:/unit_tests# ./dump-clocks.sh

clock                    parent                      flags en_cnt pre_cnt      rate

ckil                        ---                   00000030      0      0      32768

ckih1                       ---                   00000030      0      0          0

osc                         ---                   00000030      6      6   24000000

pll1_bypass_src          osc                      000002a2      1      1   24000000

pll1                     pll1_bypass_src          00000041      1      1  996000000

pll1_bypass              pll1                     000002a6      1      1  996000000

pll1_sys                 pll1_bypass              00000024      1      1  996000000

pll1_sw                  pll1_sys                 000002a0      1      1  996000000

arm                      pll1_sw                  00000004      2      2  996000000

twd                      arm                      00000024      1      1  498000000

pll2_bypass_src          osc                      000002a2      1      1   24000000

pll2                     pll2_bypass_src          00000041      1      1  528000000

pll2_bypass              pll2                     000002a6      1      1  528000000

pll2_bus                 pll2_bypass              00000024      2      2  528000000

pll2_pfd0_352m           pll2_bus                 00000000      0      0  452571428

ldb_di0_sel              pll2_pfd0_352m           000002a4      0      0  452571428

ldb_di0_div_3_5          ldb_di0_sel              00000024      0      0  129306122

ldb_di0_div_7            ldb_di0_sel              00000024      0      0   64653061

ldb_di0_div_sel          ldb_di0_div_7            000002a6      0      0   64653061

ldb_di0                  ldb_di0_div_sel          00000005      0      0   64653061

ldb_di1_sel              pll2_pfd0_352m           000002a4      0      0  452571428

ldb_di1_div_3_5          ldb_di1_sel              00000024      0      0  129306122

ldb_di1_div_7            ldb_di1_sel              00000024      0      0   64653061

ldb_di1_div_sel          ldb_di1_div_7            000002a6      0      0   64653061

ldb_di1                  ldb_di1_div_sel          00000005      0      0   64653061

ipu2_di1_sel             ldb_di1                  000002a6      0      0   64653061

ipu2_di1                 ipu2_di1_sel             00000005      0      0   64653061

ipu2_pclk1_sel           ipu2_di1                 00000000      0      0   64653061

ipu2_pclk1_div           ipu2_pclk1_sel           00000004      0      0   64653061

ipu2_pclk_1              ipu2_pclk1_div           00000004      0      0   64653061

pll2_pfd1_594m           pll2_bus                 00000000      0      0  594000000

gpu3d_shader_sel         pll2_pfd1_594m           000002a2      0      0  594000000

gpu3d_shader             gpu3d_shader_sel         00000025      0      0  594000000

pll2_pfd2_396m           pll2_bus                 00000000      1      1  396000000

pll2_198m                pll2_pfd2_396m           00000024      0      0  198000000

step                     pll2_pfd2_396m           000002a2      0      0  396000000

hsi_tx_sel               pll2_pfd2_396m           000002a2      1      1  396000000

hsi_tx_podf              hsi_tx_sel               00000025      1      1  198000000

hsi_tx                   hsi_tx_podf              00000005      1      1  198000000

usdhc1_sel               pll2_pfd2_396m           00000000      0      0  396000000

usdhc1_podf              usdhc1_sel               00000025      0      0  198000000

usdhc1                   usdhc1_podf              00000005      0      0  198000000

usdhc2_sel               pll2_pfd2_396m           00000000      0      0  396000000

usdhc2_podf              usdhc2_sel               00000025      0      0  198000000

usdhc2                   usdhc2_podf              00000005      0      0  198000000

usdhc3_sel               pll2_pfd2_396m           00000000      0      0  396000000

usdhc3_podf              usdhc3_sel               00000025      0      0  198000000

usdhc3                   usdhc3_podf              00000005      0      0  198000000

gpmi_apb                 usdhc3                   00000005      0      0  198000000

gpmi_bch_apb             usdhc3                   00000005      0      0  198000000

per1_bch                 usdhc3                   00000005      0      0  198000000

apbh_dma                 usdhc3                   00000005      0      0  198000000

usdhc4_sel               pll2_pfd2_396m           00000000      0      0  396000000

usdhc4_podf              usdhc4_sel               00000025      0      0  198000000

usdhc4                   usdhc4_podf              00000005      0      0  198000000

gpmi_bch                 usdhc4                   00000005      0      0  198000000

emi_sel                  pll2_pfd2_396m           00000000      0      0  396000000

emi_podf                 emi_sel                  00000004      0      0  198000000

enfc_sel                 pll2_pfd2_396m           000002a2      0      0  396000000

enfc_pred                enfc_sel                 00000025      0      0   79200000

enfc_podf                enfc_pred                00000025      0      0   19800000

enfc                     enfc_podf                00000005      0      0   19800000

gpmi_io                  enfc                     00000005      0      0   19800000

periph_pre               pll2_bus                 000002a0      1      1  528000000

periph                   periph_pre               00000080      2      2  528000000

ahb                      periph                   00000004      8      8  132000000

ipg                      ahb                      00000025      4      4   66000000

ipg_per                  ipg                      00000004      0      0   66000000

gpt_ipg_per              ipg_per                  00000005      0      0   66000000

i2c1                     ipg_per                  00000005      0      0   66000000

i2c2                     ipg_per                  00000005      0      0   66000000

i2c3                     ipg_per                  00000005      0      0   66000000

pwm1                     ipg_per                  00000005      0      0   66000000

pwm2                     ipg_per                  00000005      0      0   66000000

pwm3                     ipg_per                  00000005      0      0   66000000

pwm4                     ipg_per                  00000005      0      0   66000000

caam_ipg                 ipg                      00000005      1      1   66000000

can1_ipg                 ipg                      00000005      0      0   66000000

can2_ipg                 ipg                      00000005      0      0   66000000

enet                     ipg                      00000005      2      2   66000000

gpt_ipg                  ipg                      00000005      1      1   66000000

iim                      ipg                      00000005      0      0   66000000

spba                     ipg                      00000005      0      0   66000000

spdif_gclk               ipg                      00000005      0      0   66000000

ssi1_ipg                 ipg                      00000005      0      0   66000000

ssi2_ipg                 ipg                      00000005      0      0   66000000

ssi3_ipg                 ipg                      00000005      0      0   66000000

uart_ipg                 ipg                      00000005      1      1   66000000

usboh3                   ipg                      00000005      0      0   66000000

cko1_sel                 ahb                      000002a2      0      0  132000000

cko1_podf                cko1_sel                 00000025      0      0   16500000

cko1                     cko1_podf                00000024      0      0   16500000

asrc_ipg                 ahb                      00000005      0      0  132000000

asrc_mem                 ahb                      00000005      0      0  132000000

caam_mem                 ahb                      00000005      1      1  132000000

caam_aclk                ahb                      00000005      1      1  132000000

esai_ipg                 ahb                      00000005      0      0  132000000

esai_mem                 ahb                      00000005      0      0  132000000

hdmi_iahb                ahb                      00000005      1      1  132000000

ocram                    ahb                      00000004      2      2  132000000

rom                      ahb                      00000005      1      1  132000000

sata                     ahb                      00000005      1      1  132000000

sdma                     ahb                      00000005      8      8  132000000

mmdc_ch0_axi             periph                   00000004      1      1  528000000

ipu1_sel                 mmdc_ch0_axi             000002a2      0      0  528000000

ipu1_podf                ipu1_sel                 00000025      0      0  264000000

dcic1                    ipu1_podf                00000005      0      0  264000000

ipu1                     ipu1_podf                00000005      0      0  264000000

ipu1_pclk0_sel           ipu1                     00000000      0      0  264000000

ipu1_pclk0_div           ipu1_pclk0_sel           00000004      0      0          0

ipu1_pclk_0              ipu1_pclk0_div           00000004      0      0          0

ipu1_pclk1_sel           ipu1                     00000000      0      0  264000000

ipu1_pclk1_div           ipu1_pclk1_sel           00000004      0      0          0

ipu1_pclk_1              ipu1_pclk1_div           00000004      0      0          0

ipu2_sel                 mmdc_ch0_axi             000002a2      0      0  528000000

ipu2_podf                ipu2_sel                 00000025      0      0  264000000

dcic2                    ipu2_podf                00000005      0      0  264000000

ipu2                     ipu2_podf                00000005      0      0  264000000

ipu2_pclk0_sel           ipu2                     00000000      0      0  264000000

ipu2_pclk0_div           ipu2_pclk0_sel           00000004      0      0          0

ipu2_pclk_0              ipu2_pclk0_div           00000004      0      0          0

gpu3d_core_sel           mmdc_ch0_axi             000002a2      0      0  528000000

gpu3d_core_podf          gpu3d_core_sel           00000025      0      0  528000000

gpu3d_core               gpu3d_core_podf          00000005      0      0  528000000

periph2_pre              pll2_bus                 000002a2      0      0  528000000

periph2                  periph2_pre              00000080      0      0  528000000

mmdc_ch1_axi             periph2                  00000004      0      0  528000000

pll3_bypass_src          osc                      000002a2      1      1   24000000

pll3                     pll3_bypass_src          00000041      1      1  480000000

pll3_bypass              pll3                     000002a6      1      1  480000000

pll3_usb_otg             pll3_bypass              00000024      3      3  480000000

usbphy1                  pll3_usb_otg             00000024      0      0  480000000

pll3_pfd0_720m           pll3_usb_otg             00000000      0      0  720000000

pll3_pfd1_540m           pll3_usb_otg             00000000      2      2  540000000

video_27m                pll3_pfd1_540m           00000024      0      0   27000000

hdmi_isfr                pll3_pfd1_540m           00000005      1      1  540000000

axi_alt_sel              pll3_pfd1_540m           000002a2      1      1  540000000

axi_sel                  axi_alt_sel              000002a0      1      1  540000000

axi                      axi_sel                  00000004      2      2  270000000

gpu2d_axi                axi                      000002a2      0      0  270000000

gpu3d_axi                axi                      000002a2      0      0  270000000

pcie_axi_sel             axi                      000002a2      0      0  270000000

pcie_axi                 pcie_axi_sel             00000005      0      0  270000000

emi_slow_sel             axi                      00000000      1      1  270000000

emi_slow_podf            emi_slow_sel             00000004      1      1  135000000

eim_slow                 emi_slow_podf            00000005      2      2  135000000

vdo_axi_sel              axi                      000002a2      0      0  270000000

vdo_axi                  vdo_axi_sel              00000005      0      0  270000000

vdoa                     vdo_axi                  00000005      0      0  270000000

vpu_axi_sel              axi                      000002a2      0      0  270000000

vpu_axi_podf             vpu_axi_sel              00000025      0      0  270000000

vpu_axi                  vpu_axi_podf             00000005      0      0  270000000

mlb                      axi                      00000005      0      0  270000000

openvg_axi               axi                      00000005      0      0  270000000

pll3_pfd2_508m           pll3_usb_otg             00000000      0      0  508235294

ssi1_sel                 pll3_pfd2_508m           00000000      0      0  508235294

ssi1_pred                ssi1_sel                 00000025      0      0  127058823

ssi1_podf                ssi1_pred                00000025      0      0   63529411

ssi1                     ssi1_podf                00000005      0      0   63529411

ssi2_sel                 pll3_pfd2_508m           00000000      0      0  508235294

ssi2_pred                ssi2_sel                 00000025      0      0  127058823

ssi2_podf                ssi2_pred                00000025      0      0   63529411

ssi2                     ssi2_podf                00000005      0      0   63529411

ssi3_sel                 pll3_pfd2_508m           00000000      0      0  508235294

ssi3_pred                ssi3_sel                 00000025      0      0  127058823

ssi3_podf                ssi3_pred                00000025      0      0   63529411

ssi3                     ssi3_podf                00000005      0      0   63529411

pll3_pfd3_454m           pll3_usb_otg             00000000      0      0  454736842

spdif_sel                pll3_pfd3_454m           000002a2      0      0  454736842

spdif_pred               spdif_sel                00000025      0      0  227368421

spdif_podf               spdif_pred               00000025      0      0   28421052

spdif                    spdif_podf               00000005      0      0   28421052

pll3_120m                pll3_usb_otg             00000024      0      0  120000000

pll3_80m                 pll3_usb_otg             00000024      1      1   80000000

uart_serial_podf         pll3_80m                 00000025      1      1   80000000

uart_serial              uart_serial_podf         00000005      1      1   80000000

pll3_60m                 pll3_usb_otg             00000024      0      0   60000000

ecspi_root               pll3_60m                 00000025      0      0   60000000

ecspi1                   ecspi_root               00000005      0      0   60000000

ecspi2                   ecspi_root               00000005      0      0   60000000

ecspi3                   ecspi_root               00000005      0      0   60000000

ecspi4                   ecspi_root               00000005      0      0   60000000

ecspi5                   ecspi_root               00000005      0      0   60000000

can_root                 pll3_60m                 00000025      0      0   30000000

can1_serial              can_root                 00000005      0      0   30000000

can2_serial              can_root                 00000005      0      0   30000000

periph_clk2_sel          pll3_usb_otg             000002a0      0      0  480000000

periph_clk2              periph_clk2_sel          00000025      0      0  480000000

periph2_clk2_sel         pll3_usb_otg             000002a2      0      0  480000000

periph2_clk2             periph2_clk2_sel         00000025      0      0  480000000

esai_sel                 pll3_usb_otg             000002a2      0      0  480000000

esai_pred                esai_sel                 00000025      0      0  240000000

esai_podf                esai_pred                00000025      0      0   30000000

esai_extal               esai_podf                00000005      0      0   30000000

asrc_sel                 pll3_usb_otg             000002a2      0      0  480000000

asrc_pred                asrc_sel                 00000025      0      0  240000000

asrc_podf                asrc_pred                00000025      0      0   30000000

asrc                     asrc_podf                00000005      0      0   30000000

gpu2d_core_sel           pll3_usb_otg             000002a2      0      0  480000000

gpu2d_core_podf          gpu2d_core_sel           00000025      0      0  480000000

gpu2d_core               gpu2d_core_podf          00000005      0      0  480000000

pll4_bypass_src          osc                      000002a2      0      0   24000000

pll4                     pll4_bypass_src          00000041      0      0  147456000

pll4_bypass              pll4                     000002a6      0      0  147456000

pll4_audio               pll4_bypass              00000024      0      0  147456000

pll4_post_div            pll4_audio               00000025      0      0   36864000

pll4_audio_div           pll4_post_div            00000025      0      0   36864000

pll5_bypass_src          osc                      000002a2      0      0   24000000

pll5                     pll5_bypass_src          00000041      0      0  296600000

pll5_bypass              pll5                     000002a6      0      0  296600000

pll5_video               pll5_bypass              00000024      0      0  296600000

pll5_post_div            pll5_video               00000025      0      0   74150000

pll5_video_div           pll5_post_div            00000025      0      0   74150000

ipu1_di0_pre_sel         pll5_video_div           000002a6      0      0   74150000

ipu1_di0_pre             ipu1_di0_pre_sel         00000025      0      0   24716666

ipu1_di0_sel             ipu1_di0_pre             000002a6      0      0   24716666

ipu1_di0                 ipu1_di0_sel             00000005      0      0   24716666

ipu1_di1_pre_sel         pll5_video_div           000002a6      0      0   74150000

ipu1_di1_pre             ipu1_di1_pre_sel         00000025      0      0   24716666

ipu1_di1_sel             ipu1_di1_pre             000002a6      0      0   24716666

ipu1_di1                 ipu1_di1_sel             00000005      0      0   24716666

ipu2_di0_pre_sel         pll5_video_div           000002a6      0      0   74150000

ipu2_di0_pre             ipu2_di0_pre_sel         00000025      0      0   24716666

ipu2_di0_sel             ipu2_di0_pre             000002a6      0      0   24716666

ipu2_di0                 ipu2_di0_sel             00000005      0      0   24716666

ipu2_di1_pre_sel         pll5_video_div           000002a6      0      0   74150000

ipu2_di1_pre             ipu2_di1_pre_sel         00000025      0      0   24716666

pll6_bypass_src          osc                      000002a2      1      1   24000000

pll6                     pll6_bypass_src          00000041      1      1  500000000

pll6_bypass              pll6                     000002a6      1      1  500000000

pll6_enet                pll6_bypass              00000024      1      1  500000000

sata_ref                 pll6_enet                00000024      0      0  100000000

sata_ref_100m            sata_ref                 00000024      0      0  100000000

lvds1_sel                sata_ref                 000002a2      0      0  100000000

lvds1_gate               lvds1_sel                00000004      0      0  100000000

pcie_ref                 pll6_enet                00000024      0      0  125000000

pcie_ref_125m            pcie_ref                 00000024      0      0  125000000

enet_ref                 pll6_enet                00000020      1      1  125000000

pll7_bypass_src          osc                      000002a2      1      1   24000000

pll7                     pll7_bypass_src          00000041      1      1  480000000

pll7_bypass              pll7                     000002a6      1      1  480000000

pll7_usb_host            pll7_bypass              00000024      1      1  480000000

usbphy2                  pll7_usb_host            00000024      1      1  480000000

gpt_3m                   osc                      00000024      1      1    3000000

cko2_sel                 osc                      000002a2      0      0   24000000

cko2_podf                cko2_sel                 00000025      0      0   24000000

cko2                     cko2_podf                00000024      0      0   24000000

cko                      cko2                     000002a2      0      0   24000000

dummy                       ---                   00000030      2      2          0

usbphy1_gate             dummy                    00000024      1      1          0

usbphy2_gate             dummy                    00000024      1      1          0

lvds2_sel                dummy                    000002a2      0      0          0

lvds2_gate               lvds2_sel                00000004      0      0          0

anaclk1                     ---                   00000030      0      0          0

lvds1_in                 anaclk1                  00000004      0      0          0

anaclk2                     ---                   00000030      0      0          0

lvds2_in                 anaclk2                  00000004      0      0          0

orphans                     ---                   00000004      0      0          0

root@imx6qsabresd:/unit_tests#

[igorpadykov]

Hi Tarek

it may be useful to look at following resources how to check PCIe signals :

"PCIe Certification Guide for i.MX 6 :"

http://cache.freescale.com/files/32bit/doc/app_note/AN4784.pdf

"QA: About i.Mx6 PCIe compliance"

https://community.freescale.com/docs/DOC-95554

"How to setup i.MX6Dual/Quad and i.MX6Solo/DualLite Linux software for PCIe compliance Test?"

https://community.freescale.com/docs/DOC-94923

and try attached bare-metal sdk pcie test.

Best regards

igor

-----------------------------------------------------------------------------------------------------------------------

Note: If this post answers your question, please click the Correct Answer button. Thank you!

-----------------------------------------------------------------------------------------------------------------------

[Tarek]

Hi Igor,

Thanks for the information. All the documents you have provided is talking about compliance test. what I need is to bring back the PCI link to life.

I have tested the bare-metal sdk pcie test but that did not work. I get error message: Link timeout ! when running the test.

What could be the problem?

Thanks

[igorpadykov]

Hi Tarek

suggest to try other pcie module.

If RST_B is used, then check that i.MX6 processor should not be

sending any signal until > 100 msec from the point that RST_B goes high.

Best regards

igor

[Tarek]

Hi Igor,

I have tried a reference Ethernet card 100% sure that it's working in a PC and currently not working with the SabreSD board. The same card use to work before with the Sabre board.

I'm not sure about the RST_B and the signal timing. I am using The Freescale reference board and Freescale reference Linux software.

How do I modify the signals and the timing?

Also I would expect the HW/SW to be configure to accommodate most PCI cards in the market or do I need to keep changing the configuration for  each card?

Is there any Power-Gating registers for PCI that I need to check?

Thanks,

Tarek

[igorpadykov]

Hi Tarek

not all cards may be detected and AN4784 on p.3 shows how PCIe_PHY can be

adjusted by changing the IOMUXC_GPR8 register settings.

Best regards

igor

[Tarek]

Hi Igor,

Let me clarify this. I have 2 SabreSD boards running the same software. On one board PCI is working fine while the other it stopped working recently. I've tried the same PCI-NIC card and the Same SD card on both boards one is working and one is not.

The faulty board is not sending the training sequence at initialization as I can see on the oscilloscope and the PCI analyzer. The IOMUXC_GPR8 will adjust the amplitude and de-emphasis of the TX signal. But if there is no signal this is not going to help!

As I mentioned before I have tested a version of bare-metal SDK and I get Link timeout ! when running the PCI test.

Is there any point from running the attached file you provided? I'm not sure how to compile it on it's own. Do you have some instructions?

Thanks,

Tarek

[igorpadykov]

Hi Tarek

NXP has a service which process a return of failure board.

http://www.nxp.com/support/sales-and-support/buy-direct/returns-and-warranty-information:WTBUY_DTD_R...

Best regards

igor