pcie "phy link never came up" on sabresd running dora 3.10.17_beta

[scottwarner]

I'm trying to enable pcie support on a sabresd board running the dora 3.10.17 beta bsp.  I enabled PCI support and the freescale i.MX6 PCIe controller, but did not enable EP or RC support.  When the system boots I see this:

i2c i2c-0: IMX I2C adapter registered

i2c i2c-1: IMX I2C adapter registered

i2c i2c-2: IMX I2C adapter registered

pps_core: LinuxPPS API ver. 1 registered

pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>

PTP clock support registered

MIPI CSI2 driver module loaded

Switching to clocksource mxc_timer1

imx6q-pcie 1ffc000.pcie: phy link never came up

PCI host bridge to bus 0000:00

pci_bus 0000:00: root bus resource [io  0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers enabled

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01100000-0x0110ffff pref]

pci 0000:00:00.0: PCI bridge to [bus 01]

NET: Registered protocol family 2

This worked under the 3.10.9 alpha bsp, not sure what I'm missing on the beta.

Thanks,

Scott

[alexlevandovski]

Hello Yuan, can you help me to get PCIE came up with 3.10.17?

Regards

Alex

[Matt_W]

I found a different solution to this problem and I thought I would post it here because the symptoms were almost exactly like those described in this thread.

We are developing a custom iMx6dl board based on the SabreLite iMx6dl. I was trying to get a PCIe WiFi system working. Out of the 6 different WiFi modules we tried, 2 would work and 4 would not.

Turned out that the termination resistors on the PCIe reference clock were incorrect (50K instead of 50 Ohm). Once we fixed that problem, all 6 modules respond to lspci queries and everything is working. (I actually think it's more odd that any of the WiFi boards worked with this condition.)

Hope this helps,

Matt

[robo]

Hi Matt,

can you pls clarify your setup?

• Your imx6 system is EP?
• How does the card termination look like?
• You replaced each 50R resitors with 50k resistors?

thanks and regards

[leoschwab]

Welcome to the club.

Between the git repository and the LKML, PCIe support for the 3.10 kernel series seems to be highly active (there was a massive re-write between 3.10.9 and 3.10.17).  It's been three months since 3.10.17 was made available.  My PCIe-fu is not strong, so I've been waiting patiently for the next release in the hope it's better and that it enumerates and supports PCIe switches out of the box.

Schwab

[scottwarner]

I saw some of your earlier posts, it's what prompted me to port the beta since there were so many changes.  Glad I'm not the only one having problems with PCIe.  We are getting close now, just need to get the board to wake from suspend to memory.

Scott

[YixingKong]

Scoot

Had your issue got resolved? If yes, we are going to close the discussion in 3 days. If you still need help, please feel free to reply with an update to this discussion.

Thanks,

Yixing

[scottwarner]

It's not closed yet, I am working this issue in the background waiting on advice on what we should look at regarding quad vs solo differences.

Thanks,

Scott

[b47504]

Hi Scott Warner,

I have no solo board to address the issue. I requested one dl board, in the theroy, for PCIe modle, they should be the same.

I tried with intel pcie wifi card, it seems work well. I am thinking how I can  align to your environment.

Poky (Yocto Project Reference Distro) 1.5 imx6dlsabresd /dev/ttymxc0

imx6dlsabresd login: root

root@imx6dlsabresd:~# dmesg | grep pcie

root@imx6dlsabresd:~# dmesg | grep pci

pci_bus 0000:00: root bus resource [io  0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400

pci 0000:00:00.0: reg 10: [mem 0x00000000-0x000fffff]

pci 0000:00:00.0: reg 38: [mem 0x00000000-0x0000ffff pref]

pci 0000:00:00.0: supports D1

pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold

pci 0000:01:00.0: [8086:4229] type 00 class 0x028000

pci 0000:01:00.0: reg 10: [mem 0x00000000-0x00001fff 64bit]

pci 0000:01:00.0: PME# supported from D0 D3hot D3cold

pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01

pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 01

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]

pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x01101fff 64bit]

pci 0000:00:00.0: PCI bridge to [bus 01]

pci 0000:00:00.0:   bridge window [mem 0x01100000-0x011fffff]

pci_bus 0000:00: resource 4 [io  0x1000-0x10000]

pci_bus 0000:00: resource 5 [mem 0x01000000-0x01efffff]

pci_bus 0000:01: resource 1 [mem 0x01100000-0x011fffff]

ehci-pci: EHCI PCI platform driver

root@imx6dlsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 81.805 msecs

PM: suspend devices took 0.080 seconds

PM: late suspend of devices complete after 0.498 msecs

PM: noirq suspend of devices complete after 0.723 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 0.463 msecs

PM: early resume of devices complete after 0.350 msecs

PCI: enabling device 0000:00:00.0 (0000 -> 0003)

PM: resume of devices complete after 211.968 msecs

PM: resume devices took 0.210 seconds

Restarting tasks ... done.

root@imx6dlsabresd:~#

root@imx6dlsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 71.871 msecs

PM: suspend devices took 0.070 seconds

PM: late suspend of devices complete after 0.425 msecs

PM: noirq suspend of devices complete after 0.627 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 17.447 msecs

PM: early resume of devices complete after 0.348 msecs

PM: resume of devices complete after 205.462 msecs

PM: resume devices took 0.200 seconds

Restarting tasks ... done.

root@imx6dlsabresd:~# cat /proc/version

Linux version 3.10.17-1.0.0_beta+gec1af9f (b47504@mpuapae) (gcc version 4.8.1 (G

CC) ) #3 SMP Thu Apr 24 01:40:18 HKT 2014

root@imx6dlsabresd:~#

Best Regards,

[scottwarner]

Hi Yuan,

Here's the same info on our board.  It's similar, although I noticed the following line was missing from "dmesg | grep pci" on our board:

ehci-pci: EHCI PCI platform driver

That seems to be related to USB, we are not using the USB controller on our board so USB support was not selected in our kernel configuration.  I'm not sure if this is related, but seeing this reminded me that I did have an issue with USB for suspend to memory on the 3.0.35 bsp, I had to comment out the usb_power_down_handler() and usb_power_up_handler() calls in the mx6_suspend_enter() function in pm.c.  I didn't see similar calls in the 3.10.17 bsp.  Are you aware of any relation between USB and PCIE (power, clocks, etc.)?

A quick reminder of the issue I'm seeing on 3.10.17:

1.  Suspend to memory works fine when PCIE is not enabled in the kernel config

2.  Once enabled we cannot wake, debug messages point to a hang on wakeup when trying to access the config space of bus 0 to determine the power state.

3.. This occurs with or without a device connected on the PCIE bus.

4.  This board wakes from suspend to memory running the 3.0.35, without a card installed on the bus, it doesn't wake with a card installed, but that's a known issue with a software fix already in 3.10.17

/ # cat /proc/version

Linux version 3.10.17-1.0.0_beta+gec1af9f (swarner@embedded.corp.carestreamhealth.com) (gcc version 4.8.1 (GCC) ) #4 PREEMPT Wed Apr 23 13:19:45 EDT 2014

/ # dmesg | grep pcie

/ # dmesg | grep pci

pci_bus 0000:00: root bus resource [io  0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400

pci 0000:00:00.0: reg 10: [mem 0x00000000-0x000fffff]

pci 0000:00:00.0: reg 38: [mem 0x00000000-0x0000ffff pref]

pci 0000:00:00.0: supports D1

pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold

pci 0000:01:00.0: [168c:0034] type 00 class 0x028000

pci 0000:01:00.0: reg 10: [mem 0x00000000-0x0007ffff 64bit]

pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0000ffff pref]

pci 0000:01:00.0: supports D1 D2

pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold

pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01

pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 01

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: BAR 9: assigned [mem 0x01200000-0x012fffff pref]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01300000-0x0130ffff pref]

pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x0117ffff 64bit]

pci 0000:01:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]

pci 0000:00:00.0: PCI bridge to [bus 01]

pci 0000:00:00.0:   bridge window [mem 0x01100000-0x011fffff]

pci 0000:00:00.0:   bridge window [mem 0x01200000-0x012fffff pref]

pci_bus 0000:00: resource 4 [io  0x1000-0x10000]

pci_bus 0000:00: resource 5 [mem 0x01000000-0x01efffff]

pci_bus 0000:01: resource 1 [mem 0x01100000-0x011fffff]

pci_bus 0000:01: resource 2 [mem 0x01200000-0x012fffff pref]

/ # echo enabled > /sys/class/tty/ttymxc0/power/wakeup

/ # echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

PM: suspend of devices complete after 9.276 msecs

PM: late suspend of devices complete after 0.230 msecs

PM: noirq suspend of devices complete after 0.487 msecs

Thanks,

Scott

[b47504]

Hi Scott,

Could you please share your .config file to me? So I can tried it on my side.

Best Regards,

[scottwarner]

Hi Yuan,

Here's the config:

Thanks,

Scott

[b47504]

Hi Scott,

I tried with your config on my dl board. The system cannot be woke up at the second or third time.

Then, I disabled the PCIe config item in the config file, the issue still can be reproduced.

After I enabled the USB_SUPPORT item in the config file , the issue disappeared. So I suggest you can enable USB_SUPPORT, and take one try. It seems there are still some differences between us, and the issue produced on my side is not likely introduced by Pcie.

Best Regards,

Zhao Yuan

[scottwarner]

Hi Yuan,

I edited .config to enable USB_SUPPORT, but I still have the same issue.  One thing I should note though, our .dtb does not setup anything for USB so I'm not sure if enabling USB_SUPPORT in my .config really does anything.

Thanks,

Scott

[b47504]

Hi Scott,

I just get one solo evk board, I tried this  on the board(maybe you are using solo sd board), but unlucky, the issue cannot still be produced. Can you share your uImage and dtb file to me.I wanted to check if the difference of the board leading to the issue or software.

Best Regards,

[scottwarner]

Hi Yuan,

Attached are my uImage and dtb file.

Thanks!

Scott

[b47504]

Hi Scott,

Can you please dump 20c_80e0 register when the system hang? I want to check if pcie clock setting is right.

Best Regards,

[scottwarner]

Hi Yuan,

Sorry for the late reply, I'm in the middle of bringing up a new board, and had to get to a point where I could switch back to this.  Here's the register dump:

0x20c_80e0 = 0x80102001

Thanks,

Scott

[b47504]

Hi Scott,

Can you please try this patch on your side?

Best Regards,

[scottwarner]

Hi Yuan,

It seems better with this patch, If if wakes up the first time after a cold boot (power cycle) I can't make it fail after that, it wakes up every time until I cycle power.  It sometimes still will not wake up the first time after a cold boot though.  It looks like the board hangs prior to the new resume function running.  Here's the console output when it wakes up:

/ # echo enabled > /sys/class/tty/ttymxc4/power/wakeup

/ # echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

PM: suspend of devices complete after 17.365 msecs

PM: late suspend of devices complete after 0.235 msecs

PM: noirq suspend of devices complete after 0.390 msecs

start pci_update_current_state

end pci_update_current_state

PM: noirq resume of devices complete after 5.635 msecs

PM: early resume of devices complete after 0.202 msecs

start mx6_pcie_resume

end mx6_pcie_resume

dpm_run_callback(): platform_pm_resume+0x0/0x4c returns 19

PM: Device 1ffc000.pcie failed to resume: error 19

PM: resume of devices complete after 187.477 msecs

Restarting tasks ... done.

Now here is what I see when it hangs (consistent with what I saw prior to the patch):

/ # echo enabled > /sys/class/tty/ttymxc4/power/wakeup

/ # echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

PM: suspend of devices complete after 24.890 msecs

PM: late suspend of devices complete after 0.238 msecs

PM: noirq suspend of devices complete after 0.358 msecs

start pci_update_current_state

Tomorrow I'll work more on this to see if I can narrow down when it hangs. 

Thanks,

Scott

[b47504]

Maybe since the difference of the board, the uImage and dtb file did not work on my board directly.

As I know, there is also  one workaround solution about PCIE suspend/resume on Ltib 4.0 https://community.freescale.com/docs/DOC-95130.