topic Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta in i.MX Processors

pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Wed, 26 Feb 2014 21:40:34 GMT

I'm trying to enable pcie support on a sabresd board running the dora 3.10.17 beta bsp. I enabled PCI support and the freescale i.MX6 PCIe controller, but did not enable EP or RC support. When the system boots I see this:

i2c i2c-0: IMX I2C adapter registered

i2c i2c-1: IMX I2C adapter registered

i2c i2c-2: IMX I2C adapter registered

pps_core: LinuxPPS API ver. 1 registered

PTP clock support registered

MIPI CSI2 driver module loaded

Switching to clocksource mxc_timer1

imx6q-pcie 1ffc000.pcie: phy link never came up

PCI host bridge to bus 0000:00

pci_bus 0000:00: root bus resource [io 0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers enabled

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01100000-0x0110ffff pref]

pci 0000:00:00.0: PCI bridge to [bus 01]

NET: Registered protocol family 2

This worked under the 3.10.9 alpha bsp, not sure what I'm missing on the beta.

Thanks,

Scott

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

jamesbone — Wed, 12 Mar 2014 15:41:10 GMT

I know that we have some problems with the EP on the PCIE bus, so I escalated your question so we are working to provide a response .

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

b47504 — Fri, 14 Mar 2014 03:08:45 GMT

I will build the environment and try to reproduce it on my side. At the same time, I have two questions:

(1) Do you insert some pcie card into the slot? If yes, which one you are used?

(2) Can you share your changes to enable PCIE support?

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Mon, 17 Mar 2014 13:19:43 GMT

Hi Yuan,

I tried with and without a card, the card I tried with is a SparkLan WPEA-121N wifi card. I get the same error with and without the card attached. This card worked fine on the 3.10.9_alpha bsp.

I currently have the following enabled (although I tried many variations):

Under Bus Support:

PCI Support

Message Signaled Interrupts

PCI Debugging

Under PCI host controller drivers:

Freescale i.MX6 PCIe controller (only this was enabled, EP and RC were not enabled)

Let me know if that is sufficient, I could attach the entire .config if that helps.

Thanks,

Scott

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

leoschwab — Mon, 17 Mar 2014 21:41:57 GMT

Welcome to the club.

Between the git repository and the LKML, PCIe support for the 3.10 kernel series seems to be highly active (there was a massive re-write between 3.10.9 and 3.10.17). It's been three months since 3.10.17 was made available. My PCIe-fu is not strong, so I've been waiting patiently for the next release in the hope it's better and that it enumerates and supports PCIe switches out of the box.

Schwab

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Tue, 18 Mar 2014 21:23:42 GMT

Hi Yuan,

I was finally able to get our i.MX6Solo based custom board to link with the beta bsp, no luck on the Sabresd though. I was only using the Sabresd to validate what I was doing on our custom board.

I'm still seeing the same problem I saw with the alpha bsp, where the system would not resume from suspend to memory. It hangs in the pcie_pm_resume_noirq. I'm post more info regarding this tomorrow.

Thanks,

Scott

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Tue, 18 Mar 2014 21:28:02 GMT

I saw some of your earlier posts, it's what prompted me to port the beta since there were so many changes. Glad I'm not the only one having problems with PCIe. We are getting close now, just need to get the board to wake from suspend to memory.

Scott

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

b47504 — Wed, 19 Mar 2014 06:29:49 GMT

Please try this patch on your side:

diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
index 0b963c8..bd7fa53 100644
--- a/drivers/pci/host/pci-imx6.c
+++ b/drivers/pci/host/pci-imx6.c
@@ -240,15 +240,18 @@ static int imx6_pcie_deassert_core_reset(struct pcie_port *pp)
goto err_pcie_ref;
}

+/*
        if (!IS_ENABLED(CONFIG_EP_MODE_IN_EP_RC_SYS)
                        && !IS_ENABLED(CONFIG_RC_MODE_IN_EP_RC_SYS)) {
+*/
                ret = clk_prepare_enable(imx6_pcie->lvds_gate);
                if (ret) {
                        dev_err(pp->dev, "unable to enable lvds_gate\n");
                        goto err_lvds_gate;
                }
+/*
        }
-
+*/
        ret = clk_prepare_enable(imx6_pcie->pcie_axi);
        if (ret) {
                dev_err(pp->dev, "unable to enable pcie_axi\n");

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Wed, 19 Mar 2014 13:08:25 GMT

Hi Yuan,

Thanks for the patch, I tried it, but I still see the same problem. I've traced where the board hangs, as the system wakes from suspend to memory and the noirq resume functions are called:

pci-driver.c:pci_pm_resume_noirq()

- calls -

pci-driver.c:pci_pm_default_resume_early()

- calls -

pci.c:pci_power_up()

- calls -

pci.c:pci_update_current_state()

- the board then hangs on -

pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);

So it hangs on the 1st config space access. Sometimes it works and the board will wake, thought there may be a timing issue so I tried putting some delays in, but that didn't help either.

Thanks,

Scott

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

b47504 — Wed, 19 Mar 2014 14:52:12 GMT

Hi Scott,

I am a little confused, it seems that you are not talking about “phy link never came up” issue. The link between host and device has been linked up, but there are some issues in suspend and resume process?

Could you please share the steps how to reproduce the issue, if so? My pcie wifi card can be enumerated on sabersd 6q board on my side, with dora 3.10.17_beta.

Best Regards,

Zhao Yuan

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Wed, 19 Mar 2014 19:12:05 GMT

Sorry for the confusion, I was able to get the card to link on our i.MX6Solo based custom board, but still have a resume issue with PCIe. I had the exact same issue with the alpha bsp. I just do the following:

>echo enabled > /sys/class/tty/ttymxc0/power/wakeup

>echo mem > /sys/power/state

Then hit a key on the keyboard to wake. Most times it hangs, as noted above, seems the same with or without a card in the pcie slot. If I disable pcie in the kernel config, the system wakes fine. Originally we were running the 3.0.35 kernel on this board and it would resume fine without a card in the pcie slot. That bsp had a known issue when a card was attached, I see the fix for that in this bsp, but this is a different issue.

Thanks,

Scott

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

b47504 — Mon, 24 Mar 2014 02:58:39 GMT

Hi Scott,

With intel wifi card 4965 and yocto 3.10.17 beta version, my mx6q reference board did not reproduced your issue on my side, following your reproduce instruction. But I did not have hardware to verify it on i.MX6Solo.

I think you can back to 3.10.17 beta version, and apply the patch I pasted above to take one try, if you still have the issue.

root@imx6qsabresd:~# lspci

00:00.0 PCI bridge: Device 16c3:abcd (rev 01)

01:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN [Kedr)

root@imx6qsabresd:~# echo enabled > /sys/class/tty/ttymxc0/power/wakeup

root@imx6qsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 74.189 msecs

PM: suspend devices took 0.080 seconds

PM: late suspend of devices complete after 0.404 msecs

PM: noirq suspend of devices complete after 0.674 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

CPU2: shutdown

CPU3: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

CPU2: Booted secondary processor

CPU2 is up

CPU3: Booted secondary processor

CPU3 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 0.450 msecs

PM: early resume of devices complete after 0.323 msecs

PCI: enabling device 0000:00:00.0 (0000 -> 0003)

ata1: SATA link down (SStatus 0 SControl 300)

PM: resume of devices complete after 556.902 msecs

PM: resume devices took 0.560 seconds

Restarting tasks ... done.

root@imx6qsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 71.784 msecs

PM: suspend devices took 0.070 seconds

PM: late suspend of devices complete after 0.396 msecs

PM: noirq suspend of devices complete after 0.636 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

CPU2: shutdown

CPU3: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

CPU2: Booted secondary processor

CPU2 is up

CPU3: Booted secondary processor

CPU3 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 14.648 msecs

PM: early resume of devices complete after 0.321 msecs

ata1: SATA link down (SStatus 0 SControl 300)

PM: resume of devices complete after 555.601 msecs

PM: resume devices took 0.550 seconds

Restarting tasks ... done.

root@imx6qsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 81.807 msecs

PM: suspend devices took 0.080 seconds

PM: late suspend of devices complete after 0.394 msecs

PM: noirq suspend of devices complete after 0.630 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

CPU2: shutdown

CPU3: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

CPU2: Booted secondary processor

CPU2 is up

CPU3: Booted secondary processor

CPU3 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 14.818 msecs

PM: early resume of devices complete after 0.315 msecs

ata1: SATA link down (SStatus 0 SControl 300)

PM: resume of devices complete after 545.576 msecs

PM: resume devices took 0.540 seconds

Restarting tasks ... done.

root@imx6qsabresd:~#

root@imx6qsabresd:~# lspci

00:00.0 PCI bridge: Device 16c3:abcd (rev 01)

01:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN [Kedr)

root@imx6qsabresd:~# cat /proc/version

Linux version 3.10.17+gec1af9f (b47504@mpuapae) (gcc version 4.8.1 (GCC) ) #14 4

root@imx6qsabresd:~#

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Mon, 24 Mar 2014 20:30:59 GMT

Hi Yaun,

I see the same behavior with the patch:

/ # cat proc/version

Linux version 3.10.17-1.0.0_beta+gec1af9f (swarner@embedded.corp.carestreamhealth.com) (gcc version 4.8.1 (GCC) ) #48 Mon Mar 24 16:21:42 EDT 2014

/ # lspci

00:00.0 Class 0604: 16c3:abcd

01:00.0 Class 0280: 168c:0034

/ # echo enabled > /sys/class/tty/ttymxc0/power/wakeup

/ # echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

PM: suspend of devices complete after 9.197 msecs

PM: late suspend of devices complete after 0.204 msecs

PM: noirq suspend of devices complete after 0.444 msecs

imx6_pm_enter start

imx6_pm back from sleep

imx6_pm completed imx6_pm_enter

dpm_resume_noirq: 445

pci_pm_resume_noirq: 720 bus:0:0

pci_pm_default_resume_early: 428

pci_power_up: 632

pci_power_up: 635

pci_power_up: 638

pci_update_current_state: 616

I have some extra lines of debug to try to trace where it hangs; 'pci_update_current_state: 616' is in file pci.c, the line following this is:

pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);

There's a printk following this line that is not printed.

Thanks,

Scott

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

b47504 — Tue, 25 Mar 2014 09:33:53 GMT

Based on your kernel message, it seems the device was not resumed from suspend state normally, and the device configuration space was not allowed to access. So lead to the hang issue.

Do you pcie protocol packet data captured on the pcie wire to help analysis?

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Tue, 25 Mar 2014 20:06:27 GMT

That's what I thought at first also, but the wifi card in on bus 1, and it's hanging on bus 0. I also see the same problem if the card is not plugged in, here's the pcie info as the board boots without the wifi card plugged in, so there is nothing on the pcie bus for this boot:

Switching to clocksource mxc_timer1

imx6q-pcie 1ffc000.pcie: phy link never came up

PCI host bridge to bus 0000:00

pci_bus 0000:00: root bus resource [io 0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers enabled

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01100000-0x0110ffff pref]

pci 0000:00:00.0: PCI bridge to [bus 01]

Then lspci shows:

/ # lspci

00:00.0 Class 0604: 16c3:abcd

and waking from sleep without a card (looks the same as with the card):

imx6_pm_enter start

imx6_pm back from sleep

imx6_pm completed imx6_pm_enter

dpm_resume_noirq: 445

pci_pm_resume_noirq: 720 bus:0:0

pci_pm_default_resume_early: 428

pci_power_up: 632

pci_power_up: 635

pci_power_up: 638

pci_update_current_state: 616

To compare, here's the pcie info at boot and lspci with the card inserted:

Switching to clocksource mxc_timer1

PCI host bridge to bus 0000:00

pci_bus 0000:00: root bus resource [io 0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: BAR 9: assigned [mem 0x01200000-0x012fffff pref]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01300000-0x0130ffff pref]

pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x0117ffff 64bit]

pci 0000:01:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]

pci 0000:00:00.0: PCI bridge to [bus 01]

pci 0000:00:00.0: bridge window [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: bridge window [mem 0x01200000-0x012fffff pref]

/ # lspci

00:00.0 Class 0604: 16c3:abcd

01:00.0 Class 0280: 168c:0034

I'm not sure what 00:00.0 Class 0604: 16c3:abcd is, since it shows up when there is no card inserted. When we were running the 3.0.35 kernel on this board it would wake from suspend to memory if there was no card inserted, so this is new with the 3.10.x based bsps. lspci shows the same results on the 3.0.35 based bsp also:

/ # cat /proc/version

Linux version 3.0.35-2310-gc27cb38-svn6353 (swarner@embedded.corp.carestreamhealth.com) (gcc version 4.6.2 20110630 (prerelease) (Freescale MAD -- Linaro 2011.07 -- Built at 2011/08/10 09:20) ) #23 PREEMPT Wed Mar 19 16:45:46 EDT 2014

/ # lspci

00:00.0 Class 0604: 16c3:abcd

01:00.0 Class 0280: 168c:0034

Thanks,

Scott

Re: Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

b47504 — Wed, 26 Mar 2014 12:34:27 GMT

Yes, I agree with you, if you can reproduce the issue without any pcie card, we don’t need to think about the pcie wifi card side.

But it is somewhat weird, without extra pci card, we should be on the page about the environment of hardware and software, at least for 6q board and 3.10.17_beta.

And my 6q board cannot reproduce the issue in the situation. As you know, I do not have 6Solo hardware to verify if it can work.Could you please try with other 6q board or with my image?

If it can work on 6q, maybe the issue is special for 6Solo, we can focus on the difference of the two on software, as it can work on your previous versions.

Best Regards,

Zhao Yuan

Re: Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Wed, 26 Mar 2014 13:56:50 GMT

I do not see this issue on my sabresd board, I could wake successfully with both the 3.10.9_alpha and 3.10.17_beta bsp, and our 6Solo based board hung on the config space access for both versions of the bsp. So I agree that we should look at the difference between the Solo and Quad.

Thanks,

Scott

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

YixingKong — Thu, 10 Apr 2014 07:20:53 GMT

Scoot

Had your issue got resolved? If yes, we are going to close the discussion in 3 days. If you still need help, please feel free to reply with an update to this discussion.

Thanks,

Yixing

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

scottwarner — Mon, 14 Apr 2014 15:16:17 GMT

It's not closed yet, I am working this issue in the background waiting on advice on what we should look at regarding quad vs solo differences.

Thanks,

Scott

Re: pcie "phy link never came up" on sabresd running dora 3.10.17_beta

b47504 — Wed, 23 Apr 2014 10:16:22 GMT

Hi Scott Warner,

I have no solo board to address the issue. I requested one dl board, in the theroy, for PCIe modle, they should be the same.

I tried with intel pcie wifi card, it seems work well. I am thinking how I can align to your environment.

Poky (Yocto Project Reference Distro) 1.5 imx6dlsabresd /dev/ttymxc0

imx6dlsabresd login: root

root@imx6dlsabresd:~# dmesg | grep pcie

root@imx6dlsabresd:~# dmesg | grep pci

pci_bus 0000:00: root bus resource [io 0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400

pci 0000:00:00.0: reg 10: [mem 0x00000000-0x000fffff]

pci 0000:00:00.0: reg 38: [mem 0x00000000-0x0000ffff pref]

pci 0000:00:00.0: supports D1

pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold

pci 0000:01:00.0: [8086:4229] type 00 class 0x028000

pci 0000:01:00.0: reg 10: [mem 0x00000000-0x00001fff 64bit]

pci 0000:01:00.0: PME# supported from D0 D3hot D3cold

pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01

pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 01

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]

pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x01101fff 64bit]

pci 0000:00:00.0: PCI bridge to [bus 01]

pci 0000:00:00.0: bridge window [mem 0x01100000-0x011fffff]

pci_bus 0000:00: resource 4 [io 0x1000-0x10000]

pci_bus 0000:00: resource 5 [mem 0x01000000-0x01efffff]

pci_bus 0000:01: resource 1 [mem 0x01100000-0x011fffff]

ehci-pci: EHCI PCI platform driver

root@imx6dlsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 81.805 msecs

PM: suspend devices took 0.080 seconds

PM: late suspend of devices complete after 0.498 msecs

PM: noirq suspend of devices complete after 0.723 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 0.463 msecs

PM: early resume of devices complete after 0.350 msecs

PCI: enabling device 0000:00:00.0 (0000 -> 0003)

PM: resume of devices complete after 211.968 msecs

PM: resume devices took 0.210 seconds

Restarting tasks ... done.

root@imx6dlsabresd:~#

root@imx6dlsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 71.871 msecs

PM: suspend devices took 0.070 seconds

PM: late suspend of devices complete after 0.425 msecs

PM: noirq suspend of devices complete after 0.627 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 17.447 msecs

PM: early resume of devices complete after 0.348 msecs

PM: resume of devices complete after 205.462 msecs

PM: resume devices took 0.200 seconds

Restarting tasks ... done.

root@imx6dlsabresd:~# cat /proc/version

Linux version 3.10.17-1.0.0_beta+gec1af9f (b47504@mpuapae) (gcc version 4.8.1 (G

CC) ) #3 SMP Thu Apr 24 01:40:18 HKT 2014

root@imx6dlsabresd:~#

Best Regards,