pcie "phy link never came up" on sabresd running dora 3.10.17_beta

[scottwarner]

I'm trying to enable pcie support on a sabresd board running the dora 3.10.17 beta bsp.  I enabled PCI support and the freescale i.MX6 PCIe controller, but did not enable EP or RC support.  When the system boots I see this:

i2c i2c-0: IMX I2C adapter registered

i2c i2c-1: IMX I2C adapter registered

i2c i2c-2: IMX I2C adapter registered

pps_core: LinuxPPS API ver. 1 registered

pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>

PTP clock support registered

MIPI CSI2 driver module loaded

Switching to clocksource mxc_timer1

imx6q-pcie 1ffc000.pcie: phy link never came up

PCI host bridge to bus 0000:00

pci_bus 0000:00: root bus resource [io  0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers enabled

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01100000-0x0110ffff pref]

pci 0000:00:00.0: PCI bridge to [bus 01]

NET: Registered protocol family 2

This worked under the 3.10.9 alpha bsp, not sure what I'm missing on the beta.

Thanks,

Scott

[b47504]

I will build the environment and try to reproduce it on my side. At the same time, I have two questions:

(1)    Do you insert some pcie card into the slot? If yes, which one you are used?

(2)    Can you share your changes to enable PCIE support?

[scottwarner]

Hi Yuan,

I tried with and without a card, the card I tried with is a SparkLan WPEA-121N wifi card.  I get the same error with and without the card attached.  This card worked fine on the 3.10.9_alpha bsp.

I currently have the following enabled (although I tried many variations):

Under Bus Support:

     PCI Support

     Message Signaled Interrupts

     PCI Debugging

Under PCI host controller drivers:

     Freescale i.MX6 PCIe controller (only this was enabled, EP and RC were not enabled)

Let me know if that is sufficient, I could attach the entire .config if that helps.

Thanks,

Scott

[scottwarner]

Hi Yuan,

I was finally able to get our i.MX6Solo based custom board to link with the beta bsp, no luck on the Sabresd though.  I was only using the Sabresd to validate what I was doing on our custom board.

I'm still seeing the same problem I saw with the alpha bsp, where the system would not resume from suspend to memory.  It hangs in the pcie_pm_resume_noirq.  I'm post more info regarding this tomorrow.

Thanks,

Scott

[b47504]

Please try this patch on your side:

diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
index 0b963c8..bd7fa53 100644
--- a/drivers/pci/host/pci-imx6.c
+++ b/drivers/pci/host/pci-imx6.c
@@ -240,15 +240,18 @@ static int imx6_pcie_deassert_core_reset(struct pcie_port *pp)
                goto err_pcie_ref;
        }

+/*
        if (!IS_ENABLED(CONFIG_EP_MODE_IN_EP_RC_SYS)
                        && !IS_ENABLED(CONFIG_RC_MODE_IN_EP_RC_SYS)) {
+*/
                ret = clk_prepare_enable(imx6_pcie->lvds_gate);
                if (ret) {
                        dev_err(pp->dev, "unable to enable lvds_gate\n");
                        goto err_lvds_gate;
                }
+/*
        }
-
+*/
        ret = clk_prepare_enable(imx6_pcie->pcie_axi);
        if (ret) {
                dev_err(pp->dev, "unable to enable pcie_axi\n");

[scottwarner]

Hi Yuan,

Thanks for the patch, I tried it, but I still see the same problem.  I've traced where the board hangs, as the system wakes from suspend to memory and the noirq resume functions are called:

pci-driver.c:pci_pm_resume_noirq()

- calls -

pci-driver.c:pci_pm_default_resume_early()

- calls -

pci.c:pci_power_up()

- calls -

pci.c:pci_update_current_state()

- the board then hangs on -

pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);

So it hangs on the 1st config space access.  Sometimes it works and the board will wake, thought there may be a timing issue so I tried putting some delays in, but that didn't help either.

Thanks,

Scott

[b47504]

Hi Scott,

I am a little confused, it seems that you are not talking about “phy link never came up” issue. The link between host and device has been linked up, but there are some issues in suspend and resume process?

Could you please share the steps how to reproduce the issue, if so? My pcie wifi card can be enumerated on sabersd 6q board on my side, with dora 3.10.17_beta.

Best Regards,

Zhao Yuan

[scottwarner]

Sorry for the confusion, I was able to get the card to link on our i.MX6Solo based custom board, but still have a resume issue with PCIe.  I had the exact same issue with the alpha bsp.  I just do the following:

>echo enabled > /sys/class/tty/ttymxc0/power/wakeup

>echo mem > /sys/power/state

Then hit a key on the keyboard to wake.  Most times it hangs, as noted above, seems the same with or without a card in the pcie slot.  If I disable pcie in the kernel config, the system wakes fine.  Originally we were running the 3.0.35 kernel on this board and it would resume fine without a card in the pcie slot.  That bsp had a known issue when a card was attached, I see the fix for that in this bsp, but this is a different issue.

Thanks,

Scott

[b47504]

Hi  Scott,

With intel wifi card 4965 and yocto 3.10.17 beta version, my mx6q reference board did not reproduced your issue on my side, following your reproduce instruction. But I did not have hardware to  verify it on i.MX6Solo.

I think you can back to 3.10.17 beta version, and apply the patch I pasted above to take one try, if you still have the issue. 

root@imx6qsabresd:~# lspci

00:00.0 PCI bridge: Device 16c3:abcd (rev 01)

01:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN [Kedr)

root@imx6qsabresd:~# echo enabled > /sys/class/tty/ttymxc0/power/wakeup

root@imx6qsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 74.189 msecs

PM: suspend devices took 0.080 seconds

PM: late suspend of devices complete after 0.404 msecs

PM: noirq suspend of devices complete after 0.674 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

CPU2: shutdown

CPU3: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

CPU2: Booted secondary processor

CPU2 is up

CPU3: Booted secondary processor

CPU3 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 0.450 msecs

PM: early resume of devices complete after 0.323 msecs

PCI: enabling device 0000:00:00.0 (0000 -> 0003)

ata1: SATA link down (SStatus 0 SControl 300)

PM: resume of devices complete after 556.902 msecs

PM: resume devices took 0.560 seconds

Restarting tasks ... done.

root@imx6qsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 71.784 msecs

PM: suspend devices took 0.070 seconds

PM: late suspend of devices complete after 0.396 msecs

PM: noirq suspend of devices complete after 0.636 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

CPU2: shutdown

CPU3: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

CPU2: Booted secondary processor

CPU2 is up

CPU3: Booted secondary processor

CPU3 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 14.648 msecs

PM: early resume of devices complete after 0.321 msecs

ata1: SATA link down (SStatus 0 SControl 300)

PM: resume of devices complete after 555.601 msecs

PM: resume devices took 0.550 seconds

Restarting tasks ... done.

root@imx6qsabresd:~# echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

Suspending console(s) (use no_console_suspend to debug)

PM: suspend of devices complete after 81.807 msecs

PM: suspend devices took 0.080 seconds

PM: late suspend of devices complete after 0.394 msecs

PM: noirq suspend of devices complete after 0.630 msecs

Disabling non-boot CPUs ...

CPU1: shutdown

CPU2: shutdown

CPU3: shutdown

Enabling non-boot CPUs ...

CPU1: Booted secondary processor

CPU1 is up

CPU2: Booted secondary processor

CPU2 is up

CPU3: Booted secondary processor

CPU3 is up

pci 0000:01:00.0: Refused to change power state, currently in D3

PM: noirq resume of devices complete after 14.818 msecs

PM: early resume of devices complete after 0.315 msecs

ata1: SATA link down (SStatus 0 SControl 300)

PM: resume of devices complete after 545.576 msecs

PM: resume devices took 0.540 seconds

Restarting tasks ... done.

root@imx6qsabresd:~#

root@imx6qsabresd:~# lspci

00:00.0 PCI bridge: Device 16c3:abcd (rev 01)

01:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN [Kedr)

root@imx6qsabresd:~# cat /proc/version

Linux version 3.10.17+gec1af9f (b47504@mpuapae) (gcc version 4.8.1 (GCC) ) #14 4

root@imx6qsabresd:~#

[scottwarner]

Hi Yaun,

I see the same behavior with the patch:

/ # cat proc/version

Linux version 3.10.17-1.0.0_beta+gec1af9f (swarner@embedded.corp.carestreamhealth.com) (gcc version 4.8.1 (GCC) ) #48 Mon Mar 24 16:21:42 EDT 2014

/ # lspci

00:00.0 Class 0604: 16c3:abcd

01:00.0 Class 0280: 168c:0034

/ # echo enabled > /sys/class/tty/ttymxc0/power/wakeup

/ # echo mem > /sys/power/state

PM: Syncing filesystems ... done.

Freezing user space processes ... (elapsed 0.01 seconds) done.

Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.

PM: suspend of devices complete after 9.197 msecs

PM: late suspend of devices complete after 0.204 msecs

PM: noirq suspend of devices complete after 0.444 msecs

imx6_pm_enter start

imx6_pm back from sleep

imx6_pm completed imx6_pm_enter

dpm_resume_noirq: 445

pci_pm_resume_noirq: 720 bus:0:0

pci_pm_default_resume_early: 428

pci_power_up: 632

pci_power_up: 635

pci_power_up: 638

pci_update_current_state: 616

I have some extra lines of debug to try to trace where it hangs; 'pci_update_current_state: 616' is in file pci.c, the line following this is:

pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);

There's a printk following this line that is not printed.

Thanks,

Scott

[b47504]

Based on your kernel message, it seems the device was not resumed from suspend state normally, and the device configuration space was not allowed to access. So lead to the hang issue.

Do you pcie protocol packet data captured on the pcie wire to help analysis?

[scottwarner]

That's what I thought at first also, but the wifi card in on bus 1, and it's hanging on bus 0.  I also see the same problem if the card is not plugged in, here's the pcie info as the board boots without the wifi card plugged in, so there is nothing on the pcie bus for this boot:

Switching to clocksource mxc_timer1

imx6q-pcie 1ffc000.pcie: phy link never came up

PCI host bridge to bus 0000:00

pci_bus 0000:00: root bus resource [io  0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers enabled

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01100000-0x0110ffff pref]

pci 0000:00:00.0: PCI bridge to [bus 01]

Then lspci shows:

/ # lspci

00:00.0 Class 0604: 16c3:abcd

and waking from sleep without a card (looks the same as with the card):

imx6_pm_enter start

imx6_pm back from sleep

imx6_pm completed imx6_pm_enter

dpm_resume_noirq: 445

pci_pm_resume_noirq: 720 bus:0:0

pci_pm_default_resume_early: 428

pci_power_up: 632

pci_power_up: 635

pci_power_up: 638

pci_update_current_state: 616

To compare, here's the pcie info at boot and lspci with the card inserted:

Switching to clocksource mxc_timer1

PCI host bridge to bus 0000:00

pci_bus 0000:00: root bus resource [io  0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]

PCI: bus0: Fast back to back transfers disabled

PCI: bus1: Fast back to back transfers disabled

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: BAR 9: assigned [mem 0x01200000-0x012fffff pref]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01300000-0x0130ffff pref]

pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x0117ffff 64bit]

pci 0000:01:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]

pci 0000:00:00.0: PCI bridge to [bus 01]

pci 0000:00:00.0:   bridge window [mem 0x01100000-0x011fffff]

pci 0000:00:00.0:   bridge window [mem 0x01200000-0x012fffff pref]

/ # lspci

00:00.0 Class 0604: 16c3:abcd

01:00.0 Class 0280: 168c:0034

I'm not sure what 00:00.0 Class 0604: 16c3:abcd is, since it shows up when there is no card inserted.  When we were running the 3.0.35 kernel on this board it would wake from suspend to memory if there was no card inserted, so this is new with the 3.10.x based bsps.  lspci shows the same results on the 3.0.35 based bsp also:

/ # cat /proc/version

Linux version 3.0.35-2310-gc27cb38-svn6353 (swarner@embedded.corp.carestreamhealth.com) (gcc version 4.6.2 20110630 (prerelease) (Freescale MAD -- Linaro 2011.07 -- Built at 2011/08/10 09:20) ) #23 PREEMPT Wed Mar 19 16:45:46 EDT 2014

/ # lspci

00:00.0 Class 0604: 16c3:abcd

01:00.0 Class 0280: 168c:0034

Thanks,

Scott

[b47504]

Yes, I agree with you, if you can reproduce the issue without any pcie card, we don’t need to think about the pcie wifi card side.

But it is somewhat weird, without extra pci card, we should be on the page about the environment of hardware and software, at least for 6q board and 3.10.17_beta.

And my 6q board cannot reproduce the issue in the situation. As you know, I do not have 6Solo hardware to verify if it can work.Could you please try with other 6q board or with my image?

If it can work on 6q,  maybe the issue is special for 6Solo, we can focus on the difference of the two on software, as it can work on your previous versions. 

Best Regards,

Zhao Yuan

[scottwarner]

I do not see this issue on my sabresd board, I could wake successfully with both the 3.10.9_alpha and 3.10.17_beta bsp, and our 6Solo based board hung on the config space access for both versions of the bsp.  So I agree that we should look at the difference between the Solo and Quad.

Thanks,

Scott

[jamesbone]

I know that we have some problems with the EP on the PCIE bus, so I escalated your question so we are working to provide a response .