Driver access PCIE Device via IO port failed with IMX6

[peterhong]

Hello,

I'm working for driver porting PCIE device driver from x86 to ARM on Yocto kernel 3.14.52.

Our PCIE device will request BAR with MMIO and Port IO address. Our driver will use inb()/outb()

to access the PCIE device on X86. But we cant use the same method inb()/outb() with on IMX6.

when I use inb()/outb() will read all 0.

I had tried with pci_iomap() & ioread8() to do read. It's the same all 0. How should I do to read

the PCIE IO port data ?

The resource map is following:

root@imx6qsabresd:/sys/bus/pci/devices/0000:01:00.0# cat resource

0x0000000000001080 0x000000000000108f 0x0000000000040101

0x0000000001100000 0x000000000110000f 0x0000000000040200

0x0000000000001000 0x000000000000101f 0x0000000000040101 <---

0x0000000000001020 0x000000000000103f 0x0000000000040101

0x0000000000001040 0x000000000000105f 0x0000000000040101

0x0000000000001060 0x000000000000107f 0x0000000000040101

0x0000000000000000 0x0000000000000000 0x0000000000000000

Thanks

[Yuri]

Hello,

Is it possible to look at it via the sysfs? Something like..

fd = open("/sys/devices/pci0001\:00/0001\:00\:07.0/resource0", O_RDWR | O_SYNC);

ptr = mmap(0, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

printf("PCI BAR0 0x0000 = 0x%4x\n",  *((unsigned short *) ptr);

Regards,

Yuri.

[lmendes]

Linux Kernel ARM port does not allow accessing PCI I/O resources in SYSFS (including Kernel 4.1.15), because:

/drivers/pci/pci-sysfs.c - function static int pci_mmap_resource(struct kobject *kobj, struct bin_attribute *attr,
                 struct vm_area_struct *vma, int write_combine)

calls pci_mmap_page_range(pdev, vma, mmap_type, write_combine);

with mmap_type=pci_mmap_io when PCI RESOURCE is IO...

which for ARM is defined in /arch/arm/kernel/bios32.c, like this:

int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma,
            enum pci_mmap_state mmap_state, int write_combine)
{
    if (mmap_state == pci_mmap_io)
        return -EINVAL;

    /*
     * Mark this as IO
     */
    vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);

    if (remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
                 vma->vm_end - vma->vm_start,
                 vma->vm_page_prot))
        return -EAGAIN;

    return 0;
}

[bevincb]

I was facing segmentation fault when calling inb and outb functions in my PCIe driver code when I try to write to offset for toggling LED.

I found from lspci -v output that my device is not properly mapped.

I called pci_assign_resource from my driver to allocate resources and after that I was able to see valid mapping in lspci -v output as well as was able to toggle the led by calling outb(0x01,(baseaddress+LED_OFFSET))

But when I try to read from the same offset and from other readable offsets, I am getting return value from inb as zero.

Can you please provide your inputs.

Thanks 

[peterhong]

Hi

Following t sysfs - resource

0/2/3/4/5: IO resource

1: MEM Resource

root@imx6qsabresd:/sys/bus/pci/devices/0000:01:00.0# ls
broken_parity_status      enable         macro     resource   subsystem
class                     io             modalias  resource0  subsystem_device
config                    ioport_map     msi_bus   resource1  subsystem_vendor
consistent_dma_mask_bits  ioremap        power     resource2  uevent
device                    irq            remove    resource3  vendor
dma_mask_bits             local_cpulist  rescan    resource4
driver                    local_cpus     reset     resource5
root@imx6qsabresd:/sys/bus/pci/devices/0000:01:00.0# cat resource
0x0000000000001080 0x000000000000108f 0x0000000000040101
0x0000000001100000 0x000000000110000f 0x0000000000040200
0x0000000000001000 0x000000000000101f 0x0000000000040101
0x0000000000001020 0x000000000000103f 0x0000000000040101
0x0000000000001040 0x000000000000105f 0x0000000000040101
0x0000000000001060 0x000000000000107f 0x0000000000040101
0x0000000000000000 0x0000000000000000 0x0000000000000000

I had tried with your code with some modification to read resource 0~5. It's only able to access resource 1 (MEM)

others are failed. (mmap return -1)

my code:

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
int main(int argc, char* argv[]) {
    int fd;
    int i;
    unsigned char *ptr;
    fd = open("/sys/bus/pci/devices/0000:01:00.0/resource1", O_RDWR | O_SYNC);
    printf("fd:%d\n", fd);
    if (fd < 0){
        return 0;
    }
    ptr = mmap(0, 8, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    printf("ptr:%d\n", ptr);
    for (i = 0; i < 8; ++i)
        printf("0x%4x\n",  *(ptr + i));
}

Could I access PCIE IO Resource with iMX6?

Thanks

[Yuri]

Hello,

You can do an "ls /sys/bus/pci*/*" and it should enumerate the PCI or PCIe devices,

and then - change the 'open' to match.

Regards,

Yuri.

[RobertA]

Hello Yuri,

I am having the same issue trying to access a fpga connected to the IMX6 PCIe.  My lspci see the FPGA and it gets memory allocated to it.  But when i use a simple program like above it get the following errors, Unhandled fault: external abort on non-linefetch (0x1018) at 0x76fe7000, and BUS ERROR.  I know my PCIe configuration works because i have a pcie hub and a Ethernet controller installed as well and it works.  Any ideals would be helpful.

Alex

[zakariboo]

Hi Alex;

   2 years after you, I am having the same issue. when trying to access FPGA from PCIE i get the same "unhandled FAULT: externel abort....." could you please tell me how did you manage to fix that problem?

thank you

[Yuri]

Hello, Alex !

   Please check PCIe mapping - the BUS ERROR means, that an access 
out of mapped area is performed. 

Regards,

Yuri.

[RobertA]

Hello Yuri,

Thank you for your reply.  Below is my lspci of my board. I have three devices on the PCIe connected to the IMX6.  USB controller, Ethernet controller, and FPGA.  USB and ethernet work no problems.  Know i am trying to write a simple application to communicate with the FPGA over PCIe.  When i try peters code above I get the same errors as him when using mmap to open /sys/bus/pci/devices/0000:05:00.0/resource0, which is my FPGA allocated memory.  Here is a capture of my lspci.   I noticed for the Xilinx FPGA it gets assigned "Memory at 01400000 (64-bit, non-prefetchable) [disabled] [size=1M]", but why is it disabled? 

root@imx6qsabreauto:/home/xuser/blogspot/alexcode# lspci -v
00:00.0 PCI bridge: Device 16c3:abcd (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
        Bus: primary=00, secondary=00, subordinate=05, sec-latency=0
        I/O behind bridge: 00001000-00002fff
        Memory behind bridge: 01100000-015fffff
        Prefetchable memory behind bridge: 01600000-017fffff
        [virtual] Expansion ROM at 01800000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Kernel driver in use: pcieport

01:00.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal dec
ode])
        Flags: bus master, fast devsel, latency 0
        Memory at 01500000 (32-bit, non-prefetchable) [size=16K]
        Bus: primary=01, secondary=02, subordinate=05, sec-latency=0
        I/O behind bridge: 00001000-00002fff
        Memory behind bridge: 01100000-014fffff
        Prefetchable memory behind bridge: 0000000001600000-00000000017fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=4/4 Maskable+ 64bit+
        Capabilities: [68] Express Upstream Port, MSI 00
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
        Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
        Capabilities: [fb4] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Capabilities: [148] Virtual Channel
        Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
        Kernel driver in use: pcieport

02:01.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal dec
ode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 01100000-012fffff
        Prefetchable memory behind bridge: 0000000001600000-00000000017fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=4/4 Maskable+ 64bit+
        Capabilities: [68] Express Downstream Port (Slot+), MSI 00
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
        Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
        Capabilities: [fb4] Advanced Error Reporting
        Capabilities: [148] Virtual Channel
        Capabilities: [520] Access Control Services
        Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
        Kernel driver in use: pcieport

02:02.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal dec
ode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
        I/O behind bridge: 00002000-00002fff
        Memory behind bridge: 01300000-013fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=4/4 Maskable+ 64bit+
        Capabilities: [68] Express Downstream Port (Slot+), MSI 00
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
        Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
        Capabilities: [fb4] Advanced Error Reporting
        Capabilities: [148] Virtual Channel
        Capabilities: [520] Access Control Services
        Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
        Kernel driver in use: pcieport

02:03.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab) (prog-if 00 [Normal dec
ode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
        Memory behind bridge: 01400000-014fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=4/4 Maskable+ 64bit+
        Capabilities: [68] Express Downstream Port (Slot+), MSI 00
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch
        Capabilities: [100] Device Serial Number ab-86-02-10-b5-df-0e-00
        Capabilities: [fb4] Advanced Error Reporting
        Capabilities: [148] Virtual Channel
        Capabilities: [520] Access Control Services
        Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?>
        Kernel driver in use: pcieport

03:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02) (prog-if 30 [
XHCI])
        Flags: bus master, fast devsel, latency 0, IRQ 408
        Memory at 01100000 (64-bit, non-prefetchable) [size=64K]
        Memory at 01110000 (64-bit, non-prefetchable) [size=8K]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=8/8 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [c0] MSI-X: Enable- Count=8 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Device Serial Number 08-00-28-00-00-20-00-00
        Kernel driver in use: xhci_hcd

04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
        Subsystem: Intel Corporation Device 0000
        Flags: bus master, fast devsel, latency 0, IRQ 153
        Memory at 01300000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at 2000 [disabled] [size=32]
        Memory at 01320000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Endpoint, MSI 00
        Capabilities: [a0] MSI-X: Enable+ Count=3 Masked-
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: e1000e

05:00.0 Memory controller: Xilinx Corporation Device 7021
        Subsystem: Xilinx Corporation Device 0007
        Flags: fast devsel
        Memory at 01400000 (64-bit, non-prefetchable) [disabled] [size=1M]
        Capabilities: [40] Power Management version 3
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [100] Device Serial Number 00-00-00-01-01-00-0a-35

[Yuri]

Hello,

  What are Your BARs ?

Regards,

Yuri.

[peterhong]

We used PCIE analyzer to monitor the PCIE event when I read/write 0x01f80000~0x01fb0000

(PCIE IO Space). We found the event is PCIE memory read/write !!, It's seems the PCIE Host

not translate the memory space to PCIE IO space. Is anyone meet the saturation ?

[Yuri]

Hello,

From R&D:

BAR0 thru 4 are different.

BAR0+BAR1 as 64 bit addressing bar, type memory, prefetchable, mask 0xfffff, mask

writable. So Bar1 can’t be used separately.

BAR2 : 32 bit addressing , type memory, prefetchable , mask 0xfffff, mask writable

BAR3 : 32 bit addressing , type IO, mask 0xff

Regards,

Yuri.

[GregDyess]

On (most, if not all) non-x86 platforms, PCIe "IO" regions are mapped into the same address space as "Memory" regions.  Just use the same mapping API calls and access the "IO" regions as if they were "Memory" regions.  No special ioread, inp, etc are required.

[peterhong]

Hi,

Thanks for your reply. I had tried 3 methods below to read pcie io space.

1. use inb()

2. use ioremap_nocache()

3. use ioport_map()

The PCIE host will allocated mem address: 0x01100000 & io address: 0x1020~0x103f to this card.

I can use method 2 to read PCIE mem address 0x01100000 and get right value. But I tried to read IO

by method 2 with address 0x01f80000 + 0x1020, It'll return 0.

I had do brute force to read 0x01f80000~0x01fbffff, the PCIE I/O space are all zero. What should I do

to use PCIE IO to access the PCIE device ?

static ssize_t io_store(struct device *dev,
                struct device_attribute *attr, const char *buf,
                size_t count)
{
    int status;
    long port;
    status = kstrtol(buf, 16, &port);
    if (status)
        return status;
    pr_info("%s: port:%lx value:%x\n", __func__, port, inb(port));
    return count;
}
static ssize_t ioremap_store(struct device *dev,
                struct device_attribute *attr, const char *buf,
                size_t count)
{
    int status;
    long mem;
    void __iomem* io_mem;
    int *ptr;
    long tmp;
    status = kstrtol(buf, 16, &mem);
    if (status)
        return status;
    io_mem = ioremap_nocache(mem, 4);
    if (!io_mem)
        return -EPERM;
    ptr = io_mem;
    tmp = readw(io_mem);
    if (!tmp)
        pr_info("tmp == 0\n");
    pr_info("%s: mem:%lx mapped:%lx value1:%x, value2:%x\n", __func__, mem, (unsigned long)io_mem, tmp, *ptr);
    iounmap(io_mem);
    return count;
}
static ssize_t ioport_map_store(struct device *dev,
                struct device_attribute *attr, const char *buf,
                size_t count)
{
    int status;
    long mem;
    void __iomem* io_mem;
    status = kstrtol(buf, 16, &mem);
    if (status)
        return status;
    io_mem = ioport_map(mem, 1);
    if (!io_mem)
        return -EPERM;
    pr_info("%s: mem:%lx mapped:%lx value:%x\n", __func__, mem, (unsigned long)io_mem, readl(io_mem));
    iounmap(io_mem);
    return count;
}
}

[peterhong]

My PCIE device had allocated PCI IO Space with BAR0/2/3/4/5. According from the Freescale document,

the PCIE io space is start from 0x01f0_0000 to 0x01f8_0000. I had used devmem2 to read memory

0x01f0_0000 + 0x1020 or 0x01f8_0000 + 0x1020. The devmem2 will always report "BUS error".

How should I do to read the PCIE IO Space ?

Kernel message about PCIE:

dw_pcie_host_init: io_base: 1f80000

dw_pcie_host_init: io_mod_base: 1f80000

dw_pcie_host_init: config.mem_bus_addr: 1000000

dw_pcie_host_init: config.mem_mod_base: 1000000

imx6q-pcie 1ffc000.pcie: PCI host bridge to bus 0000:00

pci_bus 0000:00: root bus resource [io  0x1000-0x10000]

pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]

pci_bus 0000:00: root bus resource [bus 00-ff]

pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400

pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]

pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]

pci 0000:00:00.0: supports D1

pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold

PCI: bus0: Fast back to back transfers disabled

pci 0000:01:00.0: [1c29:1104] type 00 class 0x070005

pci 0000:01:00.0: reg 0x10: [io  0x0000-0x000f]

pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0000000f]

pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f]

pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x001f]

pci 0000:01:00.0: reg 0x20: [io  0x0000-0x001f]

pci 0000:01:00.0: reg 0x24: [io  0x0000-0x001f]

pci 0000:01:00.0: PME# supported from D0 D3hot D3cold

PCI: bus1: Fast back to back transfers disabled

pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01

pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]

pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]

pci 0000:00:00.0: BAR 6: assigned [mem 0x01200000-0x0120ffff pref]

pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]

pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]

pci 0000:01:00.0: BAR 3: assigned [io  0x1020-0x103f]

pci 0000:01:00.0: BAR 4: assigned [io  0x1040-0x105f]

pci 0000:01:00.0: BAR 5: assigned [io  0x1060-0x107f]

pci 0000:01:00.0: BAR 0: assigned [io  0x1080-0x108f]

pci 0000:01:00.0: BAR 1: assigned [mem 0x01100000-0x0110000f]

pci 0000:00:00.0: PCI bridge to [bus 01]

pci 0000:00:00.0:  bridge window [io  0x1000-0x1fff]

pci 0000:00:00.0:  bridge window [mem 0x01100000-0x011fffff]

pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt

pci 0000:01:00.0: Signaling PME through PCIe PME interrupt

pcie_pme 0000:00:00.0:pcie01: service driver pcie_pme loaded

aer 0000:00:00.0:pcie02: service driver aer loaded

devmem2

root@imx6qsabresd:~/ddd/freescale/data# ./devmem2 0x01f01020 b

/dev/mem opened.Unhandled fault: external abort on non-linefetch (0x1018) at 0x76f32020

Memory mapped at address 0x76f32000.

Bus error

root@imx6qsabresd:~/ddd/freescale/data# ./devmem2 0x01f81020 b

/dev/mem opened.Unhandled fault: external abort on non-linefetch (0x1018) at 0x76f24020

Memory mapped at address 0x76f24000.

Bus error

root@imx6qsabresd:~/ddd/freescale/data#

[art]

Please refer to the Chapter 41 of the attached document for the details on the L3.14.52 BSP PCIe driver. Most likely, the PCIe Root Complex configuration is not built into the kernel by default. Refer to the document attached for the information on how to build-in/enable it.

Have a great day,
Artur

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[peterhong]

Hi Artur

I had already configure my kernel with IMX PCIE support, and I had see my device & PCIE bus.

My problem is how to read PCI IO, I had tried with inb()/outb() and pci_iomap() & ioread8().

It's all return 0, not really data.

Thanks for your reply

[art]

Have you appropriately configured the I/O space?

[peterhong]

Hi,

I'm used lspci to show bridge & device, it's seems visible on my system.

root@imx6qsabresd:~/ddd/freescale/data# lspci

00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01)

01:00.0 Serial controller: Device 1c29:1104 (rev 01)

When I boot complete, the PCIE device had allocated resource by host, 1 MEM space and 5 IO space

How should I do to configure the I/O space ??

root@imx6qsabresd:/sys/bus/pci/devices/0000:01:00.0# cat resource

0x0000000000001080 0x000000000000108f 0x0000000000040101

0x0000000001100000 0x000000000110000f 0x0000000000040200

0x0000000000001000 0x000000000000101f 0x0000000000040101 <---

0x0000000000001020 0x000000000000103f 0x0000000000040101

0x0000000000001040 0x000000000000105f 0x0000000000040101

0x0000000000001060 0x000000000000107f 0x0000000000040101

0x0000000000000000 0x0000000000000000 0x0000000000000000