topic Accessing on PCIe memory in T-Series

Accessing on PCIe memory

utkarsh_100 — Thu, 04 Oct 2018 06:50:18 GMT

Hi All

I want to communicate T1042 processor with AXI fpga through pcie. I can able to access AXI fpga configuration details that are listed below

-> vendor ID =                   0x10ee
device ID =                   0x7022
command register =            0x0007
status register =             0x0010
revision ID =                 0x00
class code =                  0x05
sub class code =              0x80
programming interface =       0x00
cache line =                  0x10
latency time =                0x00
header type =                0x00
BIST =                        0x00
base address 0 =              0xa0000000
base address 1 =              0x00000000
base address 2 =              0x00000000
base address 3 =              0x00000000
base address 4 =              0x00000000
base address 5 =              0x00000000
cardBus CIS pointer =         0x00000000
sub system vendor ID =        0x10ee
sub system ID =               0x0007
expansion ROM base address = 0x00000000
interrupt line =              0x29
interrupt pin =               0x01
min Grant =                   0x00
max Latency =                 0x00
Capabilities - Power Management
Capabilities - Message Signaled Interrupts: 0x48 control 0x80

But when I am writing onto the address 0xa000_0000 on the fpga side ,i am not able to access the data.

Code is below

#define PCI_BAR0 0xA0000000
fpga_Write(0x0000,0x2222);
int fpga_Write(unsigned int Addr,unsigned long data){
*(unsigned int *)(PCI_BAR0+Addr) = data;
return 0;
}

Device tree configuration is below

pci1: pcie@ffe250000 {
compatible = "fsl,qoriq-pcie";
device_type = "pci";
#size-cells = <2>;
#address-cells = <3>;
bus-range = <0 0xff>;
interrupts = <21 2 0 0>;
#interrupt-cells = <1>;
reg = <0xf 0xfe250000 0 0x10000>;
/*
* EEEEEEEEEE address properties encoded
* p PCI address upper 32 bits
* PPPPPPPPPP PCI address lower 32 bits
* ccc CPU address upper 32 bits
* CCCCCCCCCC CPU address lower 32 bits
* s range size upper 32 bits
* SSSSSSSSSS range size lower 32 bits
*
* EEEEEEEEEE p PPPPPPPPPP ccc CCCCCCCCCC s SSSSSSSSSS
*/
ranges = <0x02000000 0 0xa0000000 0xf 0xa0000000 0 0x10000000
0x01000000 0 0x00000000 0xf 0xffc10000 0 0x00010000>;
interrupt-map-mask = <0xfff800 0 0 7>;
interrupt-map = <
0000 0 0 1 &mpic 41 1 0 0
0000 0 0 2 &mpic 5 1 0 0
0000 0 0 3 &mpic 6 1 0 0
0000 0 0 4 &mpic 7 1 0 0
>;
};

I have few question that are highlighted below

1. Do I need to send data though TLP layer. How should I do that

2. Checked PCI Express memory-mapped registers .Under that enable these registers

2.1 PCI Express configuration address register

2.2 PCI Express configuration data register

but not sure on which address need to write on ,since I am using pci2 controller 2 ,i am writing on 0x25_0000 or do I have to write on 0xffe250000 (highlighted in DTS configuration)

Thanks

Re: Accessing on PCIe memory

yipingwang — Sun, 04 Nov 2018 10:45:10 GMT

Hello utkarsh rawat,

1. Please check whether you could read the FPGA information through BAR address under u-boot, please refer to the following.

=> pci 2
Scanning PCI devices on bus 2
BusDevFun VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
02.00.00   0x8086     0x107d     Network controller      0x00
=> pci head 02.00.00
vendor ID =                   0x8086
device ID =                   0x107d
command register ID =         0x0006
status register =             0x0010
revision ID =                 0x06
class code =                  0x02 (Network controller)
sub class code =              0x00
programming interface =       0x00
cache line =                  0x08
latency time =                0x00
header type =                 0x00
BIST =                        0x00
base address 0 =              0xe0000000
base address 1 =              0xe0020000
base address 2 =              0x00001001
base address 3 =              0x00000000
base address 4 =              0x00000000
base address 5 =              0x00000000
cardBus CIS pointer =         0x00000000
sub system vendor ID =        0x8086
sub system ID =               0x1092
expansion ROM base address = 0xe0040000
interrupt line =              0x00
interrupt pin =               0x01
min Grant =                   0x00
max Latency =                 0x00
=> pci display 02.00.00 0xe0000000
e0000000: 107d8086 00100006 02000006 00000008
e0000010: e0000000 e0020000 00001001 00000000
e0000020: 00000000 00000000 00000000 10928086
e0000030: e0040000 000000c8 00000000 00000100

2. For PCIe device driver in u-boot to use "pci_map_bar" to map the BAR address to the local virtual address space, please refer to drivers/net/e1000.c or drivers/block/sata_sil.c.

In Linux, please refer to Documentation/PCI/pci.txt, please refer to the example code in drivers/net/ethernet/intel/e1000/e1000_main.c.

Memory (MMIO), and I/O port addresses should NOT be read directly from the PCI device config space. Use the values in the pci_dev structure as the PCI "bus address" might have been remapped to a "host physical" address by the arch/chip-set specific kernel support. See Documentation/io-mapping.txt for how to access device registers
or device memory.

For example in a PCIe device, bar 0 is used for Port IO, and bar 1 is used for the MMIO. So here we should read the physical base address from bar 1 and remap the MMIO region as the following.

mmio_start = pci_resource_start(dev, 1);

ioaddr = ioremap(mmio_start, mmio_len);

3. Regarding dts definition, the "ranges" property definition translates from a PCI address space to a CPU address space. Memory space translates from PCI memory space 0xe0000000 to processor memory space(physical) 0xc10000000. IO memory space from PCI address 0 mapped to CPU physical address 0xff8010000.

        pci1: pcie@ffe250000 {
                reg = <0xf 0xfe250000 0 0x10000>;
                ranges = <0x02000000 0x0 0xe0000000 0xc 0x10000000 0x0 0x10000000
                                 0x01000000 0x0 0x00000000 0xf 0xf8010000 0x0 0x00010000>;
                pcie@0 {
                        ranges = <0x02000000 0 0xe0000000
                                  0x02000000 0 0xe0000000
                                  0 0x10000000

                                  0x01000000 0 0x00000000
                                  0x01000000 0 0x00000000
                                  0 0x00010000>;
                };
        };

This often needs to be coincident with u-boot definition in include/configs/T104xRDB.h.

/* controller 2, Slot 2, tgtid 2, Base address 201000 */
#ifdef CONFIG_PCIE2
#define CONFIG_SYS_PCIE2_MEM_VIRT       0x90000000
#define CONFIG_SYS_PCIE2_MEM_BUS        0xe0000000
#define CONFIG_SYS_PCIE2_MEM_PHYS       0xc10000000ull
#define CONFIG_SYS_PCIE2_MEM_SIZE       0x10000000      /* 256M */
#define CONFIG_SYS_PCIE2_IO_VIRT        0xf8010000
#define CONFIG_SYS_PCIE2_IO_BUS         0x00000000
#define CONFIG_SYS_PCIE2_IO_PHYS        0xff8010000ull
#define CONFIG_SYS_PCIE2_IO_SIZE        0x00010000      /* 64k */
#endif

Have a great day,
TIC

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------