BAR 0 of PCIe EP is not accessible

[amarnathmb]

Hi,

I have a T2080 custom board running vxWorks, on which the PCIe controller 4 (RC mode) (CCSBAR + 0x270000) is connected to the upstream port of IDT PCIe switch (89PES4T4). On one of the downstream ports of the switch I have Xilinx V7 FPGA as an endpoint. The FGPA exposes two BARs, ie. BAR0 for FPGA DDR access and BAR1 for FPGA CDMA access, during enumeration PCIe controller lists FGPA with Bus: 5,Dev: 0,Fun: 0 and shows BAR0 and BAR1 available inside the header.

PCIe Header Show gives :

vendor ID =                   0x10ee
device ID =                   0x7021
command register =            0x0007
status register =             0x0010
revision ID =                 0x00
class code =                  0x05
sub class code =              0x80
programming interface =       0x00
cache line =                  0x10
latency time =                0x00
header type =                 0x00
BIST =                        0x00
base address 0 =              0xf8000000 ----------> FPGA DDR 128 MB
base address 1 =              0xa8000000  ----------> FPGA CDMA 16 MB
base address 2 =              0x00000000
base address 3 =              0x00000000
base address 4 =              0x00000000
base address 5 =              0x00000000
cardBus CIS pointer =         0x00000000
sub system vendor ID =        0x10ee
sub system ID =               0x0007
expansion ROM base address =  0x00000000
interrupt line =              0x2b
interrupt pin =               0x01
min Grant =                   0x00
max Latency =                 0x00
Capabilities - Power Management  
Capabilities - Message Signaled Interrupts: 0x48 control 0x82 Disabled, 64-bit, MME: 0 MMC: 1
    Address: 0000000000000000  Data: 0x0000
    Per-vector Mask: Unsupported  
Capabilities - PCIe: Endpoint, IRQ 0
    Device: Max Payload: 256 bytes, Phantom Funcs 1 msb, Extended Tag: 8-bit
        Acceptable Latency: L0 - <64ns, L1 - <1us
        Errors Enabled: Relaxed Ordering No Snoop
        Max Read Request 512 bytes
    Link: MAX Speed - 5.0Gb/s, MAX Width - by 1 Port - 0 ASPM - L0s
        Latency: L0s - >4us, L1 - >64us
        ASPM - Disabled, RCB - 64bytes
        Speed - 2.5Gb/s, Width - by 1
Ext Capabilities - Device Serial Number. 0x100. Version 1  
    Serial Number: 0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x0

LAW BAR settings

----------------------------------------------------------------------------
LAW        Address            Status      Size               Interface
----------------------------------------------------------------------------
LAW00    0x00f8000000  Enabled   0x08000000  Integrated Flash controller
LAW01    0x00e0000000  Enabled   0x02000000  Buffer Manager (control)
LAW02    0x00e2000000  Enabled   0x02000000  Queue Manager (control)
LAW03    0x00a0000000  Enabled   0x10000000  PCI Express 4
LAW08    0x00ee000000  Enabled   0x01000000  CCSR
LAW31    0x0000000000  Enabled   0x80000000  DDR SDRAM
----------------------------------------------------------------------------

ATMU Register Settings (CCSBAR : 0xEE000000)
WIN00  

OTAR    : 0xee270c00 0x00000000   

OTEAR : 0xee270c04 0x00000000    

OWAR  : 0xee270c10 0x80044027 

 
WIN01  (128 MB)

OTAR    : 0xee270c20 0x000a0000   

OTEAR : 0xee270c24 0x00000000   

OWBAR: 0xee270c28 0x000a0000   

OWAR  : 0xee270c30 0x8004401a

   
WIN02  (16 MB)

OTAR    : 0xee270c40 0x000a8000   

OTEAR : 0xee270c44 0x00000000   

OWBAR: 0xee270c48 0x000a8000   

OWAR  : 0xee270c50 0x80044017 

 
WIN03  

OTAR    : 0xee270c60 0x00000000   

OTEAR : 0xee270c64 0x00000000   

OWBAR: 0xee270c68 0x00000000   

OWAR  : 0xee270c70 0x00000000

After booting i'm updating BAR0 of PCIe device (5,0,0) with value 0xA0000000, after that when i try to access memory 0xA0000000 (FPGA DDR) my system hangs. But when i try to access 0xA8000000 (FPGA CDMA) it gives proper value.

Header show after updating BAR0 value:

vendor ID =                   0x10ee
device ID =                   0x7021
command register =            0x0007
status register =             0x0010
revision ID =                 0x00
class code =                  0x05
sub class code =              0x80
programming interface =       0x00
cache line =                  0x10
latency time =                0x00
header type =                 0x00
BIST =                        0x00
base address 0 =              0xa0000000
base address 1 =              0xa8000000
base address 2 =              0x00000000
base address 3 =              0x00000000
base address 4 =              0x00000000
base address 5 =              0x00000000
cardBus CIS pointer =         0x00000000
sub system vendor ID =        0x10ee
sub system ID =               0x0007
expansion ROM base address =  0x00000000
interrupt line =              0x2b
interrupt pin =               0x01
min Grant =                   0x00
max Latency =                 0x00
Capabilities - Power Management  
Capabilities - Message Signaled Interrupts: 0x48 control 0x82 Disabled, 64-bit, MME: 0 MMC: 1
    Address: 0000000000000000  Data: 0x0000
    Per-vector Mask: Unsupported  
Capabilities - PCIe: Endpoint, IRQ 0
    Device: Max Payload: 256 bytes, Phantom Funcs 1 msb, Extended Tag: 8-bit
        Acceptable Latency: L0 - <64ns, L1 - <1us
        Errors Enabled: Relaxed Ordering No Snoop
        Max Read Request 512 bytes
    Link: MAX Speed - 5.0Gb/s, MAX Width - by 1 Port - 0 ASPM - L0s
        Latency: L0s - >4us, L1 - >64us
        ASPM - Disabled, RCB - 64bytes
        Speed - 2.5Gb/s, Width - by 1
Ext Capabilities - Device Serial Number. 0x100. Version 1  
    Serial Number: 0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x0

To experiment, I modified BAR0 value with 0xA8000000 and BAR1 address with 0xA0000000, Now i can access FPGA DDR memory through 0xA8000000, but accessing 0xA0000000 hangs my system.

Header  show after interchanging BAR0 and BAR 1 values:

vendor ID =                   0x10ee
device ID =                   0x7021
command register =            0x0007
status register =             0x0010
revision ID =                 0x00
class code =                  0x05
sub class code =              0x80
programming interface =       0x00
cache line =                  0x10
latency time =                0x00
header type =                 0x00
BIST =                        0x00
base address 0 =              0xa8000000
base address 1 =              0xa0000000
base address 2 =              0x00000000
base address 3 =              0x00000000
base address 4 =              0x00000000
base address 5 =              0x00000000
cardBus CIS pointer =         0x00000000
sub system vendor ID =        0x10ee
sub system ID =               0x0007
expansion ROM base address =  0x00000000
interrupt line =              0x2b
interrupt pin =               0x01
min Grant =                   0x00
max Latency =                 0x00
Capabilities - Power Management  
Capabilities - Message Signaled Interrupts: 0x48 control 0x82 Disabled, 64-bit, MME: 0 MMC: 1
    Address: 0000000000000000  Data: 0x0000
    Per-vector Mask: Unsupported  
Capabilities - PCIe: Endpoint, IRQ 0
    Device: Max Payload: 256 bytes, Phantom Funcs 1 msb, Extended Tag: 8-bit
        Acceptable Latency: L0 - <64ns, L1 - <1us
        Errors Enabled: Relaxed Ordering No Snoop
        Max Read Request 512 bytes
    Link: MAX Speed - 5.0Gb/s, MAX Width - by 1 Port - 0 ASPM - L0s
        Latency: L0s - >4us, L1 - >64us
        ASPM - Disabled, RCB - 64bytes
        Speed - 2.5Gb/s, Width - by 1
Ext Capabilities - Device Serial Number. 0x100. Version 1  
    Serial Number: 0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x0

Can anyone help me out? What can be the issue here?

[amarnathmb · Accepted solution]

ufedor‌ thank you for your support. Sorry for late the reply, I have resolved this issue. It was an error in memory space allocation (mem and io) in my PCIe driver configuration. After allocation of proper memory spaces the switch got configured properly and thus the BAR0 and BAR1 of my FPGA.

[utkarsh_100]

How r u writing /reading onto your device ??.Is like this somewhat 

#define PCI_BAR0 0xA0000000

unsigned int data_read;

fpga_Write(0x0000,0x2222);
fpga_Read(0x0000,&data_read);

int fpga_Write(unsigned long Addr,unsigned long data)

{
*(unsigned long*)(PCI_BAR0+Addr) = data;
return 0;
}

int fpga_Read(unsigned long Addr,unsigned long *data){
*data = *(unsigned long*)(PCI_BAR0+Addr);
return 0;
}

Secondly how u r taking care of TLP ???

[amarnathmb]

Yes. Read and write are done like what you have specified. But before accessing the memory you must make sure that there is valid LAW assigned for your PCIe controller and there is a outbound window mapped for 0xA0000000 in your PCIe controller.

[utkarsh_100]

Yes we have taken care of LAW and Outbound window 

PCI Express outbound translation address register :                   a0000
PCI Express outbound translation extended address register :  0
PCI Express outbound window attributes register :                    8004401b
PCI Express outbound window base address register :             fa0000

=====================================================
PCI host # 2 PCIe: Speed - 5.0Gb/s, Width - by 2
=====================================================
[0,0,0] - (1957 0825) type=PROCESSOR
status=0x0010 ( CAP DEVSEL=0 )
command=0x0007 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE )
[1,0,0] - (10ee 7022) type=MEM_CNTLR
status=0x0010 ( CAP DEVSEL=0 )
command=0x0007 ( IO_ENABLE MEM_ENABLE MASTER_ENABLE )
bar0 in 32-bit mem space @ 0x00000000a0000000

But still when writing onto the FPGA (writing on 0xA000_0000) ,Like below

*(unsigned long*)(PCI_BAR0+Addr) = data;

But on fpga its telling I got the interrupt but the address its showing is different 0xA0107000  with different data .

Not sure how to map it .Do I have to map PCI Express outbound translation address register ??

[amarnathmb]

You have set PEXOWBAR as 0xFA000000 and PEXOTAR as 0xA0000000. Your PCIe driver as assigned BAR0 with 0xA0000000, that means 0xA0000000 must be a part of the LAW address space allotted for the PCIe controller. In that case your PEXOWBAR must be set to 0xA0000000 (Physical space address)and PEXOTAR must be set to 0xA0000000 (External address space), so that while accessing 0xA0000000 of your physical space gets translated to 0xA0000000 of your external space.

With your current settings you need to set macro PCI_BAR0 as 0xFA000000 so as to access 0xA0000000 of PCIe space. But you need to be sure that 0xFA000000 is mapped properly.

[utkarsh_100]

Hi Amarnath 

Thanks for the support , able to access fpga memory by accessing  PCI Express outbound translation address register .

I have one more doubt how to access TLP Header from the processor . 

[ufedor]

Please provide raw memory dumps for LAWs and PCIe4 Outbound ATMUs CCSR registers.

You wrote:

base address 0 =              0xf8000000 ----------> FPGA DDR 128 MB

How this corresponds to any OTAR?

[amarnathmb]

Hi Ufedor,

Thanks for replying.

"base address 0 =              0xf8000000 ----------> FPGA DDR 128 MB" is the initial header show result after booting up, later i'm updating the BAR0 register with value 0xA0000000 (you can see this in the second header shoe result 'base address 0 =              0xa0000000'). I have created an outbound window for 0xA0000000, 128 MB using outbound window 1.

WIN01  (128 MB)

OTAR    : 0xee270c20 0x000a0000   

OTEAR : 0xee270c24 0x00000000   

OWBAR: 0xee270c28 0x000a0000   

OWAR  : 0xee270c30 0x8004401a

Please find raw memory dumps for LAWs and PCIe4 Outbound ATMUs CCSR registers,

LAW BAR Dump:

NOTE: memory values are displayed in hexadecimal.
0xee000c00:  0000 0000 f800 0000 81f0 001a 0000 0000  *................*
0xee000c10:  0000 0000 e000 0000 8180 0018 0000 0000  *................*
0xee000c20:  0000 0000 e200 0000 83c0 0018 0000 0000  *................*
0xee000c30:  0000 0000 a000 0000 8030 001b 0000 0000  *.........0......*
0xee000c40:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000c50:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000c60:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000c70:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000c80:  0000 0000 ee00 0000 81e0 0017 0000 0000  *................*
0xee000c90:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000ca0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000cb0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000cc0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000cd0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000ce0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000cf0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d00:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d10:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d20:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d30:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d40:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d50:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d60:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d70:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d80:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000d90:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000da0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000db0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000dc0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000dd0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000de0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee000df0:  0000 0000 0000 0000 8100 001e 0000 0000  *................*

PCIe4 Outbound ATMUs Dump:

NOTE: memory values are displayed in hexadecimal.
0xee270c00:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270c10:  8004 4027 0000 0000 0000 0000 0000 0000  *..@'............*
0xee270c20:  000a 0000 0000 0000 000a 0000 0000 0000  *................*
0xee270c30:  8004 401a 0000 0000 0000 0000 0000 0000  *..@.............*
0xee270c40:  000a 8000 0000 0000 000a 8000 0000 0000  *................*
0xee270c50:  8004 4017 0000 0000 0000 0000 0000 0000  *..@.............*
0xee270c60:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270c70:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270c80:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270c90:  0004 4027 0000 0000 0000 0000 0000 0000  *..@'............*
0xee270ca0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270cb0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270cc0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270cd0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270ce0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270cf0:  20f4 4017 0000 0000 0000 0000 0000 0000  * .@.............*
0xee270d00:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270d10:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270d20:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270d30:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270d40:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270d50:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270d60:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270d70:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270d80:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270d90:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270da0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270db0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270dc0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270dd0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270de0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*
0xee270df0:  0000 0000 0000 0000 0000 0000 0000 0000  *................*

[ufedor]

Please additionally provide the PEx4 Type 1 configuration header registers values.

[amarnathmb]

ufedor, Type 1 configuration header for PEx4:

vendor ID =                   0x1957
device ID =                   0x0830
command register =            0x0004
status register =             0x0010
revision ID =                 0x11
class code =                  0x0b
sub class code =              0x20
programming interface =       0x00
cache line =                  0x10
latency time =                0x00
header type =                 0x01
BIST =                        0x00
base address 0 =              0x00000000
base address 1 =              0x00000000
primary bus number =          0x00
secondary bus number =        0x01
subordinate bus number =      0x05
secondary latency timer =     0x00
IO base =                     0x01
IO limit =                    0x01
secondary status =            0x2000
memory base =                 0x0000
memory limit =                0x0000
prefetch memory base =        0x0001
prefetch memory limit =       0x0001
prefetch memory base upper =  0x00000000
prefetch memory limit upper = 0x00000000
IO base upper 16 bits =       0x0000
IO limit upper 16 bits =      0x0000
expansion ROM base address =  0x00000000
interrupt line =              0x2b
interrupt pin =               0x01
bridge control =              0x0000
Capabilities - Power Management
Capabilities - Message Signaled Interrupts: 0x50 control 0x88 Disabled, 64-bit, MME: 0 MMC: 4
        Address: 0000000000000000  Data: 0x0000
        Per-vector Mask: Unsupported
Capabilities - PCIe: Root Port, IRQ 0
        Device: Max Payload: 256 bytes, Extended Tag: 5-bit
                Acceptable Latency: L0 - <64ns, L1 - <1us
                Errors Enabled: Relaxed Ordering No Snoop
                Max Read Request 512 bytes
        Link: MAX Speed - 5.0Gb/s, MAX Width - by 1 Port - 0 ASPM - L0s
                Latency: L0s - >4us, L1 - >64us
                ASPM - Disabled, RCB - 128bytes
                Speed - 2.5Gb/s, Width - by 1
        Root Control Enabled:
Ext Capabilities - Advanced Error Reporting. 0x100. Version 2. AER Control: 0xa0
        Uncorrectable : Mask 0x400000. Severity 0x462030
        Uncorrectable Status:
        Correctable : Mask 0x6000.
        Correctable Status:
        HeaderLog:
        Error Source Identification: 0x0 0x0
Ext Capabilities - Secondary PCI Express Extended Capability. 0x148. Version 1

Please find the attched ConfigSpaceDump.txt file for Config space dump for all devices detected on PEx4.

[ufedor]

You wrote:

> FPGA DDR 128 MB" is the initial header show result after booting up,

> later i'm updating the BAR0 register with value 0xA0000000

Have you reconfigured the IDT switch also?

[amarnathmb]

ufedor‌,

If you see Type 1 configuration header for IDT Switch port 4 (Attahced file ConfigSpaceDump.txt), all limits are set properly.

memory base =                 0xa800
memory limit =                0xa7f0
prefetch memory base =        0xa001
prefetch memory limit =       0x9ff1