AnsweredAssumed Answered

BAR 0 of PCIe EP is not accessible

Question asked by Amarnath MB on Aug 14, 2018
Latest reply on Aug 16, 2018 by Amarnath MB

Hi,

 

I have a T2080 custom board running vxWorks, on which the PCIe controller 4 (RC mode) (CCSBAR + 0x270000) is connected to the upstream port of IDT PCIe switch (89PES4T4). On one of the downstream ports of the switch I have Xilinx V7 FPGA as an endpoint. The FGPA exposes two BARs, ie. BAR0 for FPGA DDR access and BAR1 for FPGA CDMA access, during enumeration PCIe controller lists FGPA with Bus: 5,Dev: 0,Fun: 0 and shows BAR0 and BAR1 available inside the header.

 

PCIe Header Show gives :

vendor ID =                   0x10ee
device ID =                   0x7021
command register =            0x0007
status register =             0x0010
revision ID =                 0x00
class code =                  0x05
sub class code =              0x80
programming interface =       0x00
cache line =                  0x10
latency time =                0x00
header type =                 0x00
BIST =                        0x00
base address 0 =              0xf8000000 ----------> FPGA DDR 128 MB
base address 1 =              0xa8000000  ----------> FPGA CDMA 16 MB
base address 2 =              0x00000000
base address 3 =              0x00000000
base address 4 =              0x00000000
base address 5 =              0x00000000
cardBus CIS pointer =         0x00000000
sub system vendor ID =        0x10ee
sub system ID =               0x0007
expansion ROM base address =  0x00000000
interrupt line =              0x2b
interrupt pin =               0x01
min Grant =                   0x00
max Latency =                 0x00
Capabilities - Power Management  
Capabilities - Message Signaled Interrupts: 0x48 control 0x82 Disabled, 64-bit, MME: 0 MMC: 1
    Address: 0000000000000000  Data: 0x0000
    Per-vector Mask: Unsupported  
Capabilities - PCIe: Endpoint, IRQ 0
    Device: Max Payload: 256 bytes, Phantom Funcs 1 msb, Extended Tag: 8-bit
        Acceptable Latency: L0 - <64ns, L1 - <1us
        Errors Enabled: Relaxed Ordering No Snoop
        Max Read Request 512 bytes
    Link: MAX Speed - 5.0Gb/s, MAX Width - by 1 Port - 0 ASPM - L0s
        Latency: L0s - >4us, L1 - >64us
        ASPM - Disabled, RCB - 64bytes
        Speed - 2.5Gb/s, Width - by 1
Ext Capabilities - Device Serial Number. 0x100. Version 1  
    Serial Number: 0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x0

 

LAW BAR settings

----------------------------------------------------------------------------
LAW        Address            Status      Size               Interface
----------------------------------------------------------------------------
LAW00    0x00f8000000  Enabled   0x08000000  Integrated Flash controller
LAW01    0x00e0000000  Enabled   0x02000000  Buffer Manager (control)
LAW02    0x00e2000000  Enabled   0x02000000  Queue Manager (control)
LAW03    0x00a0000000  Enabled   0x10000000  PCI Express 4
LAW08    0x00ee000000  Enabled   0x01000000  CCSR
LAW31    0x0000000000  Enabled   0x80000000  DDR SDRAM
----------------------------------------------------------------------------

 

ATMU Register Settings (CCSBAR : 0xEE000000)
WIN00  

OTAR    : 0xee270c00 0x00000000   

OTEAR : 0xee270c04 0x00000000    

OWAR  : 0xee270c10 0x80044027 

 
WIN01  (128 MB)

OTAR    : 0xee270c20 0x000a0000   

OTEAR : 0xee270c24 0x00000000   

OWBAR: 0xee270c28 0x000a0000   

OWAR  : 0xee270c30 0x8004401a

   
WIN02  (16 MB)

OTAR    : 0xee270c40 0x000a8000   

OTEAR : 0xee270c44 0x00000000   

OWBAR: 0xee270c48 0x000a8000   

OWAR  : 0xee270c50 0x80044017 

 
WIN03  

OTAR    : 0xee270c60 0x00000000   

OTEAR : 0xee270c64 0x00000000   

OWBAR: 0xee270c68 0x00000000   

OWAR  : 0xee270c70 0x00000000

 

After booting i'm updating BAR0 of PCIe device (5,0,0) with value 0xA0000000, after that when i try to access memory 0xA0000000 (FPGA DDR) my system hangs. But when i try to access 0xA8000000 (FPGA CDMA) it gives proper value.

 

Header show after updating BAR0 value:

vendor ID =                   0x10ee
device ID =                   0x7021
command register =            0x0007
status register =             0x0010
revision ID =                 0x00
class code =                  0x05
sub class code =              0x80
programming interface =       0x00
cache line =                  0x10
latency time =                0x00
header type =                 0x00
BIST =                        0x00
base address 0 =              0xa0000000
base address 1 =              0xa8000000
base address 2 =              0x00000000
base address 3 =              0x00000000
base address 4 =              0x00000000
base address 5 =              0x00000000
cardBus CIS pointer =         0x00000000
sub system vendor ID =        0x10ee
sub system ID =               0x0007
expansion ROM base address =  0x00000000
interrupt line =              0x2b
interrupt pin =               0x01
min Grant =                   0x00
max Latency =                 0x00
Capabilities - Power Management  
Capabilities - Message Signaled Interrupts: 0x48 control 0x82 Disabled, 64-bit, MME: 0 MMC: 1
    Address: 0000000000000000  Data: 0x0000
    Per-vector Mask: Unsupported  
Capabilities - PCIe: Endpoint, IRQ 0
    Device: Max Payload: 256 bytes, Phantom Funcs 1 msb, Extended Tag: 8-bit
        Acceptable Latency: L0 - <64ns, L1 - <1us
        Errors Enabled: Relaxed Ordering No Snoop
        Max Read Request 512 bytes
    Link: MAX Speed - 5.0Gb/s, MAX Width - by 1 Port - 0 ASPM - L0s
        Latency: L0s - >4us, L1 - >64us
        ASPM - Disabled, RCB - 64bytes
        Speed - 2.5Gb/s, Width - by 1
Ext Capabilities - Device Serial Number. 0x100. Version 1  
    Serial Number: 0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x0

 

To experiment, I modified BAR0 value with 0xA8000000 and BAR1 address with 0xA0000000, Now i can access FPGA DDR memory through 0xA8000000, but accessing 0xA0000000 hangs my system.

 

Header  show after interchanging BAR0 and BAR 1 values:

vendor ID =                   0x10ee
device ID =                   0x7021
command register =            0x0007
status register =             0x0010
revision ID =                 0x00
class code =                  0x05
sub class code =              0x80
programming interface =       0x00
cache line =                  0x10
latency time =                0x00
header type =                 0x00
BIST =                        0x00
base address 0 =              0xa8000000
base address 1 =              0xa0000000
base address 2 =              0x00000000
base address 3 =              0x00000000
base address 4 =              0x00000000
base address 5 =              0x00000000
cardBus CIS pointer =         0x00000000
sub system vendor ID =        0x10ee
sub system ID =               0x0007
expansion ROM base address =  0x00000000
interrupt line =              0x2b
interrupt pin =               0x01
min Grant =                   0x00
max Latency =                 0x00
Capabilities - Power Management  
Capabilities - Message Signaled Interrupts: 0x48 control 0x82 Disabled, 64-bit, MME: 0 MMC: 1
    Address: 0000000000000000  Data: 0x0000
    Per-vector Mask: Unsupported  
Capabilities - PCIe: Endpoint, IRQ 0
    Device: Max Payload: 256 bytes, Phantom Funcs 1 msb, Extended Tag: 8-bit
        Acceptable Latency: L0 - <64ns, L1 - <1us
        Errors Enabled: Relaxed Ordering No Snoop
        Max Read Request 512 bytes
    Link: MAX Speed - 5.0Gb/s, MAX Width - by 1 Port - 0 ASPM - L0s
        Latency: L0s - >4us, L1 - >64us
        ASPM - Disabled, RCB - 64bytes
        Speed - 2.5Gb/s, Width - by 1
Ext Capabilities - Device Serial Number. 0x100. Version 1  
    Serial Number: 0x0  0x0  0x0  0x0  0x0  0x0  0x0  0x0

 

Can anyone help me out? What can be the issue here?

Outcomes