Problem configuring PCIe inbound windows on Linux Kernel

[masasi]

Hi,

I am trying to configure PCIe communications on my Freescale P1012 board by means of your driver fsl-pci.c

I need to configure inbound windows at addresses 0x3fe00000(PEX1) and 0x3fd00000(PEX2) but it seems that the function “setup_pci_atmu” sets a single inbound window at address 0x00000000, Am I correct? How can I proceed to configure these windows?

I have an uboot that configures my PCIe inbound and outbound windows. If I comment the setup of PCIe inbound memory window on the “setup_pci_atmu” function in the kernel(I have attached the modified fsl_pci.c file) so that the uboot configuration of the inbound windows is not modified and I start receiving PCIe messages, some processes running over the linux kernel (not using PCIe communications) give an exception such as:

maindiagupd[167]: unhandled signal 11 at 00000000 nip 0feed38c lr 10023768 code 30001

busybox[215]: unhandled signal 4 at 100a7944 nip 100a7944 lr 1000930c code 30001

If no PCIe messages are being received, the previous processes running over kernel work properly, so I suppose I need to include some information of the inbound windows for the Linux kernel. Could you please help me to understand how can I configure the addresses of the PCIe inbound windows on the driver?

Thanks in advance,

Maria

Original Attachment has been moved to: fsl_pci.c.zip

[威杨]

i use mpc8039, i meet pci inbound regeiste, please give me some help, 

For example:

  pci->piw[0].pitar = 0x00000c00;  //DDR_addr = 0x00c0_0000

  pci->piw[0].piwbar = 0x000a0000;  //PCI_addr (base) = 0xa000_0000

  pci->piw[0].piwbear = 0x00000000; 

  pci->piw[0].piwar = 0x80044013;  //size=1MB 

i regeister inbound ,but not working , no matter pitar how to set ,can see DDR_addr = 0x0 working,can you tell me how to set  these regerister.

THANKS

[yipingwang]

Hello masasi,

The default inbound window(BAR0) configured as 1M is used for inbound memory transaction to access CCSR memory mapped registers.

If the external PCI device is required to access the DDR of the target board, it is needed to set corresponding inbound window(for example BAR1) which is used to translate PCIe address to the local DDR address.

In the kernel file ./arch/powerpc/sysdev/fsl_pci.c, you could refer to the section ">4G of memory setup second PCI inbound window" to setup inbound memory windows in the function setup_pci_atmu according to your requirement.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[masasi]

Thank you for your help!!

I have setup the inbound memory window as you suggested on the setup_pci_atmu function but I do not receive any message, Do I have to configure the inbound memory windows on a certain range?

This is the code I included on setup_pci_atmu function:

/* PCIe can overmap inbound & outbound since RX & TX are separated */

if (early_find_capability(hose, 0, 0, PCI_CAP_ID_EXP)) {

/* Size window to exact size if power-of-two or one size up */

if ((1ull << mem_log) != mem) {

mem_log++;

if ((1ull << mem_log) > mem)

  pr_info("%s: Setting PCI inbound window "

"greater than memory size\n", name);

}

piwar |= ((mem_log - 1) & PIWAR_SZ_MASK);

/* Setup inbound memory window */

if(!strcmp("/pcie@ff70a000",name)){

out_be32(&pci->piw[win_idx].pitar,  0x0003FE00);

out_be32(&pci->piw[win_idx].piwbar, 0x000AD000);

out_be32(&pci->piw[win_idx].piwar,  piwar);

win_idx--;

hose->dma_window_base_cur = 0x00000000;

hose->dma_window_size = (resource_size_t)sz;

  /* Setup inbound memory window */

out_be32(&pci->piw[win_idx].pitar,  0x000B0100);

out_be32(&pci->piw[win_idx].piwbar, 0x000AB000);

out_be32(&pci->piw[win_idx].piwar,  piwar);

win_idx--;

hose->dma_window_base_cur = 0xB0100000;

hose->dma_window_size = (resource_size_t)sz;

}

if(!strcmp("/pcie@ff709000",name)){

out_be32(&pci->piw[win_idx].pitar,  0x0003FD00);

out_be32(&pci->piw[win_idx].piwbar, 0x000BD000);

out_be32(&pci->piw[win_idx].piwar,  piwar);

win_idx--;

hose->dma_window_base_cur = 0x00000000;

hose->dma_window_size = (resource_size_t)sz;

/* Setup inbound memory window */

out_be32(&pci->piw[win_idx].pitar,  0x000A0100);

out_be32(&pci->piw[win_idx].piwbar, 0x000BA000);

out_be32(&pci->piw[win_idx].piwar,  piwar);

win_idx--;

hose->dma_window_base_cur = 0x00000000;

hose->dma_window_size = (resource_size_t)sz;

}

/*

* if we have >4G of memory setup second PCI inbound window to

* let devices that are 64-bit address capable to work w/o

* SWIOTLB and access the full range of memory

*/

Then on the kernel user space I try to map the inbound windows by using mmap command as follows:

   int fd;

    void *where;

    // Open PCIe devices

fd=open("/dev/mem",O_RDWR);

where=mmap(0,0x100000,PROT_READ|PROT_WRITE,MAP_SHARED,fd,0x3fe00000);

    if(where==(void *)0xffffffff)

    {

close(fd);

printf("Cannot open PCIe uA-uD communications\n");

exit(-1);

    }

    pcie_inboundA=(void *) where;

}

And then read inbound window as:

Memcpy(rx_buffer, pcie_inboundA,size);

And no message is received. Is there a problem with the inbound window address range that I am using??

Thanks again for your help!

[yipingwang]

Hello masasi,

Probably there is problem with inbound ATMU configuration, please check the following.

PEXITAR1 (PCI Express inbound translation address register), it contains the starting point of translated internal platform address.

PEXIWBAR1 (PCI Express inbound window base address register ) i.e. contains the PCI Express starting point.

PEXIWAR1 - PCI Express inbound window attributes register, define the window sizes to translate and other attributes for the translations.

PEXIWAR1[IWS] Inbound window size.

For system memory

PEXIWAR1[TRGT]=0b1111 //Local memory space

For example:

  pci->piw[0].pitar = 0x00000c00;  //DDR_addr = 0x00c0_0000

  pci->piw[0].piwbar = 0x000a0000;  //PCI_addr (base) = 0xa000_0000

  pci->piw[0].piwbear = 0x00000000; 

  pci->piw[0].piwar = 0x80044013;  //size=1MB 

Please try whether it would be fine if configure these inbound windows in u-boot.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[masasi]

Thank you for your help!

Originally, I had the PCIe inbound windows configured at u-boot and did not allow the kernel to change this configuration, the communications worked properly so I am sure the inbound ATMU configuration I am using is ok. The problem is that although I am able to receive PCIe messages on the kernel user space, at some point I get a segmentation fault or illegal instruction exception (unhandled signal 4 or unhandled signal 11) on some of the processes I am running(These exceptions are caused by the reception of the PCIe messages because if I stop sending those messages, the processes work fine).

Maybe this is a problem regarding DDR configuration on the kernel?? I have a 1G DDR RAM configured by u-boot from address 0x00000000 to 0x40000000 and the PCIe inbound windows go from 0x3FD00000 to 0x3FE200000(I reserved this memory range on u-boot). When the kernel boots I do not pass the DDR size through bootargs or configure DDR memory size using the device tree, does the kernel figure out what size is the DDR? Also, do I need to reserve PCIe inbound window memory range in the kernel as well so the kernel does not try to use the memory space I reserved for PCIe incoming messages for other purposes??If so, how can I reserve it?

Thanks again!

[yipingwang]

Hello masasi,

This block of memory should be reserved in the EP device driver.

In addition, it seems that a default inbound window has been configured in u-boot to let the PCIe EP to access the whole memory map(including DDR) of the target, so in Kernel PCIe EP can access DDR memory directly.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------