PCIe NIC's do not work with Topaz Hypervisor with 4GB of DDR in each Linux Guest OS

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

PCIe NIC's do not work with Topaz Hypervisor with 4GB of DDR in each Linux Guest OS

725 Views
paterb
Contributor II

I have a T2080QDS board and I purchased a 16 GB DDR chip for it.  I am allocating 4 GB of DDR RAM to each Linux Guest OS.  When I do this, my PCIe Ethernet NICs no longer function.  They do not receive the MSI interrupt.  If I allocate 2 GB to each Linux Guest, then it works fine.

I believe I've traced the issue to the fsl_pci.c file, function setup_pci_atmu().  In this function, it is checking the msi-address-64 property, which is set because I am running Topaz hypervisor.  But since we have to extend the DDR ATMU to accommodate the MSIIR register, we now have greater than 4GB of DDR allocated which causes us to setup a 2nd PCI inbound window later in the function.  I'm assuming this is where we have issues with the Hypervisor model, as I never see the MSI interrupt happen in linux (I have debug in the fsl_msi_cascade function).

This works fine if I just boot Linux and have 16GB of RAM allocated to it.  It only doesn't work in the Hypervisor scenario.

Anyone have any ideas what might be the issue?

Thanks.

Pat

Tags (3)
0 Kudos
1 Reply

469 Views
paterb
Contributor II

I finally got this to work by adding the following ATMU window shown in bold below in the setup_pci_atmu() function

in fsl_pci.c file:

..... (code removed from beginning of function)

/*

* if we have >4G of memory setup second PCI inbound window to

* let devices that are 64-bit address capable to work w/o

* SWIOTLB and access the full range of memory

*/

  if (sz != mem) {

  mem_log = ilog2(mem);

  /* Size window up if we dont fit in exact power-of-2 */

  if ((1ull << mem_log) != mem)

  mem_log++;

  piwar = (piwar & ~PIWAR_SZ_MASK) | (mem_log - 1);

  /* Setup inbound memory window */

  out_be32(&pci->piw[win_idx].pitar,  0x00000000);

  out_be32(&pci->piw[win_idx].piwbear,

  pci64_dma_offset >> 44);

  out_be32(&pci->piw[win_idx].piwbar,

  pci64_dma_offset >> 12);

  out_be32(&pci->piw[win_idx].piwar,  piwar);

  /* Setup inbound memory window  for MSI interrupt

  * when there is 4GB of DDR RAM allocated to a core

  */

  win_idx--;

  out_be32(&pci->piw[win_idx].pitar,  0x00000000);

  out_be32(&pci->piw[win_idx].piwbear, 0);  // I assume this line and the next one equate to the address of

  out_be32(&pci->piw[win_idx].piwbar, 1);    // 0x1_0000_0000 ????

  out_be32(&pci->piw[win_idx].piwar,  piwar);

  /* 

  * install our own dma_set_mask handler to fixup dma_ops

  * and dma_offset

  */

  ppc_md.dma_set_mask = fsl_pci_dma_set_mask;

  pr_info("%s: Setup 64-bit PCI DMA window\n", name);

  }

.....

0 Kudos