i.MX8MP PCIe EP multi-function device

[kstruczy]

Hi,

I'm working with the i.MX8MP evaluation board:

https://www.nxp.com/design/development-boards/i-mx-evaluation-and-development-boards/evaluation-kit-...

My setup consists of two boards connected by the 2 x M.2 Key A+E to PCIe x1 cables and PCIe x1 extension cable (Tx to Rx signal swap).

On both boards, I'm running the latest linux-imx kernel release (tag lf-5.10.52-2.1.0).

I successfully ran ep-rc tests using the existing pci-epf-test function driver and pci_endpoint_test host driver.

I'm currently trying to enable the multi-function support (two physical functions). Based on the specs I read, I understand that the controller supports it.

I added the max-functions property to the device tree:

&pcie_ep{
    status = "okay";
    max-functions = /bits/ 8 <2>;
};

and tried to create two functions using pci-epf-test function driver.

I'm getting this error while trying to link the second function to the controller:

root@imx8mpevk:/sys/kernel/config/pci_ep# ln -s ./functions/pci_epf_test/func2 ./controllers/33800000.pcie_ep/
[ 19.335624] imx6q-pcie 33800000.pcie_ep: No free inbound window
[ 19.341586] pci_epf_test pci_epf_test.1: Failed to set BAR0
ln: failed to create symbolic link './controllers/33800000.pcie_ep/func2': Invalid argument

I think that to make it work I have to set the num-ib-windows, num-ob-windows and range for the ATU address space correctly, but I know too little about the PCIe internals and I'm lacking documentation.

My questions are:

1. Does the controller support the multi-function device?

2. If it does, any suggestions how to enable it?

Any help would be greatly appreciated.

Thanks,

Krzysztof

[Zhiming_Liu]

1. Does the controller support the multi-function device?

--> Yes ,it's written in i.MX 8M Plus Applications Processor Reference Manual.pdf

If you don not have it ，you should contact sales to get it or download from here:https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-proces...

[kstruczy]

Hi @Zhiming_Liu ,

Thanks for a quick reply.

I do have the reference manual, but I didn't find there much help in terms of enabling a multi-function device in Linux. Can you please point me to the relevant bits? Is it already supported by the drivers?

Thanks,

Krzysztof

[Zhiming_Liu]

I found it in drivers/pci/controller/dwc/pcie-designware-ep.c

Write hdr->cache_line_size to PCI_CACHE_LINE_SIZE

static int dw_pcie_ep_write_header(struct pci_epc *epc, u8 func_no,
				   struct pci_epf_header *hdr)
{
	struct dw_pcie_ep *ep = epc_get_drvdata(epc);
	struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
	unsigned int func_offset = 0;

	func_offset = dw_pcie_ep_func_select(ep, func_no);

	dw_pcie_dbi_ro_wr_en(pci);
	dw_pcie_writew_dbi(pci, func_offset + PCI_VENDOR_ID, hdr->vendorid);
	dw_pcie_writew_dbi(pci, func_offset + PCI_DEVICE_ID, hdr->deviceid);
	dw_pcie_writeb_dbi(pci, func_offset + PCI_REVISION_ID, hdr->revid);
	dw_pcie_writeb_dbi(pci, func_offset + PCI_CLASS_PROG, hdr->progif_code);
	dw_pcie_writew_dbi(pci, func_offset + PCI_CLASS_DEVICE,
			   hdr->subclass_code | hdr->baseclass_code << 8);
	dw_pcie_writeb_dbi(pci, func_offset + PCI_CACHE_LINE_SIZE,
			   hdr->cache_line_size);
	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_VENDOR_ID,
			   hdr->subsys_vendor_id);
	dw_pcie_writew_dbi(pci, func_offset + PCI_SUBSYSTEM_ID, hdr->subsys_id);
	dw_pcie_writeb_dbi(pci, func_offset + PCI_INTERRUPT_PIN,
			   hdr->interrupt_pin);
	dw_pcie_dbi_ro_wr_dis(pci);

	return 0;
}

[kstruczy]

Hi @Zhiming_Liu ,

Is the cache_line_size relevant? According to the spec it has no effect on any PCIe device behaviour.

Did you maybe mean the multi_func bit from the header type byte? I tried setting it but sadly I don't see any improvement.

Regarding the error I'm getting. It see that each BAR requires one iATU region, which makes sense, and if I understand well there are only four that can be defined for the imx8mp. The test driver tries to setup all 6 BARs (or less if there are 64-bit BARs) and it runs out of the iATU regions for the second function. I decreased the number of BARs to set up to two and I successfully created the second function - it looks more like a hack than a solution to me though. Also, in that case the "host" board detects only the second defined function.

Do you have any suggestion how to solve this properly? Is there by chance an example from NXP how to setup a multifunction PCIe endpoint?

Thanks,

Krzysztof

[Zhiming_Liu]

@kstruczy 

Can you contact local NXP FAE? This kind of case should be handled by FAE, they will post your issue to Application Team, this will be very effective to solve your issue.

[kstruczy]

Hi @Zhiming_Liu ,

Sadly, I didn't get a contact to the local FAE yet.

Can you please tell me where to find the configuration space function offset for the func_conf_select?

Thanks.

Krzysztof

[Zhiming_Liu]

You may need find it by youself, the community can only provide limit support.

[kstruczy]

Hi @Zhiming_Liu ,

Can you please tell me how to find one? I'm located in Munich, Germany.

In the meantime I ran some more tests. As I described in my previous post, I limited the number of BARs to two, so that I could distribute four iATU regions across two functions. However, the host device detects only one function. The reason for that is that the device id and vendor id is set only for the first function - at least as the host device sees it.

I checked it from the EP side and it looks like the func_conf_select() function, that returns the per function offset, is not implemented for the imx8mp. I implemented it and tried setting function offset to 0x1000 - the reference manual says:

"The controller has 4096(0x1000) bytes of register PCIe configuration and port logic
space per function."

and I assumed that it may be just that.

Unfortunately this didn't work either and host still reads 0xffffffff as vendor id and device id for the second function.

Can you please tell me if 0x1000 is the correct function offset? The address space is much larger so I assume it can be something completely different, but I can't find it in the spec.

Thanks,

Krzysztof