T-Series中的主题 Re: T1040: Problem with PCIe4 during secure boot

T1040: Problem with PCIe4 during secure boot

KSin — Tue, 30 Mar 2021 12:38:40 GMT

Hello,

There is an SSD disc connected over PCIe 4 on our T1040 board. I think, all the PCIe busses are configured correctly, because if we build the u-boot for non-secure mode, everything works correctly.

But if we try to build and run the u-boot in secure boot mode, the initialization ends with a memory trap. During debug, we have found that the memory trap is caused by reading from address 0xB0000000 (readl function) in ahci_host_init function in ahci.c module.

I try also to check the LAW and TLBCAM configurations but I did not found any significant difference which can have an impact on the PCIe.

Can you help me?

Both the u-boot (non-secure also secure) logs are attached and both the logs are containing the lists of law_table and tlb_table.

Regards,

Karel Sin

Re: T1040: Problem with PCIe4 during secure boot

ufedor — Tue, 30 Mar 2021 16:21:46 GMT

When secure boot is enabled, it will reinforce PAMU setting and it cannot be running in bypass mode.

From the Trust Architecture User Guide 2.0, section 3.1.2.1 PAMUs
#####
The core's MMU settings determine which memory ranges are accessible by each
domain, and the hypervisor prevents these settings from being altered by operating
system or application software.
In order to prevent system masters other than the cores from reading or writing sensitive
memory regions, the chip implements a number of I/O MMUs, known as peripheral
access management units or PAMUs. The PAMUs prevent internal and external DMAs
(non-CPU masters) from accessing memory for which they have not been granted explicit
access permission.
The hypervisor assigns each non-CPU master in the chip one or more logical I/O device
numbers (LIODNs). A non-CPU master asserts an LIODN (based on the ID of the
domain which requested the non-CPU master to perform an operation) on each bus
access. The PAMU grants or denies access to particular memory ranges based upon the
LIODN value. Non-CPU masters cannot alter the LIODN values they use for their
transactions, so this value serves to identify and authenticate the bus transaction as having
originated at one of a set of masters that share that particular LIODN value.
When secure boot is enabled, PAMUs block all external masters by default.
Authenticated software can change the PAMU access permissions to authorize external
masters later. This means that the chip cannot be booted as an agent when secure boot is
enabled.
#####

Please verify whether PAMU/LIODN setting is not in bypass mode.

Re: T1040: Problem with PCIe4 during secure boot

KSin — Thu, 01 Apr 2021 08:16:26 GMT

Hello, thank you for reply.

Unfortunately, I am not familiar with PAMU, so I don’t know, how to check if PAMU/LIODN is in bypass mode. But what I tried is to enable the access to PCIe. The easiest way I fount is to extend the construct_pamu_addr_table function as follow:

#ifdef CONFIG_SYS_PCIE1_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE1_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE1_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

#ifdef CONFIG_SYS_PCIE2_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE2_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE2_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

#ifdef CONFIG_SYS_PCIE3_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE3_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE3_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

#ifdef CONFIG_SYS_PCIE4_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE4_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE4_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

Unfortunately, it does not help. Is this enough to enable the PCIes in PAMU?

Never less, I don’t think, the u-boot crash is caused by PAMU. PAMU is supposed to protect the DMA, but the u-boot crashes on the readl function which is defined as:

#define readl(addr) (*(volatile u32 *) (addr))

(The address is 0xB0000000, i.e., the virtual address for PCIe4 defined in CONFIG_SYS_PCIE4_MEM_VIRT)

So, the DMA is not involved here and therefore also PAMU should not have any impact. Is this my idea correct?

Regards,

Karel

Re: T1040: Problem with PCIe4 during secure boot

KSin — Thu, 01 Apr 2021 12:25:10 GMT

Hello, thank you for reply.

#ifdef CONFIG_SYS_PCIE1_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE1_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE1_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

#ifdef CONFIG_SYS_PCIE2_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE2_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE2_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

#ifdef CONFIG_SYS_PCIE3_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE3_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE3_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

#ifdef CONFIG_SYS_PCIE4_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE4_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE4_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

Unfortunately, it does not help. Is this enough to enable the PCIes in PAMU?

Never less, I don’t think, the u-boot crash is caused by PAMU. PAMU is supposed to protect the DMA, but the u-boot crashes on the readl function which is defined as:

#define readl(addr) (*(volatile u32 *) (addr))

(The address is 0xB0000000, i.e., the virtual address for PCIe4 defined in CONFIG_SYS_PCIE4_MEM_VIRT)

So, the DMA is not involved here and therefore also PAMU should not have any impact. Is this my idea correct?

Regards,

Karel

Re: T1040: Problem with PCIe4 during secure boot

ufedor — Thu, 01 Apr 2021 16:58:18 GMT

In my previous reply, T1040RM.pdf for PAMU setting. e.g.7.3.31 PAMU Bypass Enable Register (DCFG_CCSR_PAMUBYPENR). Secure boot reenforce PAMU setting and blocking DMA, which will affect PCI operation as well. More info is in the T1040RM, section 28.3.4 LIODN

#####

Logical I/O Device Number (LIODN) is a mechanism where the memory subsystem identifies the LIODN that is assigned to an inbound PCI Express memory transaction based on its transaction ID (that is, RID), looks up its memory access permissions in a table, and performs the appropriate actions based on the outcome of the table look-up.

#####

To check/confirm the PAMU setting in uboot:

step 1: makeing sure you can read the Device Configuration/Pin Control memory map. Assume CCSRAB is at xfe000000, 0xfe0e00a0 is "Processor Version Register (DCFG_CCSR_PVR)" has a fix value of "8024_1021"

=> md 0xfe0e00a0

fe0e00a0: 80241021 85280011 00000040 00000000 .$.!.(.....@....

fe0e00b0: 00000000 00000000 00000000 00000000 ................

fe0e00c0: 00004000 00000000 00000000 00000000 ..@.............

fe0e00d0: 00000000 00000000 00000000 00000000 ................

If you are not getting "80241021" at 0xfe0e00a0, customer need to check their CCSRBAR address and swap the xFE000000 with whatever they are using.

Step 2: once you confirm they can read DCFG_CCSR_PVR, customer can read the PAMU setting by reading DCFG_CCSR_PAMUBYPENR @ xfe0e0604

=> md 0xfe0e0600

fe0e0600: 04000000 ff000000 00000000 00000000 ................

fe0e0610: 00000000 00000000 00000000 00000000 ................

fe0e0620: 00000000 00000000 00000000 00000000 ................

fe0e0630: 00000000 00000000 00000000 00000000 ................

In this case, it has the default SDK setting and in bypass mode. After customer change that, they will need assign LIODN. More info is available in the T1040RM, section

7.3.24 USB n Logical I/O Device Number register

(DCFG_CCSR_USBnLIODNR).......................................... 288

7.3.25 SD/MMC Logical I/O Device Number register

(DCFG_CCSR_SDMMCLIODNR)................................ 289

7.3.26 SATA n Logical I/O Device Number register

(DCFG_CCSR_SATAnLIODNR)..................................... 289

7.3.27 DIU Logical I/O Device Number register

(DCFG_CCSR_DIULIODNR).................................................290

7.3.28 TDM DMA Logical I/O Device Number register

(DCFG_CCSR_TDMDMALIODNR)..........................290

7.3.29 QUICC Engine Logical I/O Device Number register

(DCFG_CCSR_QELIODNR).................................291

7.3.30 DMA n Logical I/O Device Number register

(DCFG_CCSR_DMAnLIODNR)....................................... 291

Re: T1040: Problem with PCIe4 during secure boot

KSin — Wed, 07 Apr 2021 08:56:40 GMT

Hello,

I checked the DCFG_CCSR_PAMUBYPENR register, here is the result:

=> md 0xfe0e00a0

fe0e00a0: 80241021 85280011 00000040 00000000 .$.!.(.....@....

fe0e00b0: 00000000 00000000 00000000 00000000 ................

fe0e00c0: 00004000 00000000 00000000 00000000 ..@.............

fe0e00d0: 00000000 00000000 00000000 00000000 ................

fe0e00e0: 00000000 0000000f 00000000 00000000 ................

fe0e00f0: 00000000 00000000 00000000 00000000 ................

fe0e0100: 0c0d000c 0c000000 00000000 00440000 .............D..

fe0e0110: 66000002 00000012 e8306000 21000000 f........0`.!...

fe0e0120: 00000000 00000000 00000000 0003a160 ...............`

fe0e0130: 00000200 c03e5a85 00000000 00000000 .....>Z.........

fe0e0140: 00000000 00000000 00000000 00000000 ................

fe0e0150: 00000000 00000000 00000000 00000000 ................

fe0e0160: 00000000 00000000 00000000 00000000 ................

fe0e0170: 00000000 00000000 00000000 00000000 ................

fe0e0180: 00000000 00000000 00000000 00000000 ................

fe0e0190: 00000000 00000000 00000000 00000000 ................

=> md 0xfe0e0600

fe0e0600: 04000000 00000000 00000000 00000000 ................

fe0e0610: 00000000 00000000 00000000 00000000 ................

fe0e0620: 00000000 00000000 00000000 00000000 ................

fe0e0630: 00000000 00000000 00000000 00000000 ................

fe0e0640: 00000000 00000000 00000000 00000000 ................

fe0e0650: 00000000 00000000 00000000 00000000 ................

fe0e0660: 00000000 00000000 00000000 00000000 ................

fe0e0670: 00000000 00000000 00000000 00000000 ................

fe0e0680: 00000000 00000000 00000000 00000000 ................

fe0e0690: 00000000 00000000 00000000 00000000 ................

fe0e06a0: 00000000 00000000 00000000 00000000 ................

fe0e06b0: 00000000 00000000 00000000 00000000 ................

fe0e06c0: 00000000 00000000 00000000 00000000 ................

fe0e06d0: 00000000 00000000 00000000 00000000 ................

fe0e06e0: 00000000 00000000 00000000 00000000 ................

fe0e06f0: 00000000 00000000 00000000 00000000 ................

As you can see, the address 0xfe0e0600 contain zeroes, so the PAMU is not bypassed (this is logical, because in the chapter 7.3.31 is written: “On exiting RCW the register will be loaded with the inverse of secure_boot_en”).

According your reply, the LIODN needs to be assigned in this case, so I check the uboot code and found, that it is already done in the function set_liodns(). Here is an output of debug messages I added into the code (PCI is red merked):

KSI>> (liodn.c) set_liodns called

KSI>> (liodn.c) set_liodns calls set_liodn(liodn_tbl, liodn_tbl_sz: 16);

KSI>> (liodn.c) set_liodn:

compat: fsl,qman, id[0x3e, 0x0], num_ids: 1, compat_offset: 0xFFE318000 reg_offset: 0xFE318D08 configured

compat: fsl,bman, id[0x3f, 0x0], num_ids: 1, compat_offset: 0xFFE31A000 reg_offset: 0xFE31AD08 configured

compat: fsl,esdhc, id[0x228, 0x0], num_ids: 1, compat_offset: 0xFFE114000 reg_offset: 0xFE0E0530 configured

compat: fsl,pme, id[0x75, 0x0], num_ids: 1, compat_offset: 0xFFE316000 reg_offset: 0xFE316A0C configured

compat: fsl-usb2-mph, id[0x229, 0x0], num_ids: 1, compat_offset: 0xFFE210000 reg_offset: 0xFE0E0520 configured

compat: fsl-usb2-dr, id[0x22a, 0x0], num_ids: 1, compat_offset: 0xFFE211000 reg_offset: 0xFE0E0524 configured

compat: fsl,pq-sata-v2, id[0x22b, 0x0], num_ids: 1, compat_offset: 0xFFE220000 reg_offset: 0xFE0E0550 configured

compat: fsl,pq-sata-v2, id[0x22c, 0x0], num_ids: 1, compat_offset: 0xFFE221000 reg_offset: 0xFE0E0554 configured

compat: fsl,qoriq-pcie-v2.4, id[0x94, 0x0], num_ids: 1, compat_offset: 0xFFE240000 reg_offset: 0xFE240040 configured

compat: fsl,qoriq-pcie-v2.4, id[0xe4, 0x0], num_ids: 1, compat_offset: 0xFFE250000 reg_offset: 0xFE250040 configured

compat: fsl,qoriq-pcie-v2.4, id[0x134, 0x0], num_ids: 1, compat_offset: 0xFFE260000 reg_offset: 0xFE260040 configured

compat: fsl,qoriq-pcie-v2.4, id[0x184, 0x0], num_ids: 1, compat_offset: 0xFFE270000 reg_offset: 0xFE270040 configured

compat: fsl,elo3-dma, id[0x93, 0x0], num_ids: 1, compat_offset: 0xFFE100300 reg_offset: 0xFE0E0580 configured

compat: fsl,elo3-dma, id[0xe3, 0x0], num_ids: 1, compat_offset: 0xFFE101300 reg_offset: 0xFE0E0584 configured

compat: fsl,qe, id[0x22f, 0x0], num_ids: 1, compat_offset: 0xFFE140000 reg_offset: 0xFE0E0578 configured

compat: fsl,tdm1.0, id[0x230, 0x0], num_ids: 1, compat_offset: 0xFFE185000 reg_offset: 0xFE0E0574 configured

KSI>> (liodn.c) set_liodns calls set_liodn(sec_liodn_tbl, sec_liodn_tbl_sz: 24);

KSI>> (liodn.c) set_liodn:

compat: fsl,sec4.0-job-ring, id[0x1c6, 0x1CA], num_ids: 2, compat_offset: 0xFFE301000 reg_offset: 0xFE300014 configured

compat: fsl,sec-v4.0-job-ring, id[0x1c6, 0x1CA], num_ids: 2, compat_offset: 0xFFE301000 reg_offset: 0xFE300014 configured

compat: fsl,sec4.0-job-ring, id[0x1c7, 0x1CB], num_ids: 2, compat_offset: 0xFFE302000 reg_offset: 0xFE30001C configured

compat: fsl,sec-v4.0-job-ring, id[0x1c7, 0x1CB], num_ids: 2, compat_offset: 0xFFE302000 reg_offset: 0xFE30001C configured

compat: fsl,sec4.0-job-ring, id[0x1c8, 0x1CC], num_ids: 2, compat_offset: 0xFFE303000 reg_offset: 0xFE300024 configured

compat: fsl,sec-v4.0-job-ring, id[0x1c8, 0x1CC], num_ids: 2, compat_offset: 0xFFE303000 reg_offset: 0xFE300024 configured

compat: fsl,sec4.0-job-ring, id[0x1c9, 0x1CD], num_ids: 2, compat_offset: 0xFFE304000 reg_offset: 0xFE30002C configured

compat: fsl,sec-v4.0-job-ring, id[0x1c9, 0x1CD], num_ids: 2, compat_offset: 0xFFE304000 reg_offset: 0xFE30002C configured

compat: fsl,sec4.0-rtic-memory, id[0x1c5, 0x0], num_ids: 1, compat_offset: 0xFFE306100 reg_offset: 0xFE300064 configured

compat: fsl,sec-v4.0-rtic-memory, id[0x1c5, 0x0], num_ids: 1, compat_offset: 0xFFE306100 reg_offset: 0xFE300064 configured

compat: fsl,sec4.0-rtic-memory, id[0x225, 0x0], num_ids: 1, compat_offset: 0xFFE306120 reg_offset: 0xFE30006C configured

compat: fsl,sec-v4.0-rtic-memory, id[0x225, 0x0], num_ids: 1, compat_offset: 0xFFE306120 reg_offset: 0xFE30006C configured

compat: fsl,sec4.0-rtic-memory, id[0x226, 0x0], num_ids: 1, compat_offset: 0xFFE306140 reg_offset: 0xFE300074 configured

compat: fsl,sec-v4.0-rtic-memory, id[0x226, 0x0], num_ids: 1, compat_offset: 0xFFE306140 reg_offset: 0xFE300074 configured

compat: fsl,sec4.0-rtic-memory, id[0x227, 0x0], num_ids: 1, compat_offset: 0xFFE306160 reg_offset: 0xFE30007C configured

compat: fsl,sec-v4.0-rtic-memory, id[0x227, 0x0], num_ids: 1, compat_offset: 0xFFE306160 reg_offset: 0xFE30007C configured

compat: <NULL>, id[0x21d, 0x262], num_ids: 2, compat_offset: 0xFFE000000 reg_offset: 0xFE3000A4 configured

compat: <NULL>, id[0x21e, 0x263], num_ids: 2, compat_offset: 0xFFE000000 reg_offset: 0xFE3000AC configured

compat: <NULL>, id[0x21f, 0x264], num_ids: 2, compat_offset: 0xFFE000000 reg_offset: 0xFE3000B4 configured

compat: <NULL>, id[0x220, 0x265], num_ids: 2, compat_offset: 0xFFE000000 reg_offset: 0xFE3000BC configured

compat: <NULL>, id[0x221, 0x266], num_ids: 2, compat_offset: 0xFFE000000 reg_offset: 0xFE3000C4 configured

compat: <NULL>, id[0x222, 0x267], num_ids: 2, compat_offset: 0xFFE000000 reg_offset: 0xFE3000CC configured

compat: <NULL>, id[0x223, 0x268], num_ids: 2, compat_offset: 0xFFE000000 reg_offset: 0xFE3000D4 configured

compat: <NULL>, id[0x224, 0x269], num_ids: 2, compat_offset: 0xFFE000000 reg_offset: 0xFE3000DC configured

So the problem cant be here. Do you have any other idea?

Thanks,

Karel

Re: T1040: Problem with PCIe4 during secure boot

ufedor — Fri, 09 Apr 2021 17:13:45 GMT

One more question on the NSB_log.txt.
LAWBARH05: 0x00000000 LAWBARL05: 0x00000000 LAWAR05: 0x00000000
(EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBARH06: 0x00000000 LAWBARL06: 0x00000000 LAWAR06: 0x00000000
(EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBARH07: 0x00000000 LAWBARL07: 0x00000000 LAWAR07: 0x00000000
(EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBARH08: 0x00000000 LAWBARL08: 0x00000000 LAWAR08: 0x00000000
(EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBARH09: 0x00000000 LAWBARL09: 0x00000000 LAWAR09: 0x00000000
(EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBARH10: 0x00000000 LAWBARL10: 0x00000000 LAWAR10: 0x00000000
(EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBARH11: 0x00000000 LAWBARL11: 0x00000000 LAWAR11: 0x00000000
(EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBARH12: 0x00000000 LAWBARL12: 0x00000000 LAWAR12: 0x00000000
(EN: 0 TGT: 0x00 SIZE: 2 Bytes)

The LAWBAR for PCI are disabled (EN:0), and it is for "00h PCI Express 1". Since customer has the PCI4 working:
PCIe4: Root Complex, x1 gen2, regs @ 0xfe270000
0a:00.0 - 1b4b:9170 - Mass storage controller
PCIe4: Bus 09 - 0a

Can you provide the LAWBAR register dump after the "PCIe4" message?

It is also noteworthy that the LAWBAR for the SB_log.txt is different. LAWBARH01 to LAWBARH04 is repeated in LAWBARH05 to LAWBARH08. It is setting up DPAA (QMAN, BMAN, etc), not PCIe windows.
i.e.
LAWBARH01: 0x0000000f LAWBARL01: 0xf4000000 LAWAR01: 0x81800018
(EN: 1 TGT: 0x18 SIZE: 32 MiB)
LAWBARH02: 0x0000000f LAWBARL02: 0xf6000000 LAWAR02: 0x83c00018
(EN: 1 TGT: 0x3c SIZE: 32 MiB)
LAWBARH03: 0x0000000f LAWBARL03: 0x00000000 LAWAR03: 0x81d00015
(EN: 1 TGT: 0x1d SIZE: 4 MiB)
LAWBARH04: 0x0000000f LAWBARL04: 0xff800000 LAWAR04: 0x81f0000f
(EN: 1 TGT: 0x1f SIZE: 64 KiB)
LAWBARH05: 0x0000000f LAWBARL05: 0xf4000000 LAWAR05: 0x81800018
(EN: 1 TGT: 0x18 SIZE: 32 MiB)
LAWBARH06: 0x0000000f LAWBARL06: 0xf6000000 LAWAR06: 0x83c00018
(EN: 1 TGT: 0x3c SIZE: 32 MiB)
LAWBARH07: 0x0000000f LAWBARL07: 0x00000000 LAWAR07: 0x81d00015
(EN: 1 TGT: 0x1d SIZE: 4 MiB)
LAWBARH08: 0x0000000f LAWBARL08: 0xff800000 LAWAR08: 0x81f0000f
(EN: 1 TGT: 0x1f SIZE: 64 KiB)

So the uboot image is definitely different between the SB and normal boot. What version of SDK or upstream code you are based on? Is it compile from the same source code?

Re: T1040: Problem with PCIe4 during secure boot

KSin — Tue, 13 Apr 2021 07:47:36 GMT

Hello,

The only difference between the secure and non-secure version is in defconfig files (I added both in attachment). We are also not using any NXP SDK, this is just u-boot source code (downloaded from https://github.com/qoriq-open-source) builded by “make” commands.

But based on your point, I try to add the rows:

SET_LAW(CONFIG_SYS_PCIE1_MEM_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_PCIE_1),

SET_LAW(CONFIG_SYS_PCIE2_MEM_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_PCIE_2),

SET_LAW(CONFIG_SYS_PCIE3_MEM_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_PCIE_3),

SET_LAW(CONFIG_SYS_PCIE4_MEM_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_PCIE_4),

In the law_table[] and the memory trap problem is gone. I am interesting why it is necessary for the secure version only, but it is not important just now. Thank you for the help.

The second problem is, the SSD is still not found (because of timeout in ahci_device_data_io function). Now I am sure that this is caused by PAMU, because if I set the DCFG_CCSR_PAMUBYPENR bit to bypass the PAMU everything works.

I try to add the rows:

#ifdef CONFIG_SYS_PCIE1_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE1_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE1_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

#ifdef CONFIG_SYS_PCIE2_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE2_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE2_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

#ifdef CONFIG_SYS_PCIE3_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE3_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE3_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

#ifdef CONFIG_SYS_PCIE4_MEM_PHYS

tbl->start_addr[i] = CONFIG_SYS_PCIE4_MEM_PHYS;

tbl->size[i] = CONFIG_SYS_PCIE4_MEM_SIZE;

tbl->end_addr[i] = tbl->start_addr[i] + tbl->size[i] - 1;

i++;

#endif

into the construct_pamu_addr_table function and:

ret = config_pamu(&tbl, num_entries, PCIe1liodn);

ret = config_pamu(&tbl, num_entries, PCIe2liodn);

ret = config_pamu(&tbl, num_entries, PCIe3liodn);

ret = config_pamu(&tbl, num_entries, PCIe4liodn);

into the sec_config_pamu_table function (the LIONDN are defined as 148, 228, 308, 388 and configured in t1040_ids.c), but unfortunately it does not help.

What exactly are the correct steps to configure PAMU in the u-boot?

Regards,

Karel

Re: T1040: Problem with PCIe4 during secure boot

ufedor — Tue, 13 Apr 2021 17:02:30 GMT

In our SDK, the PAMU/LIODN is setup in linux. It is taking the input from dtsi file.

e.g.

...\linux-4.1/drivers/iommu/fsl_pamu.c

...\linux-4.1/arch/powerpc/boot/dts/fsl/t1040si-post.dtsi

Here is an example for PEX bus:

/linux-4.1/arch/powerpc/boot/dts/fsl/p5020si-post.dtsi: fsl,liodn-reg = <&guts 0x500>; /* PEX1LIODNR */

/linux-4.1/arch/powerpc/boot/dts/fsl/p5020si-post.dtsi: fsl,liodn-reg = <&guts 0x504>; /* PEX2LIODNR */

/linux-4.1/arch/powerpc/boot/dts/fsl/p5020si-post.dtsi: fsl,liodn-reg = <&guts 0x508>; /* PEX3LIODNR */

/linux-4.1/arch/powerpc/boot/dts/fsl/p5020si-post.dtsi: fsl,liodn-reg = <&guts 0x50c>; /* PEX4LIODNR */

Depends on your use case, as long as you enable PAMU, the LIODN is for resources isolation. The example in the SDK is just a convention, you can choose different scheme as they see fit.

Re: T1040: Problem with PCIe4 during secure boot

ufedor — Wed, 14 Apr 2021 15:39:58 GMT

I just found the uboot code you can reference.

https://source.codeaurora.org/external/qoriq/qoriq-components/u-boot/tree/arch/powerpc/cpu/mpc85xx/cpu_init.c

#include <asm/fsl_liodn.h>

#ifdef CONFIG_FSL_CORENET

set_liodns();

Re: T1040: Problem with PCIe4 during secure boot

KSin — Mon, 19 Apr 2021 13:37:51 GMT

Hello,

I just solved my problem, so let me make some summary of solution for somebody who probably has a similar one.

Problem:

I tried to configure the SATA SSD connected over PCIe – SATA bridge to the PCIe 4 in the u-boot. Everything worked fine if I build the u-boot for non-secure boot mode, but if I tried to build the u-boot for secure boot mode, the disk was not found, and the u-boot ends with memory trap caused by reading from address 0xB0000000 (0xB0000000 is configured address of PCIe 4) in ahci_host_init function.

Solution:

The memory tap was solved adding the LAW configuration for the PCIes:

SET_LAW(CONFIG_SYS_PCIE1_MEM_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_PCIE_1),

SET_LAW(CONFIG_SYS_PCIE2_MEM_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_PCIE_2),

SET_LAW(CONFIG_SYS_PCIE3_MEM_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_PCIE_3),

SET_LAW(CONFIG_SYS_PCIE4_MEM_PHYS, LAW_SIZE_256M, LAW_TRGT_IF_PCIE_4),

into the law_table[] defined in law.c file. (I am not sure, why it is necessary to do especially for secure boot configuration only.)

After this, the memory trap gone, but the disk was still not found even I added the PCIe LIODNs into the sec_config_pamu_table function (defined in pamu_table.c file).

Now I was known, that the communication was blocked by the PAMU (because if I bypass the PAMU by setting the bit BYP1 in DCFG_CCSR_PAMUBYPENR register), the disk was found.

So, I configured the PAMU interrupt to list out the PAMU status registers in case of access violation and found followed:

UDAD_UDAD = 0: NO Unauthorized Device Access Detected

AVS1_AV = 1: Access violation detected for LIODN (AVS1_LIODN): 388 (0x184)

AVS1_PDV = 0; No access violation while the PAMU enable (PE) bit is not set

AVS1_GCV = 0; No access violation during PAMU gate closed

AVS1_LAV = 010: LIODN access violation-Secondary PAACE index not within SPAACT

AVS1_WAV = 000: No window access violation

AVS1_APV = 0000: No access permission violation

AVS1_OTV = 00: No operation type violation

AVS2_OIV = 0: NO Operation Index Violation

Access Violation Address: address, in I/O address space, of the operation that encountered the error condition (AVAH, AVAL) [Hex]. 00000000 7FB03000

It means, the LIODN of PCIe4 was not added in the secondary PAACE table even it was added in the primary one.

Checking the code of the PAMU driver, I was found, that both, primary and secondary tables are defined with static size which is defined in fsl_pamu.h file as:

#define NUM_PPAACT_ENTRIES 512

#define NUM_SPAACT_ENTRIES 256

The size of 256 entries for secondary PAACE table was too small, and increasing this number solves the problem.

Regards,

Karel