iMX8 Quad Plus SMMU and BL31 questions

[flobro]

Does the imx8 Quad Max use the BL31 code, or SMMU or both?

We are investigating the "DMA Attack" on the PCIe bus using pcileech, which was successful in our imx8M Plus, and now successful against the Quad Max.

I am looking for information to successfully block the DMA attack using SMMU, but have not found better information. Please educate this person on the important security issues or our product may never make it out of review.

[Harvey021]

Hi @flobro 

Have you checked that SMMU is enabled for PCIe in your dts?

In our default imx8qm dts settings, SMMU is not enabled for PCIe. You can check imx8qm-ss-hsio.dtsi for more details.

For masters that enabled SMMU, there should be "iommus = <&smmu 0x13 0x7f80>;" in the device's node. The 0x13 is the SMMU stream ID, should be different for each master.

Regards

Harvey

[flobro]

So does the enabling of the stream prevent other devices than the SSD from asking for DMA addresses of any space it wants?

[Harvey021]

The SMMU will provide:

Translation

The addresses supplied by the client device are translated from the virtual address space into the system’s physical address space.

Protection

Operations from the client device might be prevented by the permissions held in the translation tables. You can prohibit a device to read, write, execute, or make any access to particular regions of memory.

Isolation

Transactions from one device can be differentiated from those of another device, even if both devices share a connection to the SMMU. This means that the translation and protection properties can be applied differently for each device. Each device might have its own private translation tables, or might share them with other devices, as appropriate to the application.

For more details, please check https://developer.arm.com/documentation/109242/0100/What-an-SMMU-does

Regards

Harvey

[flobro]

I added the "iommus = <0x72 0x5 0x7f80>;" to my pcie dts and when the boot starts, I get:

sc_rm_set_master_sid: resource:153 sid:5: res:3
fail set_master_sid rsrc=153 sid=0x5 err=-22

I am having an issue figuring out what is missing. I assume "sid=5" refers to my modified entry.

Any ideas?

Is there a way to determine what rcrc=153 is?

[Harvey021]

Please try to add iommus = <&smmu 0x13 0x7f80>; in the pcie's device node.

The resource number definition can be found in SCFW code platform/main/types.h.

Regards

Harvey

[flobro]

This has been added...:

&pciea{
ext_osc = <1>;
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_pciea>;
reset-gpio = <&lsio_gpio3 14 GPIO_ACTIVE_LOW>;
iommus = < &smmu 0x05 0x7f80>;
status = "okay";
};

But I still see the error fro uboot:

sc_rm_set_master_sid: resource:153 sid:5: res:3
fail set_master_sid rsrc=153 sid=0x5 err=-22

[Harvey021]

Please let me know which version of BSP you are testing with. 

From the code in Linux, PCIE_A's resource number is 152, please check why the resource number in your log is 153.

./include/dt-bindings/firmware/imx/rsrc.h:169:#define IMX_SC_R_PCIE_A                   152

Regards

Harvey

[flobro]

Excellent question.

#define SC_R_PCIE_A      152U
#define SC_R_SERDES_0 153U

If I knew, I would not be asking. Does the "SERDES" need access (I tried, no difference)?
When I remove the "iommus = < &smmu 0x05 0x7f80>;", the error from uboot goes away.

The statement when added does prevent the PCI DMA attack fro reading any memory, but this make me think there are other components that need to be added (although SERDES is a HDMI component, so I do not understand why the error)

[Harvey021]

Please try to add the followings in your pcie's dts node.

iommus = <&smmu 0x8 0x7f80>;
fsl,sc_rsrc_id = <IMX_SC_R_PCIE_A>;

Also please add "define DEBUG" in the head of uboot-imx/arch/arm/mach-imx/imx8/fdt.c to open the debug print, so that we can get more info when the issue happens in uboot.

Regards

Harvey

[flobro]

The change of pciea to SID 8 appears to eliminate the issue. The question is, "Are the SIDs predefined in the SC setup?" 

If not, how do you know what you can use without going through the whole build/ test process

node mmc@5b010000 iommus proplen 12
configure node mmc@5b010000 sid 0x11 rsrc=248
set_master_sid rsrc=248 sid=0x11 err=0
node mmc@5b020000 iommus proplen 12
configure node mmc@5b020000 sid 0x11 rsrc=249
set_master_sid rsrc=249 sid=0x11 err=0
node mmc@5b030000 iommus proplen 12
configure node mmc@5b030000 sid 0x11 rsrc=250
set_master_sid rsrc=250 sid=0x11 err=0
node ethernet@5b040000 iommus proplen 12
configure node ethernet@5b040000 sid 0x12 rsrc=251
set_master_sid rsrc=251 sid=0x12 err=0
node ethernet@5b050000 iommus proplen 12
configure node ethernet@5b050000 sid 0x12 rsrc=252
set_master_sid rsrc=252 sid=0x12 err=0
node usb@5b110000 iommus proplen 12
configure node usb@5b110000 sid 0x4 rsrc=262
set_master_sid rsrc=262 sid=0x4 err=0
node usb@5b120000 iommus proplen 12
node bus@5f000000 iommus proplen 12
node pcie@0x5f000000 iommus proplen 12
configure node pcie@0x5f000000 sid 0x8 for 1 resources
set_master_sid rsrc=152 sid=0x8 err=0
node sata@5f020000 iommus proplen 12
configure node sata@5f020000 sid 0x13 for 1 resources
set_master_sid rsrc=170 sid=0x13 err=0

[Harvey021 · Accepted solution]

The SID is not predefined in the SOC. But users need to make sure that each bus master has different SID. In our default dts settings, we have already assigned SID to some masters, like mmc and ethernet. When you try to add new SID for a master, please note to avoid those SID that has already been used.

I think the previous issue that you met is not related to SID, it's more related to missing "fsl,sc_rsrc_id = <IMX_SC_R_PCIE_A>;"

Regards

Harvey

[flobro]

I do appreciate the insight. I added this to the "base" pceia entry in the dts and changed my SID back to 5 just to verify and it is like magic, no more error.

I also see in my dts that the SATA entry was the only other one with the "fsl,sc_rsrc_id"  tag. So the the remaining question is:

How are the other masters with "iommu" entries not getting errors (how are the SIDS defined when not in the device tree - ethernet, eMMC, USB)?

Thanks for the help, I will accept this as the solution - but would appreciate an answer to this last question.

[Harvey021]

Please refer to imx8qm-ss-conn.dtsi

Regards

Harvey

[flobro]

I see the "iommus = <&smmu ..." statements in that file, but there are no other "fsl,sc_rsrc_id" statements. So how do you know what the SIDs are for these?

[Harvey021]

fsl,sc_rsrc_id is needed for PCIE, but not for others. You can check the code in uboot, arch/arm/mach-imx/imx8/fdt.c.

Regards

Harvey

[flobro]

This was successful in blocking the "pcileech", but introduced a new issue where our pcie (NVME) SSD is blocked.

The kernel throws:

arm-smmu 51400000.iommu: Unhandled context fault: fsr=0x402, iova=0x90081000, fsynr=0x130001, cbfrsynra=0x1445, cb=2

How do I go about making an exception for the SSD plugged into the PCIE socket? I do not see anything in the device tree that relates to SSD or NVME on PCIE

[Harvey021]

Will reply back to you in your new ticket.

Regards

Harvey

[Harvey021]

Hi @flobro 

Sorry to reply to you with delay. I'm working on the issue with internal team.

Regards

Harvey

[flobro]

We use a SSD on the mini PCIe on the Quad Max, and when the SSD is removed and a "Screamer" is connected, the DMA hack is successful.

So the question is,  Which layers of this incredible stack would be involved in allowing the SSD to operate, but nothing else on the PCIe bus? OPTEE? ATF? Or is it even possible?

[flobro]

I see topics including SCFW, Arm trusted Firmware, and OPTEE...   So if we need to secure DMA, which of the above (or combination of them) are required to stop unauthorized access?

I am starting to think the SMMU is just a tool that is used by one of the above to aid in this process (just a "virtual mapper").