Order of DPAA2 dpni instances / eth interfaces

[felixn]

What determines the order in which dpni instances are probed / the order in which eth interfaces are created?

We have five network interfaces, and want to bring them up in a deterministic order i.e. always assign the same ethX to a certain MAC. How can we ensure that?

This is the connections snippet from the DPL:

connections {
    connection@0 {
        endpoint1 = "dpni@0";
        endpoint2 = "dpmac@16";
    };

    connection@1 {
        endpoint1 = "dpni@1";
        endpoint2 = "dpmac@13";
    };

    connection@2 {
        endpoint1 = "dpni@2";
        endpoint2 = "dpmac@14";
    };

    connection@3 {
        endpoint1 = "dpni@3";
        endpoint2 = "dpmac@15";
    };

    connection@4 {
        endpoint1 = "dpni@4";
        endpoint2 = "dpmac@1";
    };
};

The kernel seems to probe dpni instances in reverse order - is this reliable?

[    5.832792] fsl_dpaa2_eth dpni.4: Probed interface eth0
[    6.034414] fsl_dpaa2_eth dpni.3: Probed interface eth1
[    6.223226] fsl_dpaa2_eth dpni.2: Probed interface eth2
[    6.411562] fsl_dpaa2_eth dpni.1: Probed interface eth3
[    6.612304] fsl_dpaa2_eth dpni.0: Probed interface eth4

This leads to the eth devices having inverse order of the dpni instances. Can we avoid that, and have dpni.0 <-> eth0, dpni.1 <-> eth1, ...

# file /sys/class/net/*
/sys/class/net/eth0: symbolic link to ../../devices/platform/soc/80c000000.fsl-mc/dprc.1/dpni.4/net/eth0
/sys/class/net/eth1: symbolic link to ../../devices/platform/soc/80c000000.fsl-mc/dprc.1/dpni.3/net/eth1
/sys/class/net/eth2: symbolic link to ../../devices/platform/soc/80c000000.fsl-mc/dprc.1/dpni.2/net/eth2
/sys/class/net/eth3: symbolic link to ../../devices/platform/soc/80c000000.fsl-mc/dprc.1/dpni.1/net/eth3
/sys/class/net/eth4: symbolic link to ../../devices/platform/soc/80c000000.fsl-mc/dprc.1/dpni.0/net/eth4

[yipingwang · Accepted solution]

Please refer to the following update from the SE team.

Hello, it is not surely the inverse order of the dpni and the interface name, for example:
root@localhost:~# dmesg | grep "Probed interface"
[ 6.370376] fsl_dpaa2_eth dpni.1: Probed interface eth1
[ 6.727306] fsl_dpaa2_eth dpni.0: Probed interface eth2
[ 342.719423] fsl_dpaa2_eth dpni.2: Probed interface eth0
[ 495.610245] fsl_dpaa2_eth dpni.3: Probed interface eth3
[ 4287.641594] fsl_dpaa2_eth dpni.4: Probed interface eth4

For the dpni name and corresponding network interface name, they are different on searching for the corresponding numbers
1. For the dpni.x, the numbers for dpni are from the container_id, and the container id is from the response of MC, the customer may reference the dprc_get_container_id(), seems this parameter is from the firmware of MC, not set by the programmer
2. For the ethX, the name that are set is from the code source:
register_netdevice --> dev_get_valid_name --> dev_alloc_name_ns --> __dev_alloc_name
it is the kernel code which is respond for selecting the name/number of the network interface in certain manner(seems bit map), not controlled by the dpaa2 driver
it cannot be easily bound with the same number of dpni.(number) and eth(number).

[mcbridematt]

The best way I found to deterministically order them is to assign DPNI<->DPMAC connections in reverse order of DPNI's, as you mentioned.

Like so:

connection@1{
    endpoint1 = "dpni@9";
    endpoint2 = "dpmac@7";
};

connection@2{
    endpoint1 = "dpni@8";
    endpoint2 = "dpmac@8";
};

connection@3{
    endpoint1 = "dpni@7";
    endpoint2 = "dpmac@9";
};

So in the above, the port attached to DPMAC7 will always be eth0, DPMAC8 eth1, DPMAC9 eth2 etc.

This won't get them in order of dpni.0/dpni.1/etc., though you can always rename them in userspace the way you want (e.g ip link set eth1 name ethX)