P2041 exception on reboot

[adeel]

Hi,

I am using SDK-1.4 and I saw BUG_ON at qman_high.c:1085 when I do reboot. I am using CPU unplug patch and it also didn't solve it. Please, help in understanding what this BUG_ON means in function qman_poll_dqrr(); we are not using any shared portal in the DTS.

Thanks and Regards,

Adeel

[yipingwang]

Hello Muhammad,

Would you please provide the error log?

Did you modify the dts file? If so, please provide the dts file.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[adeel]

Hi Yiping,

Following is the error log:

------------[ cut here ]------------

kernel BUG at drivers/staging/fsl_qbman/qman_high.c:1085!

Oops: Exception in kernel mode, sig: 5 [#1]

PREEMPT SMP NR_CPUS=8 NAMC-P2041

Modules linked in: nfsd exportfs

NIP: c04ad168 LR: c04ad138 CTR: c03ee160

REGS: effd9e70 TRAP: 0700   Not tainted  (3.8.13-rt9-QorIQ-SDK-V1.4)

MSR: 00029002 <CE,EE,ME>  CR: 42044042  XER: 20000000

TASK = c07e6300[0] 'swapper/0' THREAD: c0820000 CPU: 0

GPR00: c04ad138 effd9f20 c07e6300 00000000 00000040 00000001 0000000c 00000000

GPR08: 003f0000 00020000 00000001 76dc4190 22044044 65bbaed9 ffffffff 0000000c

GPR16: 00000100 c07082a4 c07e866c 00200000 c0820000 00000000 c07e13c0 c0820000

GPR24: 00000001 c1ad1a88 00000040 c0820000 00000040 c1ad5ee4 c1ad5ee4 c1acf800

NIP [c04ad168] qman_poll_dqrr+0x48/0x250

LR [c04ad138] qman_poll_dqrr+0x18/0x250

Call Trace:

[effd9f20] [c04ad138] qman_poll_dqrr+0x18/0x250 (unreliable)

[effd9f50] [c03ee180] dpaa_eth_poll+0x20/0x60

[effd9f70] [c04e98c4] net_rx_action+0x134/0x1f0

[effd9fa0] [c0031930] __do_softirq+0x110/0x1c0

[effd9ff0] [c000d2ac] call_do_softirq+0x14/0x24

[c0821e70] [c0004728] do_softirq+0xc8/0x100

[c0821e90] [c003219c] irq_exit+0xac/0xf0

[c0821ea0] [c0004424] do_IRQ+0xf4/0x190

[c0821ed0] [c000f430] ret_from_except+0x0/0x18

--- Exception: 501 at cpu_idle+0x118/0x190

    LR = cpu_idle+0x118/0x190

[c0821f90] [c0007c38] cpu_idle+0xe8/0x190 (unreliable)

[c0821fc0] [c07a17bc] start_kernel+0x2e8/0x2fc

[c0821ff0] [c00003c8] skpinv+0x2b4/0x2f0

Instruction dump:

3d20c083 3fe0c07e 3929c120 3bfff800 5463103a 7d29182e 7fff4a14 813f1020

2f890000 409e020c 813f1004 5529039c <0f090000> 895f081c 811f0814 3b800001

---[ end trace de93eba219afe152 ]---

This is a custom board so we have to modify DTS file. Following are the Portal setup in DTS:

cpus {

        #address-cells = <1>;

        #size-cells = <0>;

        cpu0: PowerPC,e500mc@0 {

            device_type = "cpu";

            reg = <0>;

            next-level-cache = <&L2_0>;

            L2_0: l2-cache {

                next-level-cache = <&cpc>;

            };

        };

        cpu1: PowerPC,e500mc@1 {

            device_type = "cpu";

            reg = <1>;

            next-level-cache = <&L2_1>;

            L2_1: l2-cache {

                next-level-cache = <&cpc>;

            };

        };

        cpu2: PowerPC,e500mc@2 {

            device_type = "cpu";

            reg = <2>;

            next-level-cache = <&L2_2>;

            L2_2: l2-cache {

                next-level-cache = <&cpc>;

            };

        };

        cpu3: PowerPC,e500mc@3 {

            device_type = "cpu";

            reg = <3>;

            next-level-cache = <&L2_3>;

            L2_3: l2-cache {

                next-level-cache = <&cpc>;

            };

        };

    };

    memory {

        device_type = "memory";

    };

    bman-portals@f4000000 {

        #address-cells = <0x1>;

        #size-cells = <0x1>;

        compatible = "simple-bus";

        ranges = <0x0 0xf4000000 0x200000>;

        bman-portal@0 {

            cell-index = <0x0>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0x0 0x4000 0x100000 0x1000>;

            interrupts = <105 2 0 0>;

        };

        bman-portal@4000 {

            cell-index = <0x1>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0x4000 0x4000 0x101000 0x1000>;

            interrupts = <107 2 0 0>;

        };

        bman-portal@8000 {

            cell-index = <2>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0x8000 0x4000 0x102000 0x1000>;

            interrupts = <109 2 0 0>;

        };

        bman-portal@c000 {

            cell-index = <0x3>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0xc000 0x4000 0x103000 0x1000>;

            interrupts = <111 2 0 0>;

        };

        bman-portal@10000 {

            cell-index = <0x4>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0x10000 0x4000 0x104000 0x1000>;

            interrupts = <113 2 0 0>;

        };

        bman-portal@14000 {

            cell-index = <0x5>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0x14000 0x4000 0x105000 0x1000>;

            interrupts = <115 2 0 0>;

        };

        bman-portal@18000 {

            cell-index = <0x6>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0x18000 0x4000 0x106000 0x1000>;

            interrupts = <117 2 0 0>;

        };

        bman-portal@1c000 {

            cell-index = <0x7>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0x1c000 0x4000 0x107000 0x1000>;

            interrupts = <119 2 0 0>;

        };

        bman-portal@20000 {

            cell-index = <0x8>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0x20000 0x4000 0x108000 0x1000>;

            interrupts = <121 2 0 0>;

        };

        bman-portal@24000 {

            cell-index = <0x9>;

            compatible = "fsl,p2041-bman-portal", "fsl,bman-portal";

            reg = <0x24000 0x4000 0x109000 0x1000>;

            interrupts = <123 2 0 0>;

        };

        bman-bpids@0 {

            compatible = "fsl,bpid-range";

            fsl,bpid-range = <0x20 0x20>;

        };

    };

    qman-portals@f4200000 {

        #address-cells = <0x1>;

        #size-cells = <0x1>;

        compatible = "simple-bus";

        ranges = <0x0 0xf4200000 0x200000>;

        qportal0: qman-portal@0 {

            cell-index = <0x0>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0x0 0x4000 0x100000 0x1000>;

            interrupts = <104 0x2 0 0>;

            fsl,qman-channel-id = <0x0>;

        };

        qportal1: qman-portal@4000 {

            cell-index = <0x1>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0x4000 0x4000 0x101000 0x1000>;

            interrupts = <106 0x2 0 0>;

            fsl,qman-channel-id = <0x1>;

        };

        qportal2: qman-portal@8000 {

            cell-index = <0x2>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0x8000 0x4000 0x102000 0x1000>;

            interrupts = <108 0x2 0 0>;

            fsl,qman-channel-id = <0x2>;

        };

        qportal3: qman-portal@c000 {

            cell-index = <0x3>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0xc000 0x4000 0x103000 0x1000>;

            interrupts = <110 0x2 0 0>;

            fsl,qman-channel-id = <0x3>;

        };

        qportal4: qman-portal@10000 {

            cell-index = <0x4>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0x10000 0x4000 0x104000 0x1000>;

            interrupts = <112 0x2 0 0>;

            fsl,qman-channel-id = <0x4>;

        };

        qportal5: qman-portal@14000 {

            cell-index = <0x5>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0x14000 0x4000 0x105000 0x1000>;

            interrupts = <114 0x2 0 0>;

            fsl,qman-channel-id = <0x5>;

        };

        qportal6: qman-portal@18000 {

            cell-index = <0x6>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0x18000 0x4000 0x106000 0x1000>;

            interrupts = <116 0x2 0 0>;

            fsl,qman-channel-id = <0x6>;

        };

        qportal7: qman-portal@1c000 {

            cell-index = <0x7>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0x1c000 0x4000 0x107000 0x1000>;

            interrupts = <118 0x2 0 0>;

            fsl,qman-channel-id = <0x7>;

        };

        qportal8: qman-portal@20000 {

            cell-index = <0x8>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0x20000 0x4000 0x108000 0x1000>;

            interrupts = <120 0x2 0 0>;

            fsl,qman-channel-id = <0x8>;

        };

        qportal9: qman-portal@24000 {

            cell-index = <0x9>;

            compatible = "fsl,p2041-qman-portal", "fsl,qman-portal";

            reg = <0x24000 0x4000 0x109000 0x1000>;

            interrupts = <122 0x2 0 0>;

            fsl,qman-channel-id = <0x9>;

        };

        qman-fqids@0 {

            compatible = "fsl,fqid-range";

            fsl,fqid-range = <0x100 0x100>;

        };

        qman-fqids@1 {

            compatible = "fsl,fqid-range";

            fsl,fqid-range = <32768 32768>;

        };

        qman-pools@0 {

            compatible = "fsl,pool-channel-range";

            fsl,pool-channel-range = <0x21 0xf>;

        };

        qman-cgrids@0 {

            compatible = "fsl,cgrid-range";

            fsl,cgrid-range = <0x0 0x100>;

        };

    };

Please ask if you need more information.

[yipingwang]

Hello Muhammad,

I verified SDK 1.4 kernel images with PREEMPT enabled, and no problem with reboot command.

I found a problem in your dts file, bman-portals and qman-portals addresses look not correct, you need to use physical addresses in the dts file, please refer to the default dts file.

	bportals: bman-portals@ff4000000 {
	ranges = <0x0 0xf 0xf4000000 0x200000>;
	};

	qportals: qman-portals@ff4200000 {
	ranges = <0x0 0xf 0xf4200000 0x200000>;
	};

But in this case, you should not boot up kernel at all, I am a little confused.

Would you please attach your whole dts file, Kernel configuration file, and Kernel boot log?

Did you also modify u-boot source?

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[adeel]

Hi,

I think there is a purpose for this BUG_ON statement and only explanation of this statement could tell us what we are doing wrong. So, would it be possible to get explanation of this BUG_ON statement in qman_high.c:1085?

Best Regards,

Adeel

[yipingwang]

Hello Muhammad Adeel,

After research and discussion, we consider this problem is DPAA driver defect in SDK 1.4, and two patches provided in SDK 1.5 should fix this problem, please refer to the attachment.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[yipingwang]

Hello Muhammad,

Please refer to the following function at BUG_ON, it means DPAA module is still receiving network packages even you enter "reboot" command on the target, QMAN driver doesn't  receive stop signal, it looks like a schedule issue, probably it is caused by PREEMPT configuration.

int qman_poll_dqrr(unsigned int limit)

{

        struct qman_portal *p = get_poll_portal();

        int ret;

#ifdef CONFIG_FSL_DPA_PORTAL_SHARE

        if (unlikely(p->sharing_redirect))

                ret = -EINVAL;

        else

#endif

        {

                BUG_ON(p->irq_sources & QM_PIRQ_DQRI);

                ret = __poll_portal_fast(p, limit);

        }

        put_poll_portal();

        return ret;

}

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[yipingwang]

Please check whether the following Kernel configuration could be helpful for your problem.

# Kernel options

#

# CONFIG_HZ_100 is not set

CONFIG_HZ_250=y

# CONFIG_HZ_300 is not set

# CONFIG_HZ_1000 is not set

CONFIG_HZ=250

CONFIG_SCHED_HRTICK=y

CONFIG_PREEMPT=y

CONFIG_PREEMPT_RT_BASE=y

CONFIG_HAVE_PREEMPT_LAZY=y

CONFIG_PREEMPT_LAZY=y

# CONFIG_PREEMPT_NONE is not set

# CONFIG_PREEMPT_VOLUNTARY is not set

# CONFIG_PREEMPT__LL is not set

# CONFIG_PREEMPT_RTB is not set

CONFIG_PREEMPT_RT_FULL=y

CONFIG_PREEMPT_COUNT=y

CONFIG_BINFMT_ELF=y

# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set

# CONFIG_HAVE_AOUT is not set

CONFIG_BINFMT_MISC=m

CONFIG_COREDUMP=y

CONFIG_HUGETLB_PAGE_SIZE_VARIABLE=y

# CONFIG_MATH_EMULATION is not set

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------