Kernel exception while tftp the file to RAM

[chandrashekhar]

Hi team,

We are seeing kernel exceptions on t1042 based board while trasferring file through tftp in RAM area. These exceptions are not which always comes. It comes once in a while, below is the log message,

root@t1040rdb:/media/ram# busybox tftp -b 64000 -g -r /auto/tftp-blr-users3/chanmish/arad_test/bcm.user 202.153.144.25
/auto/tftp-blr-users  14% |*************************************************************************************************************************Unable to handle kernel paging request for instruction fetch
Faulting instruction address: 0x1b201000
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=8 T104x RDB
Modules linked in: qoriq_dbg(O)
NIP: 1b201000 LR: c0506118 CTR: 1b201000
REGS: ee5edea0 TRAP: 0400   Tainted: G           O  (3.8.13-rt9-QorIQ-SDK-T1040-BSP0.2)
MSR: 00021002 <CE,ME>  CR: 22044028  XER: 00000000
TASK = c0848300[0] 'swapper/0' THREAD: c0884000 CPU: 0
GPR00: c008daec ee5edf50 c0848300 c190e740 e86253d0 f11e9240 00000240 00000000
GPR08: 00000000 1b201000 00000000 00000002 ffffffff 8db520f4 ffffffff 7ffb0a2c
GPR16: c0000a00 00000014 c07686fc 00000001 c0768738 c0880000 ffffff80 40000000
GPR24: 00000000 000fffff 00020000 00000001 ee5ec000 e83cbc00 f11e9240 c190e740
NIP [1b201000] 0x1b201000
LR [c0506118] portal_isr+0x1c8/0x240
Call Trace:
[ee5edf50] [c05060dc] portal_isr+0x18c/0x240 (unreliable)
[ee5edf80] [c008daec] handle_irq_event_percpu+0xbc/0x220
[ee5edfc0] [c008dc94] handle_irq_event+0x44/0x80
[ee5edfe0] [c00916d0] handle_fasteoi_irq+0xd0/0x180
[ee5edff0] [c000d9c8] call_handle_irq+0x18/0x28
[c0885ea0] [c000437c] do_IRQ+0x11c/0x190
[c0885ed0] [c000fbb8] ret_from_except+0x0/0x18
--- Exception: 501 at cpu_idle+0xb8/0x180
    LR = cpu_idle+0xb8/0x180
[c0885f90] [c0007a08] cpu_idle+0x58/0x180 (unreliable)
[c0885fc0] [c08027a8] start_kernel+0x2d0/0x2e4
[c0885ff0] [c00003e8] skpinv+0x2d4/0x310
Instruction dump:
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
---[ end trace 104530bb0a694050 ]---

Kernel panic - not syncing: Fatal exception in interrupt
Rebooting in 180 seconds..

Regards,

-Chandra

[lunminliang]

Can you clarify below items:

1) Is this customers board or one of freescale RDB/QDS boards? If customer own board are you able to replicate this on one of freescale boards?

2) What is the failure rate and is it the same across all boards?

3) What Ethernet interfaces are enabled on the board and does the issue show on all interfaces?

4) Do you use freescale SDK software? If so what release?

5) Do any other applications/tests show issues? E.g. if running iperf on the interface does it run for long periods without issue. And do you do DDR tests that show the DDR is stable?

6) Does changing the address the file is tftped to have any affect on the failure rate?

[chandrashekhar]

Hi,

Below is the answer,

1) Yes, this is the custom board. Same I could not see on replicated on freescale (t1040qds) board.

2) We saw the same in two boards.

3) We are using mac-4 for tftp.Other ports are unused .

4) We are using freescale SDK 1.4.

5) Currently I am seeing this issue while tftp only. Other application where eth port is not used is not showing error. We tested DDR with walkimg 1  algo but it does not show any error. I will recheck the DDR error registers after crash and update you.

6) No. changing address also showing the failure.

Regards,

Chandra

[lunminliang]

And one more question, do you use file system from SDK and which one do you used? If so can you test different options(fsl-image-minimal, -core, -full) to see if the issue occurs in all builds?

[lunminliang]

I would suggest to use SDK1.6 as it's the first with official support for T104x.

If problem remains,

1. can you please share a complete boot log from U-Boot start-up to the test failing.

2. can you confirm if results such as iperf are as expected? to check if it's state without packet loss and errors.

3  has alternative application been tried for transferring the same file used in tftp, say WinSCP?

[yipingwang]

Hello Chandra,

Did you enable PREEMPT_RT in the Kernel configuration file? If so, there is a fix in the latter release.

Would you please also share your Kernel configuration file?

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[chandrashekhar]

Hi Yiping,

Pl find the attached config file which we are using for kernel build.

Regards,

Chandra

[yipingwang]

Hello Chandra,

Just now I checked your Kernel configuration file, it is very different with the default configuration file, especially missing QMAN, BMAN configurations, actually the function "portal_isr" is BMAN Portal interrupt handler.

I suggest you perform the modification based on the default Kernel configuration.

You could get Kernel configuration file(.config)  in the source code folder "build_t1042rdb_release/tmp/work/t1042rdb-fsl-linux/linux-qoriq-sdk/3.8-r11.1/git" with the following command.

>bitbake virtual/kernel -c configure -f

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[chandrashekhar]

HI Yiping,

I am attaching the two config files here  taken from  "build_t1042rdb_release/tmp/work/t1042rdb-fsl-linux/linux-qoriq-sdk/3.8-r11.1/git" folder.

1) config_t1040.config -> this is what we are using.

2) config_t1042.config -> default file.

We modified config_t1040.config to set default  baudrate to 9600 and  enabled marvell phy support as our board has Marvell PHY instead of realtek PHY as in t1040qds board. Do you find anything wrong in these files?

Regards,

Chandra

[yipingwang]

Hello Chandra,

The modified Kernel configuration file looks OK.

What's about the result to run the new uImage on your target board? Does it work normally?

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[chandrashekhar]

Hi Yiping,

As I mentioned earlier we don't see kernel crash always while tftp. It crashes once out of 10 times. We are using the uImage built with  config_t1040.config.

Regards,

Chandra

[yipingwang]

Hello Chandra,

You mentioned previously that this problem couldn't been reproduced on Freescale demo board. Did you use the same version SDK images(3.8.13-rt9-QorIQ-SDK-T1040-BSP0.2) on Freescale demo board?

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[chandrashekhar]

Hi Yiping,

Yes, I used same version of sdk on freescale board  as well.

Regards,

Chandra

[yipingwang]

Hello Chandra,

Please check whether the following could help you to do further investigation for this problem.

1. Under u-boot, use tftp to download the testing file and compute the md5sum values, then repeat and check whether the problem will occur.  If the transcending problem also can be reproduced in u-boot, probably the hardware problem needs to be considered about.

2. Please try to use other network application, for example netperf, I attached netperf related user manual to you. Please check whether Kernel call trace could occur, and at the same place.

If the Kernel call trace happens at the same place every time, I think probably the hardware unstable or DDR configuration problem could be excluded.

3. Please unload the qoriq-dbg module.

  >>modprobe -r qoriq-dbg

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[chandrashekhar]

Hi Yiping,

I will tryout the options mentioned by you and update you later with results.

Regards,

Chandra