Unable to handle kernel NULL pointer dereference at virtual address 00000000?

[gbiradar]

Hi Guys,

              Well i'm working on imx6solo custom board i got this kernel ooops message at boot time. we have similar board we tested on that the board was working fine but coming to this board we are getting this message. i have done ddr stress test it has pass the test. i have restarted the board many times but same error.

boot1:

==========================================================================

mmc1: new SDIO card at address 0001

Unable to handle kernel NULL pointer dereference at virtual address 00000000

pgd = 80004000

[00000000] *pgd=00000000

Internal error: Oops: 805 [#1] PREEMPT SMP

Modules linked in:

CPU: 0    Not tainted  (3.0.35-2666-gbdde708-svn200 #43)

PC is at _fw_imx_epdc_E60_V220_fw_bin+0x123054/0x12eec7

LR is at __find_get_block+0x104/0x244

pc : [<805ffd44>]    lr : [<8010d974>]    psr: a0000013

sp : 9ffcbc98  ip : 00000000  fp : 8bc95054

r10: 98393bc0  r9 : 9ffcbd14  r8 : 00000002

r7 : 000002dd  r6 : 00000002  r5 : 9fcb0800  r4 : 00000009

r3 : 00000004  r2 : 40000804  r1 : 8b979ec0  r0 : 9c056240

Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel

Control: 10c53c7d  Table: 1000404a  DAC: 00000015

Process swapper (pid: 1, stack limit = 0x9ffca2f0)

Stack: (0x9ffcbc98 to 0x9ffcc000)

bc80:                                                       00001000 00000001

bca0: 9ffcbcbc 803f7dc0 9c0189c0 9ffcbcec 00000000 00000008 000000d9 00000000

bcc0: ffffffff 0000000a 00000002 00000002 9c0189c0 80084860 8010bee0 9c018a00

bce0: 9c018c00 803f83dc f0000009 9fcb0800 00000000 8bc71594 00000013 00000001

bd00: 9fcb0400 9fd4bf00 8094c628 8018dd68 9fcb0400 000002dd 000002e3 00000007

bd20: 00000000 00000000 9ffcbd60 9fcb0800 00000000 801910f4 8bc71400 9fd0aa00

bd40: 9fcb0400 80907cc0 00002840 800e0f88 8094c628 80169cdc 00001000 9fcb0800

bd60: 9fcb0400 8bc71594 8bc71400 9fd0aa00 9fcb0400 9fd4bf00 8094c628 80169d00

bd80: 8094d520 00000000 00000001 00000014 9ffe0df0 0ffffffe 0000000c 00000000

bda0: 00000000 00000000 9c018100 9fd0ab84 00000001 00000000 9fcb0500 00000000

bdc0: 00000ee3 00000000 9fcb04e0 00000000 00000001 00000000 00000009 00000000

bde0: 00000000 00000000 9fcb04c0 0000000a 9fcb065c 00000000 9fcb04a0 00000000

be00: 00000000 00000000 00000000 00000000 00000000 9fcb0400 00000001 9fd5bc00

be20: 9fd5bc0c 9fd0aa00 9fd0b660 00000000 00000000 801daddc 9fd0aa00 00000000

be40: 00004003 00000000 00000009 9c002200 00000083 9c002270 00008000 9fd0aa00

be60: 9fd0b660 00000000 00000000 800e8470 62636d6d 70306b6c 00000031 9ffed7b8

be80: 000066dc 803fbbf8 9fd0b660 0000000a 00000006 9fededc0 8091bf04 8091bf04

bea0: 00008000 00000000 9fd0b680 80164688 80168164 80907d2c 00000000 800e8bd8

bec0: 8091bf04 8091bf04 9fededc0 8091bf04 9fd0b680 00000000 9fd0b660 800ffe04

bee0: 8091bf04 9fd0b660 00000000 9fd0b680 00008000 80100244 00000020 00000000

bf00: 00008000 00000000 00008000 8010194c 9ffec0a0 9c00dbb0 0000000a 800c2378

bf20: 00000000 8085dd7c 00001000 9fd6a000 8085dd7c 00000000 00008000 8085dd7c

bf40: 00000000 00000000 00000000 80101fbc 00000000 80008b0c 00000000 9fd0b660

bf60: 9fd0b680 00000000 00000013 9fd69000 9fd69000 00008000 8002a1d0 80008bdc

bf80: 00000000 80910000 80919ea0 80919e80 00000013 00000000 00000080 8008adbc

bfa0: 80957d84 9fd69000 80039aa4 8085dd7c 0000b301 8093eac0 8002a1e5 80039aa4

bfc0: 00000013 00000000 00000000 80008f00 800299b4 800299b4 80039aa4 800299b4

bfe0: 800299b4 800089b0 00000000 00000000 800088a8 80039aa4 cdafffdf ceff9bbe

Code: 78000000 000012da 80000000 000012e4 (17000000)

---[ end trace e1369762383e64b9 ]---

boot2:

===============================================================================

mmc1: new SDIO card at address 0001

Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP

Modules linked in:

CPU: 0    Not tainted  (3.0.35-2666-gbdde708-svn200 #43)

PC is at _fw_imx_epdc_E60_V220_fw_bin+0xda358/0x12eec7

LR is at do_one_pass+0x668/0x728

pc : [<805b7048>]    lr : [<8018dc00>]    psr: 60000013

sp : 9ffcbc98  ip : 8bc013e8  fp : 8bca10a4

r10: 98393bc0  r9 : 9ffcbd14  r8 : 00000002

r7 : 000002de  r6 : 00000002  r5 : 9fd48800  r4 : 9c056500

r3 : 8bc013e8  r2 : 00000002  r1 : 00000000  r0 : 00000000

Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel

Control: 10c53c7d  Table: 1000404a  DAC: 00000015

Process swapper (pid: 1, stack limit = 0x9ffca2f0)

Stack: (0x9ffcbc98 to 0x9ffcc000)

bc80:                                                       00001000 00000001

bca0: 9ffcbcbc 803f7dc0 9c0189c0 9ffcbcec 00000000 00000008 000000a1 00000000

bcc0: ffffffff 00000020 00000002 00000002 9c0189c0 80084860 8010bee0 9c018d00

bce0: 9c054180 803f83dc f0000009 9fd48800 00000000 8bc71594 00000013 00000001

bd00: 9fd48400 9fe7cf00 8094c628 8018dd68 9fd48400 000002dd 000002e3 0000001c

bd20: 00000000 00000000 9ffcbd60 9fd48800 00000000 801910f4 8bc71400 9fd0aa00

bd40: 9fd48400 80907cc0 00002840 800e0f88 8094c628 80169cdc 00001000 9fd48800

bd60: 9fd48400 8bc71594 8bc71400 9fd0aa00 9fd48400 9fe7cf00 8094c628 80169d00

bd80: 8094d520 00000000 00000001 00000014 9ffe0df0 0ffffffe 0000000c 00000000

bda0: 00000000 00000000 9c018100 9fd0ab84 00000001 00000000 9fd48500 00000000

bdc0: 00000ee3 00000000 9fd484e0 00000000 00000001 00000000 00000009 00000000

bde0: 00000000 00000000 9fd484c0 0000000a 9fd4865c 00000000 9fd484a0 00000000

be00: 00000000 00000000 00000000 00000000 00000000 9fd48400 00000001 9fd58c00

be20: 9fd58c0c 9fd0aa00 9fd0b660 00000000 00000000 801daddc 9fd0aa00 00000000

be40: 00004003 00000000 00000009 9c002200 00000083 9c002270 00008000 9fd0aa00

be60: 9fd0b660 00000000 00000000 800e8470 62636d6d 70306b6c 00000031 9ffed7b8

be80: 000066dc 803fbbf8 9fd0b660 0000000a 00000006 9fededc0 8091bf04 8091bf04

bea0: 00008000 00000000 9fd0b680 80164688 80168164 80907d2c 00000000 800e8bd8

bec0: 8091bf04 8091bf04 9fededc0 8091bf04 9fd0b680 00000000 9fd0b660 800ffe04

bee0: 8091bf04 9fd0b660 00000000 9fd0b680 00008000 80100244 00000020 00000000

bf00: 00008000 00000000 00008000 8010194c 9ffec0a0 9c00dbb0 0000000a 800c2378

bf20: 00000000 8085dd7c 00001000 9fd6a000 8085dd7c 00000000 00008000 8085dd7c

bf40: 00000000 00000000 00000000 80101fbc 00000000 80008b0c 00000000 9fd0b660

bf60: 9fd0b680 00000000 00000013 9fd69000 9fd69000 00008000 8002a1d0 80008bdc

bf80: 00000000 80910000 80919ea0 80919e80 00000013 00000000 00000080 8008adbc

bfa0: 80957d84 9fd69000 80039aa4 8085dd7c 0000b301 8093eac0 8002a1e5 80039aa4

bfc0: 00000013 00000000 00000000 80008f00 800299b4 800299b4 80039aa4 800299b4

bfe0: 800299b4 800089b0 00000000 00000000 800088a8 80039aa4 cdafffdf ceff9bbe

Code: 00001093 80000000 000010c6 88000000 (000010f9)

---[ end trace 0a93ebe5cbb05840 ]---

boot3:

=================================================================================

Synthesizing initial hotplug events

udevd (1106): /proc/1106/oom_adj is deprecated, please use /proc/1106/oom_score_adj instead.

Unable to handle kernel paging request at virtual address 0c061002

pgd = 9fd34000

[0c061002] *pgd=00000000

Internal error: Oops: 5 [#1] PREEMPT SMP

Modules linked in:

CPU: 0    Not tainted  (3.0.35-2666-gbdde708-svn200 #43)

PC is at kmem_cache_alloc+0x2c/0x110

LR is at alloc_pid+0x18/0x3b8

pc : [<800e0eb0>]    lr : [<80081418>]    psr: 00000013

sp : 9feafeb0  ip : 9feafeb0  fp : 00000000

r10: 80907cc0  r9 : 00000000  r8 : 8093fc40

r7 : 80919b4c  r6 : 9feae000  r5 : 000000d0  r4 : 0c061002

r3 : 9feae000  r2 : 80907cc0  r1 : 000000d0  r0 : 00000000

Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user

Control: 10c53c7d  Table: 2fd3404a  DAC: 00000015

Process udevd (pid: 1106, stack limit = 0x9feae2f0)

Stack: (0x9feafeb0 to 0x9feb0000)

fea0:                                     00000003 80081418 8093eb0c 9fc46f20

fec0: 01200011 9feae000 80919b4c 8093fc40 00000000 00000000 00000000 80081418

fee0: 9fd21fb0 9fc46f20 9feaffb0 800897c0 00000000 80d19b28 9fd20000 9fc46f20

ff00: 01200011 9feae000 80919b4c 8093fc40 00000000 00000000 00000000 80069cb0

ff20: 9feaffb0 800cce18 00000000 7ee33860 9feaffb0 9fc47044 2ab58000 00000000

ff40: 9fe04684 01200011 00000000 9feaffb0 00000000 7ee33860 9feae000 00000000

ff60: 7ee3388c 8006a1fc 2ab39068 00000000 00000000 2ab57000 9fe04680 80039144

ff80: 9feae000 00000000 00000037 800cf220 2ab39068 7ee33860 2ac8e000 00000078

ffa0: 80039144 80038fc0 00000000 2ab39068 01200011 00000000 00000000 00000000

ffc0: 2ab39068 7ee33860 2ac8e000 00000078 00000000 00000000 00000000 7ee3388c

ffe0: 2ab394c0 7ee33860 00000000 2abfeb48 60000010 01200011 0b33efbf a397d34b

[<800e0eb0>] (kmem_cache_alloc+0x2c/0x110) from [<80081418>] (alloc_pid+0x18/0x3b8)

[<80081418>] (alloc_pid+0x18/0x3b8) from [<80069cb0>] (copy_process+0x84c/0xcd8)

[<80069cb0>] (copy_process+0x84c/0xcd8) from [<8006a1fc>] (do_fork+0x9c/0x214)

[<8006a1fc>] (do_fork+0x9c/0x214) from [<80038fc0>] (ret_fast_syscall+0x0/0x30)

Code: e1a05001 e1a0a002 e58de004 e5930014 (e5941000)

---[ end trace 2aa3b82f113f85b2 ]---

ehci_fsl_bus_suspend begins, Host 1

ehci_fsl_bus_suspend ends, Host 1

Unable to handle kernel paging request at virtual address 0c061002

pgd = 80004000

[0c061002] *pgd=00000000

Internal error: Oops: 5 [#2] PREEMPT SMP

Modules linked in:

CPU: 0    Tainted: G      D      (3.0.35-2666-gbdde708-svn200 #43)

PC is at kmem_cache_alloc+0x2c/0x110

LR is at alloc_pid+0x18/0x3b8

pc : [<800e0eb0>]    lr : [<80081418>]    psr: 00000113

sp : 9ffd3e80  ip : 9ffd3e80  fp : 00000000

r10: 80907cc0  r9 : 00000000  r8 : 8093fc40

r7 : 80919b4c  r6 : 9ffd2000  r5 : 000000d0  r4 : 0c061002

r3 : 9ffd2000  r2 : 80907cc0  r1 : 000000d0  r0 : 00000000

Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel

Control: 10c53c7d  Table: 2fd3004a  DAC: 00000015

Process kthreadd (pid: 2, stack limit = 0x9ffd22f0)

Stack: (0x9ffd3e80 to 0x9ffd4000)

3e80: 00000003 80081418 8093eb0c 9fc44000 00800711 9ffd2000 80919b4c 8093fc40

3ea0: 00000000 00000000 00000000 80081418 9fd83fb0 9fc44000 9ffd3f80 800897c0

3ec0: 00000000 80d19b28 9fd82000 9fc44000 00800711 9ffd2000 80919b4c 8093fc40

3ee0: 00000000 00000000 00000000 80069cb0 9ffd3f80 8008979c 00000000 00000000

3f00: 9ffd3f80 9fc44124 9ffcc3a0 00000000 9ffce7e0 00800711 00000000 9ffd3f80

3f20: 00000000 00000000 00000000 9fc83f10 00000000 8006a1fc 00000000 00000000

3f40: 00000000 f8abefef edeaedfb fe3fbe67 800320c0 800320c0 800311b4 800320c0

3f60: 80084188 9fc83f10 00000611 00000000 00000001 80039b7c 00000000 00000000

3f80: 00000000 00000000 00000000 00000000 9fc83f10 80084188 80039aa4 00000013

3fa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 80039a94

3fc0: 00000093 00000000 9fc83f34 80919c54 9ffd2000 800844ac 00000000 00000000

3fe0: 800843d8 80039aa4 00000013 00000000 00000000 80039aa4 cdafff5f ceff9bbe

[<800e0eb0>] (kmem_cache_alloc+0x2c/0x110) from [<80081418>] (alloc_pid+0x18/0x3b8)

[<80081418>] (alloc_pid+0x18/0x3b8) from [<80069cb0>] (copy_process+0x84c/0xcd8)

[<80069cb0>] (copy_process+0x84c/0xcd8) from [<8006a1fc>] (do_fork+0x9c/0x214)

[<8006a1fc>] (do_fork+0x9c/0x214) from [<80039b7c>] (kernel_thread+0x74/0x7c)

[<80039b7c>] (kernel_thread+0x74/0x7c) from [<800844ac>] (kthreadd+0xd4/0x138)

[<800844ac>] (kthreadd+0xd4/0x138) from [<80039aa4>] (kernel_thread_exit+0x0/0x8)

Code: e1a05001 e1a0a002 e58de004 e5930014 (e5941000)

---[ end trace 2aa3b82f113f85b3 ]---

AX

----------------------------------------------------------------------------------------------------------------------------

can anyone tell what is the issue.

regards,

Ganesh

[jamesbone]

Hello Ganesh,

My only advise it is to do SWAP change of the device to see if the problem still persist.

[jamesbone]

Hello Ganesh,

Based on the error of the log that you are sharing it seems that you are having memory issues,  maybe from your layout or in the manufacturing, have you try to run any memory stress tool to see if the memory it is working correctly?

i.MX6/7 DDR Stress Test Tool V2.40

i.Mx6DQSDL DDR3 Script Aid

Have a great day,
Jaime

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[gbiradar]

Hi jamesbone​,

i have already done the stress test. their is no issue with that.

[jamesbone]

Hello Ganesh,

Are you booting from the SDCard?  or are you using any other, have you try to debug a little further, to see which driver it is making the Kernel Ooops?

Are you using a Wifi on the SD Card or any other device in USB port.

Another option it is to slow down the Frequency of the DDR to see if fix the problem

Have a great day,
Jaime

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[gbiradar]

HI jamesbone​ yes we are using sd card as well as wifi also. yes we disable many application from boot time so that we can see the problem. coming to slow down of frequency same configuration is used on other board(production) also it is working fine but only one board is having issue.

[jamesbone]

Hello Ganesh,

If several boards are working correctly and only one it is having the crash, then it has to be a problem related to the manufacturing process, check the voltaje levels of the board, the frequency on the SD Cards.  Need to be a hardware issue.

Have a great day,
Jaime

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[gbiradar]

Hi jamesbone​

update: we already did that

1-> voltage levels are correct

2-> frequency of sd card is also correct

one thing, does freescale provide any application/utility to test imx6 BGA chip.

regards,

Ganesh Biradar

[jamesbone]

Hello Ganesh,

Nop we do not have any Unit Test that can help to test the BGA.  So the I.MX it is generating correctly the signals of the SD Card?  you are seeing activity on the pins?, what about changing the part from one of the boards that are working? are there any differences between the two devices in the marking?

[gbiradar]

Hi jamesbone​ we did all testing manually and results are compared with other working board it is same no issue. it's mass production so component are same for all board no change at all.

[gbiradar]

update ->

1 -> we are using sd boot port 3.

2 -> wifi is also sdio interface on port 1

3 -> we have isolated driver with multiple possibilities

But still crash is happening.