How to debug and trace the Linux kernel,drivers,and android system?

[JustinWu2]

Dear ALL:

Recently,I use our customised board to develop.After running the android system,the board will stop. It shows the following messages.

root@android:/ # Unable to handle kernel paging request at virtual address 6d05dbc0
pgd = c0004000
[6d05dbc0] *pgd=00000000
Internal error: Oops: 80000005 [#1] PREEMPT SMP
Modules linked in:
CPU: 1    Not tainted  (3.0.35-ga81bc7d #1)
PC is at 0x6d05dbc0
LR is at clk_get_rate+0x34/0x40
pc : [<6d05dbc0>]    lr : [<c005dbc0>]    psr: a0000093
sp : d7127eb8  ip : d9802008  fp : c0667de8
r10: c0667d94  r9 : 60000013  r8 : c0039fb8
r7 : c0667d94  r6 : c0669d8c  r5 : c0695690  r4 : 3b5dc153
r3 : 00000053  r2 : 80002053  r1 : c0667de8  r0 : 3b5dc100
Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 10c53c7d  Table: 2212004a  DAC: 00000015

LR: 0xc005db40:
db40  e1a0e00f e594f040 e1a04000 e59f0004 eb1169c9 eaffffeb c066f8c0 e2503000
db60  e92d4010 0a000006 e3730a01 8a000004 e5933044 e3530000 0a000001 e12fff33
db80  e8bd8010 e3a00000 e8bd8010 e3500000 e92d4010 1a000003 ea000009 e5900028
dba0  e3500000 0a000006 e3700a01 8a000004 e590303c e3530000 0afffff7 e12fff33
dbc0  e8bd8010 e3a00000 e8bd8010 e92d4010 e1a04001 ebffffec e3a03000 e8840009
dbe0  e1a00003 e8bd8010 e3500000 0a000002 e3700a01 91d003d0 912fff1e e3a00000
dc00  e12fff1e e52de004 e30d2bcc e24dd00c e34c2005 e3a03000 e30dc86c e34cc05b
dc20  e58dc000 eb02e676 e28dd00c e8bd8000 e92d41f0 e1a06b00 e7e38550 e1a03001

SP: 0xd7127e38:
7e38  00000001 d617d040 000055f0 00000000 ffff9e04 c0fd1040 00000001 c00b8304
7e58  00000001 0000040f 00000005 c0669d8c c0667d94 c0040194 3b5dc100 c0667de8
7e78  80002053 00000053 3b5dc153 c0695690 c0669d8c c0667d94 c0039fb8 60000013
7e98  c0667d94 c0667de8 d9802008 d7127eb8 c005dbc0 6d05dbc0 a0000093 ffffffff
7eb8  3b5dc100 c005dbc0 3b5dc100 c0057a48 c0667de8 c0667d94 d6177860 c0669fb8
7ed8  001312d0 c0695884 001312d0 001312d0 00000000 179a7b00 c04bb5fc c005db48
7ef8  001312d0 3b5dc100 3b5dc100 c0060808 d6326b20 00000004 3b5dc100 c065dac0
7f18  00000004 c065dca0 d7127f4c c0060b10 d7127f5c 127ae569 d7127f58 c0fe0568

IP: 0xd9801f88:
1f88  ******** ******** ******** ******** ******** ******** ******** ********
1fa8  ******** ******** ******** ******** ******** ******** ******** ********
1fc8  ******** ******** ******** ******** ******** ******** ******** ********
1fe8  ******** ******** ******** ******** ******** ******** 00000349 00000000
2008  00000000 00000001 002bc7eb ffffffff ffffffff 00000000 00000000 08b75240
2028  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
2048  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
2068  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

FP: 0xc0667d68:
7d68  42570257 00004840 02400240 00000848 4039363c 00004848 3a39333f 00000850
7d88  38414441 00004850 472d4833 68627061 616d645f 6b6c635f 00000000 00000000
7da8  00000000 00000000 00000000 d4034778 00000000 c0668868 c066b3bc 00000400
7dc8  f40c4068 00000000 00000000 00000000 00000000 c0052938 c0052978 00000000
7de8  316c6c70 7379735f 69616d5f 6c635f6e 0000006b 00000000 00000000 00000000
7e08  d40a6448 00000000 c0667e3c 00000000 00000001 00000000 00000000 c00540d0
7e28  c0056cd8 00000000 c0057298 c00565d8 00000000 5f63736f 006b6c63 00000000
7e48  00000000 00000000 00000000 00000000 00000000 d402f888 00000000 00000000

R1: 0xc0667d68:
7d68  42570257 00004840 02400240 00000848 4039363c 00004848 3a39333f 00000850
7d88  38414441 00004850 472d4833 68627061 616d645f 6b6c635f 00000000 00000000
7da8  00000000 00000000 00000000 d4034778 00000000 c0668868 c066b3bc 00000400
7dc8  f40c4068 00000000 00000000 00000000 00000000 c0052938 c0052978 00000000
7de8  316c6c70 7379735f 69616d5f 6c635f6e 0000006b 00000000 00000000 00000000
7e08  d40a6448 00000000 c0667e3c 00000000 00000001 00000000 00000000 c00540d0
7e28  c0056cd8 00000000 c0057298 c00565d8 00000000 5f63736f 006b6c63 00000000
7e48  00000000 00000000 00000000 00000000 00000000 d402f888 00000000 00000000

R5: 0xc0695610:
5610  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000003
5630  c0599ecc d6177860 c0669fb8 00000003 c0667068 c0599ed8 d61777e0 c0599eb4
5650  d61777a0 00000000 00000000 d99f0000 d9a00000 d9a08000 d9a10100 0000000f
5670  d67d2dc0 d98b4000 00000008 d98b5000 c0667cfc d9964000 d9964000 1f78a400
5690  00000000 00000000 00008000 016e3600 00000000 00000000 00000000 00000000
56b0  d9802000 00000001 d9804000 00000003 c0667068 00000001 00000000 00000000
56d0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
56f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

R6: 0xc0669d0c:
9d0c  c066aae0 c0669d24 c0669cfc 00000000 c059a538 c066ab34 c0669d38 c0669d10
9d2c  00000000 c059a544 c066ab88 c0669d4c c0669d24 00000000 c059a548 c066abdc
9d4c  c0669d60 c0669d38 00000000 c059a550 c066ac30 c0669d74 c0669d4c c059a564
9d6c  00000000 c066ac84 c0669d88 c0669d60 c059a570 00000000 c066acd8 c0669d9c
9d8c  c0669d74 c059a57c 00000000 c066ad2c c0669db0 c0669d88 c059a588 00000000
9dac  c066ad80 c0669dc4 c0669d9c 00000000 c059a594 c06688bc c0669dd8 c0669db0
9dcc  c059a5a0 00000000 c066add4 c0669dec c0669dc4 00000000 c059a5a8 c066a00c
9dec  c0669e00 c0669dd8 00000000 c0599c14 c066ae7c c0669e14 c0669dec 00000000

R7: 0xc0667d14:
7d14  000004c0 00000000 000004c4 00000000 000004c8 00000000 000004cc 00000000
7d34  000004d0 00000000 000004d4 00000000 000004d8 00000000 0000008b 00000090
7d54  0000083c 42490249 00000840 02470247 0000483c 42570257 00004840 02400240
7d74  00000848 4039363c 00004848 3a39333f 00000850 38414441 00004850 472d4833
7d94  68627061 616d645f 6b6c635f 00000000 00000000 00000000 00000000 00000000
7db4  d4034778 00000000 c0668868 c066b3bc 00000400 f40c4068 00000000 00000000
7dd4  00000000 00000000 c0052938 c0052978 00000000 316c6c70 7379735f 69616d5f
7df4  6c635f6e 0000006b 00000000 00000000 00000000 d40a6448 00000000 c0667e3c

R8: 0xc0039f38:
9f38  2e696e6a 00006f73 0000029a 01100018 6462696c 636b7369 69666e6f 6f732e67
9f58  0000029b 01080010 6462696c 6f732e6c 0000029c 010a0014 6462696c 2e316d72
9f78  00006f73 0000029d 010e0018 6462696c 5f316d72 2e696e6a 00006f73 0000029e
9f98  0112001c 6462696c 72666d72 77656d61 2e6b726f 00006f73 0000029f 01160020
9fb8  6462696c 72666d72 77656d61 5f6b726f 2e696e6a 00006f73 000002a0 01090014
9fd8  6462696c 732e6d76 0000006f 000002a1 010c001c 6562696c 636e6566 6f732e65
9ff8  00000000 00000000 ea001be3 e92d41f0 e5903040 e590503c e24dd088 e203101f
a018  e2133020 13a02002 03a02004 e3510013 e0625005 e1a04000 e580503c 05956000

R10: 0xc0667d14:
7d14  000004c0 00000000 000004c4 00000000 000004c8 00000000 000004cc 00000000
7d34  000004d0 00000000 000004d4 00000000 000004d8 00000000 0000008b 00000090
7d54  0000083c 42490249 00000840 02470247 0000483c 42570257 00004840 02400240
7d74  00000848 4039363c 00004848 3a39333f 00000850 38414441 00004850 472d4833
7d94  68627061 616d645f 6b6c635f 00000000 00000000 00000000 00000000 00000000
7db4  d4034778 00000000 c0668868 c066b3bc 00000400 f40c4068 00000000 00000000
7dd4  00000000 00000000 c0052938 c0052978 00000000 316c6c70 7379735f 69616d5f
7df4  6c635f6e 0000006b 00000000 00000000 00000000 d40a6448 00000000 c0667e3c
Process kinteractiveup (pid: 1088, stack limit = 0xd71262f0)
Stack: (0xd7127eb8 to 0xd7128000)
7ea0:                                                       3b5dc100 c005dbc0
7ec0: 3b5dc100 c0057a48 c0667de8 c0667d94 d6177860 c0669fb8 001312d0 c0695884
7ee0: 001312d0 001312d0 00000000 179a7b00 c04bb5fc c005db48 001312d0 3b5dc100
7f00: 3b5dc100 c0060808 d6326b20 00000004 3b5dc100 c065dac0 00000004 c065dca0
7f20: d7127f4c c0060b10 d7127f5c 127ae569 d7127f58 c0fe0568 00000003 c065db00
7f40: 000f32a0 c00767f4 c0fe0568 00000003 00060ae0 000f32a0 00000001 00000000
7f60: c0fe0de8 c065dac4 c065dac4 d7126000 c06b0f98 c065db00 000f32a0 c065dca0
7f80: c0fc8da8 c0310234 00000000 c031511c c0038da8 c0038da8 d7127fbc d602ff18
7fa0: d7127fcc 00000000 c0314fdc 00000000 00000000 00000000 00000000 c008c328
7fc0: d602ff18 00000000 00000000 00000000 00000000 00000000 d7127fd8 d7127fd8
7fe0: 00000000 d602ff18 c008c2a4 c00410b4 00000013 c00410b4 ffffffff ffffffff
Code: bad PC value
---[ end trace fd736a84a3b147be ]---
Kernel panic - not syncing: Fatal exception
[<c0046614>] (unwind_backtrace+0x0/0xfc) from [<c04b6c90>] (panic+0x74/0x198)
[<c04b6c90>] (panic+0x74/0x198) from [<c004430c>] (die+0x234/0x2a4)
[<c004430c>] (die+0x234/0x2a4) from [<c0049e78>] (__do_kernel_fault+0x64/0x84)
[<c0049e78>] (__do_kernel_fault+0x64/0x84) from [<c004a018>] (do_page_fault+0x180/0x2e0)
[<c004a018>] (do_page_fault+0x180/0x2e0) from [<c003a3cc>] (do_PrefetchAbort+0x34/0x98)
[<c003a3cc>] (do_PrefetchAbort+0x34/0x98) from [<c0040194>] (__pabt_svc+0x74/0xc0)
Exception stack(0xd7127e70 to 0xd7127eb8)
7e60:                                     3b5dc100 c0667de8 80002053 00000053
7e80: 3b5dc153 c0695690 c0669d8c c0667d94 c0039fb8 60000013 c0667d94 c0667de8
7ea0: d9802008 d7127eb8 c005dbc0 6d05dbc0 a0000093 ffffffff
[<c0040194>] (__pabt_svc+0x74/0xc0) from [<6d05dbc0>] (0x6d05dbc0)
CPU2: stopping
[<c0046614>] (unwind_backtrace+0x0/0xfc) from [<c003a2d4>] (do_IPI+0x188/0x1bc)
[<c003a2d4>] (do_IPI+0x188/0x1bc) from [<c004000c>] (__irq_svc+0x4c/0xe8)
Exception stack(0xd606df90 to 0xd606dfd8)
df80:                                     00000000 c065dac4 d606dfd8 00000406
dfa0: d606c000 c06950a4 c04bb5ec c0665e54 1000406a 412fc09a 00000000 00000000
dfc0: 002df3b9 d606dfd8 c004e0b4 c004115c 20000013 ffffffff
[<c004000c>] (__irq_svc+0x4c/0xe8) from [<c004115c>] (default_idle+0x24/0x28)
[<c004115c>] (default_idle+0x24/0x28) from [<c0041534>] (cpu_idle+0xbc/0xfc)
[<c0041534>] (cpu_idle+0xbc/0xfc) from [<104b3914>] (0x104b3914)
CPU0: stopping
[<c0046614>] (unwind_backtrace+0x0/0xfc) from [<c003a2d4>] (do_IPI+0x188/0x1bc)
[<c003a2d4>] (do_IPI+0x188/0x1bc) from [<c004000c>] (__irq_svc+0x4c/0xe8)
Exception stack(0xd3625e18 to 0xd3625e60)
5e00:                                                       d3625e60 00000036
5e20: 0000000a 00000000 00000000 00000040 d3624000 c064e040 00000000 c064e040
5e40: c069aaa0 c0038214 c069aaa0 d3625e60 c00779cc c0077668 20000113 ffffffff
[<c004000c>] (__irq_svc+0x4c/0xe8) from [<c0077668>] (__do_softirq+0x58/0x170)
[<c0077668>] (__do_softirq+0x58/0x170) from [<c00779cc>] (irq_exit+0x90/0x98)
[<c00779cc>] (irq_exit+0x90/0x98) from [<c0040fe4>] (handle_IRQ+0x50/0xac)
[<c0040fe4>] (handle_IRQ+0x50/0xac) from [<c004000c>] (__irq_svc+0x4c/0xe8)
[<c004000c>] (__irq_svc+0x4c/0xe8) from [<c04b7a48>] (preempt_schedule_irq+0x40/0x70)
[<c04b7a48>] (preempt_schedule_irq+0x40/0x70) from [<c00400b0>] (svc_preempt+0x8/0x18)
[<c00400b0>] (svc_preempt+0x8/0x18) from [<c008580c>] (create_worker+0x0/0x184)
[<c008580c>] (create_worker+0x0/0x184) from [<c0039540>] (gcwq_nr_running+0x0/0x20)
CPU3: stopping
[<c0046614>] (unwind_backtrace+0x0/0xfc) from [<c003a2d4>] (do_IPI+0x188/0x1bc)
[<c003a2d4>] (do_IPI+0x188/0x1bc) from [<c004000c>] (__irq_svc+0x4c/0xe8)
Exception stack(0xd6077f90 to 0xd6077fd8)
7f80:                                     00000000 c065dac4 d6077fd8 00000406
7fa0: d6076000 c06950a4 c04bb5ec c0665e54 1000406a 412fc09a 00000000 00000000
7fc0: 05e65dc7 d6077fd8 c004e0b4 c004115c 20000013 ffffffff
[<c004000c>] (__irq_svc+0x4c/0xe8) from [<c004115c>] (default_idle+0x24/0x28)
[<c004115c>] (default_idle+0x24/0x28) from [<c0041534>] (cpu_idle+0xbc/0xfc)
[<c0041534>] (cpu_idle+0xbc/0xfc) from [<104b3914>] (0x104b3914)

Do you have any idea about this? Can we use some tools besides ARM ICE to trace and debug the system?

Are there any documents related to debug and trace the system?

Best Regards,

Justin Wu

Original Attachment has been moved to: log_zuni_data_0708_m_Logcat2_log.txt.zip

[sinanakman · Accepted solution]

Hi Justin

In general you can debug your kernel by enabling debug output or adding printk's. Alternatively you can use a JTAG debugger to debug a kernel. It seems you have a custom kernel with a custom board. So far I can see that the problem comes up in the SMP code. It seems Paul will be able to help you further once you provide him your device type (what SoC is this board based on). Meanwhile, I suggest that you review all the config options in your config file and see if they make sense. You can try to turn off the SMP and see if the problem goes away and then you can debug the SMP code. Hope this helps ...