iMX6Q Automotive crashes with NULL pointer dereference

[halreid]

After fabricating one set of boards, all which boot fine, we've fabricated another set that use the automotive grade iMX6Q and different eMMC memory, which is supposed to be plug-in compatible. The image is programmed over the USB OTG using the mfgtool, but on boot, it often looks like this:

U-Boot 0E90  (Jul 22 2015 - 14:03:30)

CPU: Freescale i.MX6 family TO1.5 at 792 MHz

Temperature:   29 C, calibration data 0x56d4bf7d

mx6q pll1: 792MHz

mx6q pll2: 528MHz

mx6q pll3: 480MHz

mx6q pll8: 50MHz

ipg clock     : 66000000Hz

ipg per clock : 66000000Hz

uart clock    : 80000000Hz

cspi clock    : 60000000Hz

ahb clock     : 132000000Hz

axi clock   : 264000000Hz

emi_slow clock: 132000000Hz

ddr clock     : 528000000Hz

usdhc1 clock  : 198000000Hz

usdhc2 clock  : 198000000Hz

usdhc3 clock  : 198000000Hz

usdhc4 clock  : 198000000Hz

nfc clock     : 24000000Hz

Board: i.MX6Q-SABRESD: unknown-board Board: 0x63015 [POR ]

Boot Device: MMC

I2C:   ready

DRAM:  512 MB

MMC:   FSL_USDHC: 0,FSL_USDHC: 1,FSL_USDHC: 2,FSL_USDHC: 3

In:    serial

Out:   serial

Err:   serial

Net:   mxc_fec_initialize called

got MAC address from IIM: 00:00:00:00:00:00

enet_board_init->reseting switch.

FEC0 [PRIME]

Hit any key to stop autoboot:  0

Set voltage succeed!

Name            Voltage

vddpu:          1200000

Loading file "/boot/uImage" from mmc device 2:2 (xxc2)

3569756 bytes read

## Booting kernel from Legacy Image at 10800000 ...

   Image Name:   Linux-3.0.35

   Image Type:   ARM Linux Kernel Image (uncompressed)

   Data Size:    3569692 Bytes =  3.4 MB

   Load Address: 10008000

   Entry Point:  10008000

   Verifying Checksum ... OK

   Loading Kernel Image ... OK

OK

Starting kernel ...

Can't recover from RESET HBA!

hub 1-0:1.0: unable to enumerate USB device on port 1

Unable to handle kernel NULL pointer dereference at virtual address 00000008

pgd = 94434000

[00000008] *pgd=24750831, *pte=00000000, *ppte=00000000

Internal error: Oops: 17 [#1] PREEMPT SMP

Modules linked in:

CPU: 3    Not tainted  (3.0.35 #410)

PC is at rb_insert_color+0x28/0x168

LR is at vma_adjust+0x370/0x458

pc : [<802287c8>]    lr : [<800d29b4>]    psr: 60000013

sp : 9491dec0  ip : 947f9e70  fp : 00000000

r10: 00000000  r9 : 00000000  r8 : 947136c4

r7 : 947136c0  r6 : 947f9e8c  r5 : 2ad9d000  r4 : 00000000

r3 : 00000000  r2 : 00000000  r1 : 947136c4  r0 : 947f9e8c

Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user

Control: 10c53c7d  Table: 2443404a  DAC: 00000015

Process sudo (pid: 1431, stack limit = 0x9491c2f0)

Stack: (0x9491dec0 to 0x9491e000)

dec0: 943c1630 943c1b58 00000000 947136c0 00000000 800d29b4 9491df0c 947136c0

dee0: 2aaf4000 2ad9d000 943c1650 00000000 2ad9d000 2ad9a000 0002ad9a 00000000

df00: 943c1668 943c1630 943c1650 2ad9d000 0002ad9d 947f9e70 943c1630 2ad9d000

df20: 806dd5f0 947136c0 9491c000 00000000 00000000 800d2c1c 947f9e70 947f9e70

df40: 947136c4 947136c0 00000000 2ad9d000 2ae1e000 943c1630 943c1108 800d2fd4

df60: 947136fc 804c7748 00000000 947136fc 00081000 2ad9d000 947136c0 8003dfc4

df80: 9491c000 00000000 00000000 800d440c 2ad9a248 2ad9d000 2abb4008 00000007

dfa0: 0000005b 8003de40 2ad9d000 2abb4008 2ad9d000 00081000 00081000 2ad9a248

dfc0: 2ad9d000 2abb4008 00000007 0000005b 00000000 00000000 7e818e14 00000000

dfe0: 2ad9d008 7e817ad8 2acde2ac 2ad342ac 60000010 2ad9d000 27ffc811 27ffcc11

[<802287c8>] (rb_insert_color+0x28/0x168) from [<800d29b4>] (vma_adjust+0x370/0x458)

[<800d29b4>] (vma_adjust+0x370/0x458) from [<800d2c1c>] (__split_vma+0x180/0x1a8)

[<800d2c1c>] (__split_vma+0x180/0x1a8) from [<800d2fd4>] (do_munmap+0xac/0x2a4)

[<800d2fd4>] (do_munmap+0xac/0x2a4) from [<800d440c>] (sys_munmap+0x3c/0x50)

[<800d440c>] (sys_munmap+0x3c/0x50) from [<8003de40>] (ret_fast_syscall+0x0/0x30)

Code: e5952000 e3120001 1a000013 e3c24003 (e5943008)

---[ end trace 41cb41707687c66e ]---

ehci_fsl_bus_suspend begins, Host 1

ehci_fsl_bus_suspend ends, Host 1

otg udc wakeup, host sends resume signal

irq 75: nobody cared (try booting with the "irqpoll" option)

[<80044d28>] (unwind_backtrace+0x0/0xf8) from [<800a51c4>] (__report_bad_irq+0x24/0xbc)

[<800a51c4>] (__report_bad_irq+0x24/0xbc) from [<800a5504>] (note_interrupt+0x1e4/0x240)

[<800a5504>] (note_interrupt+0x1e4/0x240) from [<800a3a70>] (handle_irq_event_percpu+0xa8/0x180)

[<800a3a70>] (handle_irq_event_percpu+0xa8/0x180) from [<800a3b84>] (handle_irq_event+0x3c/0x5c)

[<800a3b84>] (handle_irq_event+0x3c/0x5c) from [<800a5ec4>] (handle_fasteoi_irq+0x98/0x158)

[<800a5ec4>] (handle_fasteoi_irq+0x98/0x158) from [<800a338c>] (generic_handle_irq+0x28/0x3c)

[<800a338c>] (generic_handle_irq+0x28/0x3c) from [<8003e6f0>] (handle_IRQ+0x4c/0xac)

[<8003e6f0>] (handle_IRQ+0x4c/0xac) from [<8003d88c>] (__irq_svc+0x4c/0xe8)

[<8003d88c>] (__irq_svc+0x4c/0xe8) from [<8003e99c>] (default_idle+0x24/0x28)

[<8003e99c>] (default_idle+0x24/0x28) from [<8003eb94>] (cpu_idle+0xbc/0xf8)

[<8003eb94>] (cpu_idle+0xbc/0xf8) from [<800088ac>] (start_kernel+0x290/0x2e0)

[<800088ac>] (start_kernel+0x290/0x2e0) from [<10008040>] (0x10008040)

handlers:

[<8005ec24>] usb_wakeup_handler

[<8031fc78>] fsl_otg_isr

[<8031bb40>] ehci_fsl_pre_irq

[<80305300>] usb_hcd_irq

[<80324bdc>] fsl_udc_irq

Disabling IRQ #75

/usr/bin/pcireset.sh: cd: line 71: can't cd to /sys/devices/pci0000:00/0000:00:00.0/

/usr/bin/pcireset.sh: line 74: can't create remove: nonexistent directory

/usr/bin/pcireset.sh: cd: line 75: can't cd to /sys/devices/pci0000:00/pci_bus/0000:00

/usr/bin/pcireset.sh: line 78: can't create rescan: nonexistent directory

eth0: no IPv6 routers present

The errors are not always the same, and i've run the DDR memory test without problems. Has anyone seen this kind of problem show up before?

[halreid]

Took my board to the BGA cage and them swap out the automotive grade for an industrial chip, and it works fine. Maybe we got a batch of bad automotive parts.