i.MX6D: IPU task cause kernel dump - 1080p Stream - after many hours

[Tarek]

Hello,

I'm streaming a single 1080p stream 30 fps to a small form factor computer running i.MX6D "Utilite Standard". After running it for several hours there is  about 3 seconds latency which didn't exist at the beginning of playing the stream. When I checked the dmesg I found the following kernel message. I've applied this patch but still didn't help. I'm running Yocto distro , Kernel 4.1.0

Is there any suggestion from freescale?

[ALLOC] mem alloc size = 0x4fe807

[ALLOC] mem alloc cpu_addr = 0xff500000

start=0x51e00000, pgoff=0x17000, size=0x4ff000

[ALLOC] mem alloc size = 0x14000

[ALLOC] mem alloc cpu_addr = 0xffd60000

[ALLOC] mem alloc size = 0x2f1c00

[ALLOC] mem alloc cpu_addr = 0xfd300000

start=0x4fc00000, pgoff=0x18400, size=0x2f2000

vpu: memory phy:0x16c00000 is phy memory

vpu: memory phy:0x16df6800 is phy memory

vpu: memory phy:0x18c00000 is phy memory

vpu: memory phy:0x18df6800 is phy memory

vpu: memory phy:0x17c00000 is phy memory

vpu: memory phy:0x17df6800 is phy memory

imx-ipuv3 imx-ipuv3.0: IDMAC12's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.1: IDMAC12's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.0: IDMAC12's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.1: IDMAC12's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.0: IDMAC12's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.1: IDMAC12's EBA0 is not 8-byte aligned

Unable to handle kernel NULL pointer dereference at virtual address 00000000

pgd = c0004000

[00000000] *pgd=00000000

Internal error: Oops: 17 [#1] SMP

Modules linked in: nfsd exportfs igb pwm_bl btmrvl_sdio backlight btmrvl

CPU: 0    Not tainted  (3.0.35-2666-gbdde708+yocto+g7097df7 #1)

PC is at __wake_up_common+0x2c/0x94

LR is at __wake_up+0x3c/0x50

pc : [<c006e6cc>]    lr : [<c006eebc>]    psr: a00f0093

sp : e9de3c30  ip : fffffff4  fp : e9de3c5c

r10: 00000004  r9 : 00000003  r8 : 00000000

r7 : 00000000  r6 : e421b2a4  r5 : 600f0013  r4 : e421b2a0

r3 : 00000000  r2 : 00000001  r1 : 00000000  r0 : e421b2a0

Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel

Control: 10c53c7d  Table: 347e804a  DAC: 00000015

Process ipu2_task (pid: 394, stack limit = 0xe9de22f0)

Stack: (0xe9de3c30 to 0xe9de4000)

3c20:                                     e421bc00 00000001 e421b28c e421b2a0

3c40: 600f0013 00000001 00000003 00000000 e9de3d44 00000004 e9de3c84 c006eebc

3c60: 00000000 ffffffff c07977cc e421bc00 00000001 e421b28c 00000094 00200200

3c80: 00000000 c03ca6a8 00000001 00000000 c0713d20 c00422d8 ffffffff f2a00100

3ca0: 00000403 00000094 00200200 c00477b8 c0745d18 600f0013 e9de3ce8 00000000

3cc0: 600f0013 e9de2000 c0745d0c 00000094 00200200 600f0013 00000000 e421bc00

3ce0: 00000094 00200200 e9de3d44 00000004 00000000 c03cb2bc c0713d20 e9fd1c00

3d00: c0040ee0 c0713aa0 cce11ee0 e9d3d000 00000003 e9de2000 c0745d0c c07977cc

3d20: c009ab58 00000001 e9de2038 e9d3d038 e9de3d54 c053e8b8 e9d3d000 00000031

3d40: e9de2031 00000000 e9d3f800 c009ab58 e9de3d50 e9de3d50 00000780 00000430

3d60: 3231564e 000003b0 00000210 000003d0 00000220 18c00000 00000000 00000000

3d80: 00000780 00000438 50424752 00000000 000003b0 00000210 000003d0 00000228

3da0: 16000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

3dc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 000003ed

3de0: e9a9e200 e4196200 03da13da 00000780 00000430 3231564e 00000000 00000210

3e00: 000003d0 00000220 18c00000 00000000 00000000 00000780 00000438 50424752

3e20: 00000000 00000000 00000210 000003d0 00000228 16000000 00000000 00000000

3e40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

3e60: 00000000 00000000 00000000 00000000 000003ed e9a9e200 e9a54600 03da13d9

3e80: 00000780 00000430 3231564e 000003b0 00000008 000003d0 00000220 18c00000

3ea0: 00000000 00000000 00000780 00000438 50424752 00000000 000003b0 00000000

3ec0: 000003d0 00000228 16000000 00000000 00000000 00000000 00000000 00000000

3ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

3f00: 00000000 000003ed e9a9e200 e41b4200 03da13d6 00000780 00000430 3231564e

3f20: 00000000 00000008 000003d0 00000220 18c00000 00000000 00000000 00000780

3f40: 00000438 50424752 00000000 00000000 00000000 000003d0 00000228 16000000

3f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

3f80: 00000000 00000000 00000000 00000000 00000000 00000000 000003ed e9a9e200

3fa0: e4196a00 03da13d5 e9de3fcc e9fdde38 c0797828 c03cb1c0 00000013 00000000

3fc0: 00000000 00000000 00000000 c009a2c8 00000000 c0797828 00000000 00000000

3fe0: e9de3fe0 e9de3fe0 e9fdde38 c009a244 c0048dc8 c0048dc8 fbeb6f75 bf2eedff

[<c006e6cc>] (__wake_up_common+0x2c/0x94) from [<c006eebc>] (__wake_up+0x3c/0x50)

[<c006eebc>] (__wake_up+0x3c/0x50) from [<c03ca6a8>] (get_res_do_task+0x10c/0x5c0)

[<c03ca6a8>] (get_res_do_task+0x10c/0x5c0) from [<c03cb2bc>] (ipu_task_thread+0xfc/0xb58)

[<c03cb2bc>] (ipu_task_thread+0xfc/0xb58) from [<c009a2c8>] (kthread+0x84/0x88)

[<c009a2c8>] (kthread+0x84/0x88) from [<c0048dc8>] (kernel_thread_exit+0x0/0x8)

Code: e5b61004 e59b7004 e1560001 e241c00c (e5914000)

---[ end trace 5e701b621010734c ]---

[Tarek · Accepted solution]

I have applied this patch to Kernel 3.0.35 rel 4.1.0: linux-2.6-imx.git - Freescale i.MX Linux Tree

And re-ordered the app color components to avoid the IPUv3 IDMAC  bug: linux-2.6-imx.git - Freescale i.MX Linux Tree

The problem is not seen again so I consider it as fixed.

Thanks,

Tarek

[YixingKong]

Tarek

Had your issue got resolved? If yes, we are going to close the discussion in 3 days. If you still need help, please feel

free to reply with an update to this discussion.

Thanks,
Yixing

[Tarek]

Yixing,

The issue is not resolved. Can you please ask the IPU experts at Freescale to help?

[YixingKong]

Tarek

Had your issue got resolved by Qiang Li's reply? If yes, we are going to close the discussion in 3 days. If you still need help, please feel free to reply with an update to this discussion.

Thanks,

Yixing

[qiang_li-mpu_se]

From the log, it shows the ipu task input parameters (buffer address, width and height) will be wrong:

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input height overflow

imx-ipuv3 imx-ipuv3.1: IDMAC12's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.0: IDMAC11's EBA0 is not 8-byte aligned

I think you should check the "struct ipu_task", before call IPU_QUEUE_TASK.

[YixingKong]

Tarek

I will create a branch of your issue to internal group and assign an engineer to work on it.

Thanks,

Yixing

[Tarek]

The same Oops message again but this time in ipu1_task. The question remains what is the second IPU doing?

imx-ipuv3 imx-ipuv3.1: IDMAC12's EBA0 is not 8-byte aligned

[ALLOC] mem alloc cpu_addr = 0xff280000

start=0x42fa3000, pgoff=0x18100, size=0x7c000

wm8731_slv_mode_clock_enable: "imx-ssi.1" rate = 11289600 (= 11289600)

[ALLOC] mem alloc size = 0x4fe807

[ALLOC] mem alloc cpu_addr = 0xfed00000

start=0x50980000, pgoff=0x19000, size=0x4ff000

[ALLOC] mem alloc size = 0x14000

[ALLOC] mem alloc cpu_addr = 0xff260000

[ALLOC] mem alloc size = 0x2f1c00

[ALLOC] mem alloc cpu_addr = 0xfea00000

start=0x4f66c000, pgoff=0x18400, size=0x2f2000

vpu: memory phy:0x19800000 is phy memory

vpu: memory phy:0x199f6800 is phy memory

vpu: memory phy:0x16000000 is phy memory

vpu: memory phy:0x161f6800 is phy memory

vpu: memory phy:0x16800000 is phy memory

vpu: memory phy:0x169f6800 is phy memory

[FREE] mem freed cpu_addr = 0xff260000

[FREE] mem freed cpu_addr = 0xfea00000

[FREE] mem freed cpu_addr = 0xfed00000

size=0x4000,  page no.=0x2040

start=0x42fb9000, pgoff=0x18100, size=0x7c000

[ALLOC] mem alloc size = 0x4fe807

[ALLOC] mem alloc cpu_addr = 0xfed00000

start=0x507cd000, pgoff=0x16000, size=0x4ff000

[ALLOC] mem alloc size = 0x14000

[ALLOC] mem alloc cpu_addr = 0xffd60000

[ALLOC] mem alloc size = 0x2f1c00

[ALLOC] mem alloc cpu_addr = 0xfea00000

start=0x4f500000, pgoff=0x19c00, size=0x2f2000

vpu: memory phy:0x18400000 is phy memory

vpu: memory phy:0x185f6800 is phy memory

vpu: memory phy:0x19800000 is phy memory

vpu: memory phy:0x199f6800 is phy memory

vpu: memory phy:0x19400000 is phy memory

vpu: memory phy:0x195f6800 is phy memory

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

imx-ipuv3 imx-ipuv3.0: IDMAC11's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.0: IDMAC12's EBA0 is not 8-byte aligned

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

imx-ipuv3 imx-ipuv3.0: IDMAC11's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.1: IDMAC12's EBA0 is not 8-byte aligned

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input width overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

mxc_ipu mxc_ipu: split mode input height overflow

imx-ipuv3 imx-ipuv3.0: IDMAC12's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.0: IDMAC11's EBA0 is not 8-byte aligned

Unable to handle kernel NULL pointer dereference at virtual address 00000000

pgd = c0004000

[00000000] *pgd=00000000

Internal error: Oops: 17 [#1] SMP

Modules linked in: nfsd exportfs btmrvl_sdio btmrvl pwm_bl backlight igb

CPU: 0    Not tainted  (3.0.35-2666-gbdde708+yocto+g7097df7 #6)

PC is at __wake_up_common+0x2c/0x94

LR is at __wake_up+0x3c/0x50

pc : [<c006e6cc>]    lr : [<c006eebc>]    psr: a00f0093

sp : e9c33c30  ip : fffffff4  fp : e9c33c5c

r10: 00000004  r9 : 00000003  r8 : 00000000

r7 : 00000000  r6 : e421a0a4  r5 : 600f0013  r4 : e421a0a0

r3 : 00000000  r2 : 00000001  r1 : 00000000  r0 : e421a0a0

Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel

Control: 10c53c7d  Table: 347d404a  DAC: 00000015

Process ipu1_task (pid: 393, stack limit = 0xe9c322f0)

Stack: (0xe9c33c30 to 0xe9c34000)

3c20:                                     00000096 00000001 00200200 e421a0a0

3c40: 600f0013 00000001 00000003 00000000 e9c33d44 00000004 e9c33c84 c006eebc

3c60: 00000000 e9c33d44 00000004 e4142600 00000001 e421a08c 00000094 00200200

3c80: 00000000 c03ca6a8 00000001 c009ab58 e9c33ccc c009ab64 e421a75c c006e6f4

3ca0: c0778f80 00000001 e9c33cbc e421a764 600f0013 00000001 00000003 00000000

3cc0: 00100100 00000004 e9c33cf4 c006eebc 00000000 600f0013 00000000 e4142600

3ce0: 00000094 00200200 e9c33d44 00000004 00000000 c03cb2bc 00000000 00000000

3d00: 00000000 00000000 00000000 00000000 00000002 e9c32000 c0745d0c c07977cc

3d20: c009ab58 00000001 e9c32038 00000000 00000000 00000000 00000000 00000031

3d40: 00000001 00000000 e9d3f400 c009ab58 e9c33d50 e9c33d50 00000780 00000430

3d60: 3231564e 000003b0 00000210 000003d0 00000220 17c00000 00000000 00000000

3d80: 00000780 00000438 50424752 00000000 000003b0 00000210 000003d0 00000228

3da0: 17000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

3dc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 000003ed

3de0: e421a600 e4137e00 01016e9a 00000780 00000430 3231564e 00000000 00000210

3e00: 000003d0 00000220 17c00000 00000000 00000000 00000780 00000438 50424752

3e20: 00000000 00000000 00000210 000003d0 00000228 17000000 00000000 00000000

3e40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

3e60: 00000000 00000000 00000000 00000000 000003ed e421a600 e9a5f000 01016e99

3e80: 00000780 00000430 3231564e 000003b0 00000008 000003d0 00000220 17c00000

3ea0: 00000000 00000000 00000780 00000438 50424752 00000000 000003b0 00000000

3ec0: 000003d0 00000228 17000000 00000000 00000000 00000000 00000000 00000000

3ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

3f00: 00000000 000003ed e421a600 e9bd9000 01016e96 00000780 00000430 3231564e

3f20: 00000000 00000008 000003d0 00000220 17c00000 00000000 00000000 00000780

3f40: 00000438 50424752 00000000 00000000 00000000 000003d0 00000228 17000000

3f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

3f80: 00000000 00000000 00000000 00000000 00000000 00000000 000003ed e421a600

3fa0: e4143e00 01016e95 e9c33fcc e9fdde38 c079781c c03cb1c0 00000013 00000000

3fc0: 00000000 00000000 00000000 c009a2c8 00000000 c079781c 00000000 00000000

3fe0: e9c33fe0 e9c33fe0 e9fdde38 c009a244 c0048dc8 c0048dc8 ffeb6ff5 bf2eedff

[<c006e6cc>] (__wake_up_common+0x2c/0x94) from [<c006eebc>] (__wake_up+0x3c/0x50)

[<c006eebc>] (__wake_up+0x3c/0x50) from [<c03ca6a8>] (get_res_do_task+0x10c/0x5c0)

[<c03ca6a8>] (get_res_do_task+0x10c/0x5c0) from [<c03cb2bc>] (ipu_task_thread+0xfc/0xb58)

[<c03cb2bc>] (ipu_task_thread+0xfc/0xb58) from [<c009a2c8>] (kthread+0x84/0x88)

[<c009a2c8>] (kthread+0x84/0x88) from [<c0048dc8>] (kernel_thread_exit+0x0/0x8)

Code: e5b61004 e59b7004 e1560001 e241c00c (e5914000)

---[ end trace 2d22059a6dcbab92 ]---

[LeonardoSandova]

Is this log coming from a 3.10.17 system?

[Tarek]

No I'm still on 3.0.35. Do you mean just using Kernel 3.10.9 Or something elese? What is the best way of doing that in Yocto?

I found something interesting, this error only occurs if I kill the application using the ctrl+c and the restart the application without a reboot.

I think stopping the application this way leaves some kernel objects in unknown state. It looks like the IPU drivers needs to clear it's variables/memory properly before starting a new session. This might be related to my previous question.

[LeonardoSandova]

If you are familiar with Yocto, upgrading a system is just a matter of using the release manifest, syncing and baking...

http://git.freescale.com/git/cgit.cgi/imx/meta-fsl-bsp-release.git/plain/imx/doc/fsl-yocto-release-m...

Control+C (just one time) is fine, it should send the pipeline to NULL, and all resources should be freed correctly......

[Tarek]

When you say 3.10.17 SYSTEM Is it just the kernel or more than that?

Can you please confirm the steps:

1. cp fsl-yocto-release-manifest_dora_3.10.17-1.0.0_beta.xml .repo/manifest.xml

2. repo sync

3. source ./setup-environment build

4. ./fsl-setup-release.sh -b build -e fb

5. bitbake image

Right?

Thanks

[LeonardoSandova]

Hi Tarek. Sorry, I was not clear. As Otavio mentioned, I meant the whole BSP. The steps you wrote are correct.

Leo

[OtavioSalvador]

I think LeonardoSandovalGonzalez means the full 3.10.17 beta BSP.

Using the community master-next branch should give you same results as it has the kernel, gpu and vpu/ipu integrated there.         

[Tarek]

Thanks Otavio.

The board I'm using is not supported in the 3.10.17 kernel and I will have to do it myself.

Is there any "how to" document to implement a new board?

Thanks

[OtavioSalvador]

No; but I backported the GPU driver to the linux-imx 3.0.35 so if the board is supported by meta-fsl-arm or meta-fsl-arm-extra you should be good.

[LeonardoSandova]

Tarek, would you mind testing on a Yocto 3.10.17 system?

I believe you are using gstreamer, so also please share the pipeline.

[Tarek]

Hi Leo,

The gstreamer pipeline:

appsrc  stream-type=0 blocksize=4069 block=TRUE max-latency=0 max-bytes=512*1024 ! vpudec frame-plus=2 ! mfw_isink sync=FALSE

To test 3.10.17 kernel I need to do a lot of work to port the board "Utilite" bsp. Compu-lab does not provide the Latest kernel bsp.

Is the new Kernel stable now? When are you releasing the BSP?

If the available Kernel is final I will start working on the port if you think this is going to improve the situation.

Thanks,

Tarek

[LeonardoSandova]

Hi Tarek,

Yocto now can build a 3.10.17 system, so it would be really good just to launch that pipeline on this version. Kernel is stable.

In the other hand, we have another scenario (endless playback and a other memory intensive tasks doing at the background) causing incremental delays for every playback and apparently it was related to memory fragmentation. We still see these delays but much shorter on latest BSP, so worth giving it a try. Unfortunately there is no gstreamer repo so we can easily point the differences, but at least you can compare tarballs.

Leo

[Tarek]

Hi Leo,

Thanks for the update. Then I will have to switch to the latest system at some point ... But one question:

In the dmesg above I notice that both IPU.0 and IPU.1 are allocating memory:

mx-ipuv3 imx-ipuv3.0: IDMAC12's EBA0 is not 8-byte aligned

imx-ipuv3 imx-ipuv3.1: IDMAC12's EBA0 is not 8-byte aligned

I only have one HDMI output and I'm not using the second output at all. Could it be some kind of corruption that cause IPU.1 to think it needs to display the image while it should remain silent?

Thanks,

Tarek