iMX6SX i2c might be locked and can't recover

[mus]

We recently enabled a i2c device, vl53L0X. Sometimes soft rebooting causes the i2c bus this device is connected to become inoperable. Removing and reinserting the drivers didn’t help. Hard rebooting fixes the issue, but this is not an option for the end user.

I see that the Linux kernel already has i2c bus reset in the kernel driver. We are using v4.9.x.

When we try to unbind the specific i2c driver, kernel crashes. All loadable modules are unloaded before unbinding. Unbinding and rebinding the other 2 i2c buses doesn’t cause this issue.

# echo "21a8000.i2c" > /sys/bus/platform/drivers/imx-i2c/unbind

[  407.226686] Unable to handle kernel paging request at virtual address 7f0fc33c

[  407.231336] pgd = 993f0000

[  407.233772] [7f0fc33c] *pgd=986be811, *pte=00000000, *ppte=00000000

[  407.240040] Internal error: Oops: 7 [#1] PREEMPT ARM

[  407.244966] Modules linked in: [last unloaded: galcore]

[  407.250196] CPU: 0 PID: 174 Comm: sh Tainted: G           O    4.9.11-VenOS #98

[  407.257468] Hardware name: Freescale i.MX6 SoloX (Device Tree)

[  407.263286] task: 98166a80 task.stack: 9851c000

[  407.267833] PC is at i2c_do_del_adapter+0x14/0x80

[  407.272499] LR is at __process_removed_adapter+0x14/0x1c

[  407.277792] pc : [<804239a8>]    lr : [<80423a28>]    psr: 20040013

[  407.277792] sp : 9851dd58  ip : 9851dd80  fp : 9851dd7c

[  407.289239] r10: 98c30d00  r9 : 981b81c8  r8 : 981b8010

[  407.294450] r7 : 98108210  r6 : 80423a14  r5 : 7f0fc2d4  r4 : 00000000

[  407.300958] r3 : 00000001  r2 : 00000000  r1 : 981b8010  r0 : 7f0fc2d4

[  407.307478] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none

[  407.314589] Control: 10c53c7d  Table: 993f0059  DAC: 00000051

[  407.320317] Process sh (pid: 174, stack limit = 0x9851c208)

[  407.325872] Stack: (0x9851dd58 to 0x9851e000)

[  407.330221] dd40:                                                       00000000 981b8010

[  407.338392] dd60: 80423a14 98108210 98c30d0c 981b81c8 9851dd8c 9851dd80 80423a28 804239a0

[  407.346550] dd80: 9851ddb4 9851dd90 803bd484 80423a20 98033f5c 983c3634 60040013 981b8010

[  407.354709] dda0: 981b8010 981b8010 9851dddc 9851ddb8 80424860 803bd444 981b8010 98108200

[  407.362869] ddc0: 0000000c 98108210 98c30d0c 9851df78 9851de04 9851dde0 8042904c 804247e8

[  407.371027] dde0: 98108210 98108210 0000000c 80923e1c 98c30d0c 9851df78 9851de1c 9851de08

[  407.379187] de00: 803c09f0 80429024 98108210 80923e1c 9851de34 9851de20 803bed9c 803c09d4

[  407.387347] de20: 98108244 98108210 9851de4c 9851de38 803bee3c 803bed04 98108210 80920c80

[  407.395507] de40: 9851de6c 9851de50 803bdb70 803bee24 803bdaf0 00000000 98c30140 00000000

[  407.403666] de60: 9851de84 9851de70 803bd180 803bdafc 803bd158 00000000 9851de9c 9851de88

[  407.411826] de80: 80246a90 803bd164 00000000 00000000 9851ded4 9851dea0 802461e4 80246a5c

[  407.419985] dea0: 00000000 00000000 982ec310 802460f0 99460d80 9851df78 9851df78 00000000

[  407.428145] dec0: 0000000c 00000000 9851df44 9851ded8 801dd2d8 802460fc 00000007 801124d4

[  407.436306] dee0: 00068b20 9851dfb0 008e0274 00000000 9851dfac 9851df00 801013c4 801124e0

[  407.444465] df00: 98166a78 9851df50 9851df4c 9851df18 8012428c 8014ae18 9851df54 fffffff6

[  407.452624] df20: 0000000c 99460d80 008e3ea0 9851df78 00000000 0000000c 9851df74 9851df48

[  407.460784] df40: 801de1b0 801dd2b4 80124940 8013b268 99460d80 99460d80 00000000 00000000

[  407.468943] df60: 008e3ea0 0000000c 9851dfa4 9851df78 801defb8 801de110 00000000 00000000

[  407.477103] df80: 00000000 00000001 008e3ea0 00000004 80107bc4 9851c000 00000000 9851dfa8

[  407.485262] dfa0: 80107a00 801def80 00000000 00000001 00000001 008e3ea0 0000000c 00000000

[  407.493423] dfc0: 00000000 00000001 008e3ea0 00000004 000c5c7c 00000020 00000000 00092ed4

[  407.501583] dfe0: 00000000 7ed9965c 0001a588 76eb6538 40040030 00000001 9bfd9811 9bfd9c11

[  407.509717] Backtrace:

[  407.512198] [<80423994>] (i2c_do_del_adapter) from [<80423a28>] (__process_removed_adapter+0x14/0x1c)

[  407.521380]  r9:981b81c8 r8:98c30d0c r7:98108210 r6:80423a14 r5:981b8010 r4:00000000

[  407.529115] [<80423a14>] (__process_removed_adapter) from [<803bd484>] (bus_for_each_drv+0x4c/0x9c)

[  407.538130] [<803bd438>] (bus_for_each_drv) from [<80424860>] (i2c_del_adapter+0x84/0x1a4)

[  407.546365]  r6:981b8010 r5:981b8010 r4:981b8010

[  407.550980] [<804247dc>] (i2c_del_adapter) from [<8042904c>] (i2c_imx_remove+0x34/0x134)

[  407.559048]  r9:9851df78 r8:98c30d0c r7:98108210 r6:0000000c r5:98108200 r4:981b8010

[  407.566782] [<80429018>] (i2c_imx_remove) from [<803c09f0>] (platform_drv_remove+0x28/0x48)

[  407.575107]  r9:9851df78 r8:98c30d0c r7:80923e1c r6:0000000c r5:98108210 r4:98108210

[  407.582842] [<803c09c8>] (platform_drv_remove) from [<803bed9c>] (__device_release_driver+0xa4/0x120)

[  407.592019]  r5:80923e1c r4:98108210

[  407.595598] [<803becf8>] (__device_release_driver) from [<803bee3c>] (device_release_driver+0x24/0x30)

[  407.604866]  r5:98108210 r4:98108244

[  407.608444] [<803bee18>] (device_release_driver) from [<803bdb70>] (unbind_store+0x80/0xfc)

[  407.616758]  r5:80920c80 r4:98108210

[  407.620333] [<803bdaf0>] (unbind_store) from [<803bd180>] (drv_attr_store+0x28/0x34)

[  407.628050]  r7:00000000 r6:98c30140 r5:00000000 r4:803bdaf0

[  407.633716] [<803bd158>] (drv_attr_store) from [<80246a90>] (sysfs_kf_write+0x40/0x4c)

[  407.641584]  r5:00000000 r4:803bd158

[  407.645171] [<80246a50>] (sysfs_kf_write) from [<802461e4>] (kernfs_fop_write+0xf4/0x1f8)

[  407.653303]  r5:00000000 r4:00000000

[  407.656892] [<802460f0>] (kernfs_fop_write) from [<801dd2d8>] (__vfs_write+0x30/0x118)

[  407.664776]  r10:00000000 r9:0000000c r8:00000000 r7:9851df78 r6:9851df78 r5:99460d80

[  407.672570]  r4:802460f0

[  407.675114] [<801dd2a8>] (__vfs_write) from [<801de1b0>] (vfs_write+0xac/0x170)

[  407.682397]  r9:0000000c r8:00000000 r7:9851df78 r6:008e3ea0 r5:99460d80 r4:0000000c

[  407.690130] [<801de104>] (vfs_write) from [<801defb8>] (SyS_write+0x44/0x98)

[  407.697155]  r9:0000000c r8:008e3ea0 r7:00000000 r6:00000000 r5:99460d80 r4:99460d80

[  407.704897] [<801def74>] (SyS_write) from [<80107a00>] (ret_fast_syscall+0x0/0x3c)

[  407.712432]  r9:9851c000 r8:80107bc4 r7:00000004 r6:008e3ea0 r5:00000001 r4:00000000

[  407.720159] Code: e92ddbf0 e24cb004 e1a05000 e1a08001 (e5b52068)

[  407.726400] ---[ end trace a5443a099ca774b7 ]---

[  407.731222] Kernel panic - not syncing: Fatal exception

[  407.736020] Rebooting in 1 seconds..[    0.000000] Booting Linux on physical CPU 0x0

How do we fix this issue?

[igorpadykov]

Hi mus

one can try suggestions (in particular reprogram the I2C Clock pins as a

GPIO and toggle (at 400kHz or so) nine times) provided on

I2C reset 

Try with nxp linux from source.codeaurora.org/external/imx/linux-imx  repository
linux-imx - i.MX Linux kernel 

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[mus]

Igor,

As far as I can tell, these suggestions are for older kernels. In the post, it states that "So this facility was added in Linux 3.10, but NXP didn't add it to their driver until 4.4".

Recovery functionality is in the kernel driver already in function i2c_imx_init_recovery_info:

i2c-imx.c\busses\i2c\drivers - linux-imx - i.MX Linux kernel 

As you can see in my post, trying to unbind the i2c driver is causing a crash. If I leave all modules loaded, unbinding hangs. How can I invoke the recovery?

[igorpadykov]

Hi mus

in general you can try to debug it with oscilloscope.

If  vl53L0X  device has reset signal, try to assert it.

Issue may be related to that specific vl53L0X i2c device, not i.MX6SX,

as i2c recovery worked fine as stated in mentioned links.

Suggest to apply to vendor tech support of that i2c device for any

specific "recovery" options.

For "soft rebooting " nxp recommends to design board in such way that

all power supplies were shortly turned off. This provides reliable solution

for reboot. On i.MX6SX Sabre SD design it is done with PMIC_PWRON signal,

p.20 SPF-27962.pdf schematic :

Design files, including hardware schematics, Gerbers, and OrCAD files

Best regards
igor

[mus]

If the I2C SDA is stuck low. How can we induce the bus reset?

I see that there is recovery logic included in the i2c driver in i2c_imx_init_recovery_info.