RT685 i3c port with Zephyr - driver with i2c devices dies after minutes of use

[galenc]

I have a system based around the MIMXRT685-EVK running Zephyr.  I have the i3c port configured and interfacing with a few i2c devices.  They all work for some time, but depending of frequency of communication the bus/driver will just die after a few minutes.

I experienced this on the evk and on our custom board.  And similarly with all the peripherals on the bus or just a single one. 

here you can see the scope (yellow SDA, green SCL, violet device interrupt) it just stops trying any comms on the bus when it should be servicing the interrupt:logic output

It happens for this device which is imu - lsm6dso (a zephyr built in driver), sampling at higher frequency from interrupts. Even communicating with another device at slower frequency it will stay up longer but eventually crash (maybe 5-10 minutes with a polling message every second).

Here is the warnings we saw from the i3c_mcux driver just before the bus goes dead.  the errors are failed attempts to read from device registers (the bus in the scope has no activity from subsequent attempts, shown in red as errors):
warning from i3c_mcux before bus crash

Has anyone seen any issues similar with the i3c bus and zephyr implementation specifically?

our whole bus is essentially i2c devices we don't need the features of i3c (we are using it because we are using all the other ports) if there is a way it might be more stable if we "force" it into i2c mode all the time? 

It don't know if there is a bug in the driver or some way to better configure it?  we are using it pretty much as it came.  see device tree sections:

&i3c0 {
	status = "okay";

	i2c-scl-hz = <400000>;
	i3c-scl-hz = <400000>;
	i3c-od-scl-hz = <400000>;

	clk-divider = <12>;
	clk-divider-slow = <1>;
	clk-divider-tc = <1>;

	pinctrl-0 = <&pinmux_i3c>;
	pinctrl-names = "default";

	lsm6dso0: lsm6dso@6a0000000000000050 {
		status = "okay";
		compatible = "st,lsm6dso";
		reg = <0x6a 0x00 0x50>;
		irq-gpios = <&gpio1 10 GPIO_ACTIVE_HIGH>;
	};
};

	pinmux_i3c: pinmux_i3c {
		group0 {
			pinmux = <I3C0_SCL_PIO2_29>,
					 <I3C0_SDA_PIO2_30>;
			input-enable;
			slew-rate = "slow";
			drive-strength = "high";
		};

		group1 {
			pinmux = <I3C0_PUR_PIO2_31>;
			slew-rate = "normal";
			drive-strength = "normal";
		};
	};

Any help is much appreciated.

Thanks, Galen 

[galenc]

Yes, I agree this is a good idea for debug. I will set this up. I will be traveling for the next two weeks (starting tomorrow) but will report my results when I return.

Thanks for your continued help on this issue.

[Dezheng_Tang]

I tried to isolate the problem if Zephyr OS causes the problem or not and have run a standalone test on the I2C device using I3C controller. Here is a what I have done: on RT685 EVK board, there is a codec WM8904 on the I3C bus, if you use any audio related SDK example, this codec will be initialized using I3C controller (over I2C), I made minor change to do register reading periodically, monitor the bus and have never seen timeout happened unless you force it into one of these conditions below. It doesn't look our I3C controller could time out easily unless the STALL period is > 100us.

TIMEOUT Time-out error. The module has stalled too long in a frame. This happens:
• When the TX FIFO or RX FIFO is not handled and the bus is stuck in the middle of a
message,
• When no STOP was issued and between messages,
• When IBI manual is used and no decision was made.
The maximum stall period is 10 kHz or 100 us.

It's hard to time how often bus recovery should be called, hopefully, investigate why timeout happens so often first.

I don't know about the IO error, it looks like MCTRL and MSTATUS reading is OK, completion bit is set correctly.

[00:24:52.345,217] <dbg> i3c_mcux: mcux_i3c_has_error: ERROR: MCTRL 0x0000d250 MSTATUS 0x00009403 MERRWARN 0x00100000

[Dezheng_Tang]

Correction: I DON'T think bus will die if timeout bit is set. 

[Sam_Gao]

Hi @galenc 

I chcked the i3c source code from Zephyr https://github.com/zephyrproject-rtos/zephyr/blob/70f55bccf8af2866331f6fd84e05c600f4457704/drivers/i... 

These codes in this file which follow Apache-2.0 were contributed from freescale, intel, nxp. I may need a little time to check if I can find right person to deal with this. Also, you can use Zephyr community ( Discussions and Issues ), they will support you directly.

About this I3C question(emit, IBI), I would like to share common driver from NXP: 

https://github.com/nxp-mcuxpresso/mcux-sdk/blob/main/drivers/i3c/fsl_i3c.c#L690

[Sam_Gao]

Hi @galenc 

It seems some issues releated to I3C, Zephyr, lsm6dso sensor and SDK. I try to clairfy what I found one by one as below, hope these comments help you find the root cause.

1. It seems support wm8904 codec as default I3C in Zephyr

 https://github.com/zephyrproject-rtos/zephyr/blob/main/boards/nxp/mimxrt685_evk/mimxrt685_evk_mimxrt... 

https://github.com/zephyrproject-rtos/zephyr/blob/main/boards/nxp/mimxrt685_evk/mimxrt685_evk-pinctr... 

&i3c0 {
	pinctrl-0 = <&pinmux_i3c>;
	pinctrl-names = "default";

	status = "okay";

	audio_codec: wm8904@1a0000000000000000 {
		compatible = "wolfson,wm8904";
		reg = <0x1a 0 0>;

		clock-source = "MCLK";

		clocks = <&clkctl0 MCUX_AUDIO_MCLK>;
		clock-names = "mclk";
	};
};

 And actually I find it seems only support I2C(not mentioned I3C) from the introdcution https://github.com/zephyrproject-rtos/zephyr/blob/main/boards/nxp/mimxrt685_evk/doc/index.rst 

PIO0_17	I2C	I2C SDA
PIO0_18	I2C	I2C SCL

2. If you use EVKMIMXRT685, I would like to recommend you to use offical SDK instead for testing. Please note the necessary board setting of I3C before your quick start, https://github.com/nxp-mcuxpresso/mcux-sdk-examples/tree/main/evkmimxrt685/driver_examples/i3c/polli... 

 3. Also please check the sensor part(salve) independently whether this sensor support I3C or not.

4. Currently it is diffcult to give clear direction or sw/hw issue.

B.R,

Sam