iMX6S I2C locks up in multi-master mode

[kvv]

Hello,

1. System description:

A custom board with iMX6S SoC is connected via I2C bus to SMBus controller of some x86 board(with Intel 8 Series Chipset). There are several temperature and voltage sensors on the x86 board, which are physically connected/soldered to the SMBus. The iMX6S module periodically reads these sensors to decide if h/w is functional.

And a few words about software: a standard Yocto BSP from http://freescale.github.io/ with 3.10.17 kernel is launched on our iMX6S and some decent Linux distribution(like Fedora 20) is used on the x86 board.

All works well.

2. Problem:

As we don't know whether the SMBus controller on x86-board will be used by customers(e.g. to access the same sensors or to communicate with some other device), we've decided to perform some stress tests: we continuously read the same register on the same sensor from both sides simultaneously(from iMX6 and from x86 board). After several minutes(1-30) the bus(I2C/SMBus) locks up: some device holds a SDA line low all the time. And when we manually simulate some clocks on SCL, the SDA becomes high again.

This problem:

• is reproduced with other sensors/devices on SMBus and with other x86 boards(with different chipsets);
• can't be reproduced with only one I2C/SMBus master is present;
• can't be reproduced with other I2C controllers(when iMX6 is replaced by another board).

3. Our attempts to solve/work around it:

Unfortunately for us, we can't determine a root cause of this problem. But as such problems(I2C lock up) are very common, we've decided to integrate some workaround into i2c-imx driver. And it seems that a workaround that was proposed for iMX53 SoC doesn't work on iMX6:

"The following procedure can be used to force this I2C module to generate SCLn so that the device driving SDAn can finish its transaction:

1.1. Disable the I2C module and set the master bit by setting I2CnCR to 0x20.

1.2. Enable the I2C module by setting I2CnCR to 0xA0.

1.3. Read I2CnDR.

1.4. Return the I2C module to slave mode by setting I2CnCR to 0x80."

(From https://community.freescale.com/thread/316813 )

In our case, when SDA is low, we can't set the master bit to 1, because the I2C controller instantly reverts it to 0. And at that time we can't observe any activity on SCL line with oscilloscope. (Note: this bit sets to 1 perfectly when SDA is high :smileysad: ).

Probably we are doing something wrong, or maybe another trick should be used on iMX6. But we've tried to write to other registers of this I2C controller when SDA is low - all the same and the SCL pulses are not generated.

It seems that the only possible solution(from software point of view) is to reconfigure SCL as GPIO and use it to generate several clocks. Then to reconfigure it back (but it's too clunky).

If someone had experienced similar problem, could you please share your thoughts on it? Very probably that we've missed something important.

Best regards,

Viktor.

[Yuri]

  As for the problem in itself (SDA low) – perhaps it makes sense to

adjust I2C pull up resistors to avoid noisy environment influence.
Please take a look at I2C specs for resistor values range. 

  The idea to reconfigure SCL as GPIO looks as reasonable and

also may be applied.

Have a great day,
Yuri

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[kvv]

Hello Yuri,

JFYI some results of our investigation/work:

We haven't found a root cause but implemented the "software" workaround, which was described above, and successfully tested it on our system. Corresponding patches can be found in the attachment(some clean up of the code may be needed).

Please note the following snippet which verifies that the bus is really locked up(and not held by another I2C master).:

+      for (i = 0; i < 10; i++) {

+              temp = gpio_get_value(i2c_imx->gpio_pin);

+              if (temp == 0) {

+                      dev_dbg(&i2c_imx->adapter.dev, "bus is busy\n");

+                      goto return_to_default;

+              }

+              udelay(3);

This code should be reworked by using GPIO interrupts instead(see description of GPIOx_ICR registers in datasheet).

Also we had to revert the following patch: http://comments.gmane.org/gmane.linux.drivers.i2c/19593 as our I2C bus locks up less often without it.

As this result is acceptable for us, we are going to stop any further investigation.

Thanks again for your response.

Best regards,

Viktor.

[kvv]

Hello Yuri,

big thanks for your reply! I'll provide a detailed feedback as soon as we give it a try.

Best regards,

Viktor.

[kvv]

As for the problem in itself (SDA low) – perhaps it makes sense to

adjust I2C pull up resistors to avoid noisy environment influence.
Please take a look at I2C specs for resistor values range. 

Just several results of our testing. We've tried all available values but no luck so far(I2C periodically locks up):

- 100K Ohm Pull Down;

- 47K Ohm Pull Up;

- 100K Ohm Pull Up;

- 22K Ohm Pull Up.

Our h/w engineer also can't find any problems with physical pull up resistors: both SCL and SDA lines on both sides are pulled up by 10K resistors.

Thus, for now we are trying to decide which workaround should be used:

- "software": to pinmux the SCL signal into GPIO when SDA is locked up and generate several clocks;

- "hardware": to connect another GPIO pin to the SCL line and generate pulses with it.

We are going to test both options and compare the results/costs. I will update this post as soon as I get them.

Best regards,

Viktor.