Lost I2C interrupts when Linux boots on LS1043A

[met]

Hi all,

We are struggling with sporadic I2C failures during boot of boards with the LS1043 and most likely the LS1046A, when Linux probes devices from the device tree. It happens with Linux v4.14 from LSDK-1809 and LSDK-1806, and we have not seen it with Linux v4.1 from fsl-sdk-v2.0.x.

I am at my wit's end with this issue, and hope someone can help. At this point I can't tell if it is a hardware or software problem. For all I know, the 4.14 kernel may have uncovered a silicon bug..

The failure is reproducible on the LS1043ARDB. I built an image with flexbuild from LSDK-1809, using the attached Linux kernel config and device tree. (The .diff file can be "git apply"-ed.) The kernel config matches what we use on our affected boards. I modified the RDB device tree to ensure a comparable number of I2C drivers are loaded as on our own boards.

The failures occur during around 20 % of boots on our boards. When it fails on the LS1043ARDB, the probe error can be found in dmesg:

root@TinyDistro:~# dmesg | grep -E "(i2c|0-00[0-9a-f]{2}[ :])"
[ 0.391795] imx-i2c 2180000.i2c: can't get pinctrl, bus recovery not supported
[ 0.399578] i2c i2c-0: IMX I2C adapter registered
[ 0.404351] i2c i2c-0: using dma0chan16 (tx) and dma0chan17 (rx) for DMA transfers
[ 0.938649] at24 0-0052: 65536 byte 24c512 EEPROM, writable, 1 bytes/write
[ 0.946284] at24 0-0053: 65536 byte 24c512 EEPROM, writable, 1 bytes/write
[ 1.366416] rtc-ds1307: probe of 0-0068 failed with error -110
[ 1.373454] i2c /dev entries driver
[ 1.379686] ina2xx 0-0040: power monitor ina220 (Rshunt = 1000 uOhm)
[ 1.387350] lm90 0-004c: 0-004c supply vcc not found, using dummy regulator

Note that it fails in between successful device probes. The bus works fine before and after the failures.

As for the error, -110 is -ETIMEDOUT (see errno.h), and I've found that it stems from the first call to i2c_imx_trx_complete() in i2c_imx_write(), in the file drivers/i2c/busses/i2c-imx.c. After the slave address has been written, the driver times out while waiting for the transaction complete interrupt. The interrupt is enabled and the interrupt flag gets set, but the interrupt handler does not run. I have scoped the I2C bus and see that the LS1043 sends the slave address and that the slave ACKs, but then nothing more happens before the timeout triggers.

For reference, here's the code:

- drivers/i2c/busses/i2c-imx.c line 801:

static int i2c_imx_write(struct imx_i2c_struct *i2c_imx, struct i2c_msg *msgs)
{
    int i, result;

    dev_dbg(&i2c_imx->adapter.dev, "<%s> write slave address: addr=0x%x\n",
        __func__, msgs->addr << 1);

    /* write slave address */
    imx_i2c_write_reg(msgs->addr << 1, i2c_imx, IMX_I2C_I2DR);
    result = i2c_imx_trx_complete(i2c_imx);
    if (result)
        return result;

...

- drivers/i2c/busses/i2c-imx.c line 506:

static int i2c_imx_trx_complete(struct imx_i2c_struct *i2c_imx)
{
    wait_event_timeout(i2c_imx->queue, i2c_imx->i2csr & I2SR_IIF, HZ / 10);

    if (unlikely(!(i2c_imx->i2csr & I2SR_IIF))) {
        dev_dbg(&i2c_imx->adapter.dev, "<%s> Timeout\n", __func__);
           return -ETIMEDOUT;
    }

...

...there's not much that can go wrong there. I've double checked that the interrupt flag is set before -ETIMEDOUT is returned. It is. But the interrupt handler never runs, so i2csr does not get updated and the queue does not get "woken up".

On our own boards, it fails in this way for 1-3 successive devices in the probing sequence. If I remove the affected devices from the device tree, the failures happen with the next devices in the probing sequence, even if they are on a different I2C bus. So the failures do not happen for specific I2C devices, nor for a specific I2C bus. It is just a window in time at boot where I2C module interrupts seem to get lost.

I have not seen the failures after the system has booted, i.e., if I repeatedly load/unload drivers for I2C devices.

Please help!

Best regards,
Martin Etnestad