LPC4350 i2C1 LPCOpen 2.12 I2C1 Gets Stuck in Endless Loop using Chip_I2C_MasterTransfer

[lpcware]

Content originally posted in LPCWare by gregd on Tue Sep 01 14:03:24 MST 2015
Hello,

I am having problems with the Chip_I2C_MasterTransfer function in LPCOpen 2.12.  I have an application that communicates to several different I2C slaves using I2C1.  The M0 core application handles the I2C communications.  I am using the polled version without interrupts.  The application will run fine for weeks/months and then lock up.  Each time that I have seen the lockup, I connect with my IAR debugger using the "attach to running target" option and I find that the code is locked up in the Chip_I2C_EventHandlerPolling function in i2c_18xx_43xx.c.  Actually I have defined my own handler which is shown below.  The problem occurs when the SI flag never gets set in the CONSET register.  This causes and endless loop waiting for the SI bit. 

In one of the prior instances where I saw this problem, the I2EN and STA bits were both set but the SI bit was never set.  It seemed to match the description in section 46.10.6.3 of the LPC4350 user manual.  I implemented a custom My_I2C_EventHandlerPolling function (shown below) to force access to the bus as described in the manual.  I also found some other examples on LPCWare that showed fixes very similar to the one I implemented. 

In my most recent occurrence of the problem that I have captured in my IAR debugger today, only the I2EN flag is set in the CONSET register.  The check that I have requiring both the I2EN and STA bits to be set was never true.  This means that I am once again stuck in this endless loop.  Additionally, I am not sure if simply setting the STO bit is the proper way to trigger an exit out of the endless loop either.  Looking at the SDA and SCL lines on my scope, both are resting high.  Analyzing all of the other information available to me in the debugger in this case:  I was sending 3 bytes of data to an i2c humidity/temperature sensor.  From the iic structure declared in the My_I2C_EventHandlerPolling function (attached as an image file), it looks like all three bytes were transmitted, txSz = 0.  All of the flags appear to be in the normal condition as if the code had just completed sending the 3 bytes successfully, except that the status is still = I2C_STATUS_BUSY.  Apparently after sending the last byte, the SI bit was never set, or maybe cleared somehow after being set and before being detected.  The STATUS register was equal to 0xF8 so no state information is available. 

Any ideas on what may have caused this or the proper way to fix/recover from it.  I certainly wish that NXP would jump on this and solve the problem in the LPCOpen library.  I would think that NXP, being the I2C inventor would want to provide the best support of the I2C bus in their latest processors! 

Here is my custom version of the EventHandler:

// Redefined I2C_EventHandlerPolling to allow for timeouts when the
// Start Flag (SA) is set but no State Change Interrupt Flag SI gets set.
// In this case, you have to force bus access by setting the STOP flag STO
#define MAX_I2C_ATTEMPTS_BEFORE_FORCING     ( 1000000 )
/* Chip polling event handler */
void My_I2C_EventHandlerPolling(I2C_ID_T id, I2C_EVENT_T event)
{
    uint32_t attempts = 0;
struct i2c_interface *iic = get_i2c_interface(id);
volatile I2C_STATUS_T *stat;

/* Only WAIT event needs to be handled */
if (event != I2C_EVENT_WAIT) {
return;
}

stat = &iic->mXfer->status;
/* Call the state change handler till xfer is done */
while (*stat == I2C_STATUS_BUSY) {
if (Chip_I2C_IsStateChanged(id)) {
Chip_I2C_MasterStateHandler(id);
} else {
                        // 6/22/2015 GAD added to allow recovery from and error condition
                        // that doesn't allow the SI bit to be set in the CONSET register.
                        // See the following section in the users guide: 
                        // 46.10.6.3 Forced access to the I2C-bus
                        if (iic->ip->CONSET == (I2C_CON_STA | I2C_CON_I2EN)) {
                               // if we have been stuck with only the STA and I2EN bits set for
                               // a long time then force access by setting the STO bit.
                               if (++attempts > MAX_I2C_ATTEMPTS_BEFORE_FORCING) {
                                     iic->ip->CONSET = I2C_I2CONSET_STO;           
                                     attempts = 0;
                                     i2c.i2c_SI_timeouts[id]++;
                                }
                         } else
                                 attempts = 0;
                  }
}
}


Here is the main calling code:

void i2c1_read_humidity( void ) {
    I2C_XFER_T transferMCfg;
    I2C_STATUS_T status;

    if (humidity_sensor.mode == trigger_read) {
        transferMCfg.slaveAddr = HUMIDITY_SENSOR_ADDR;
        transferMCfg.rxBuff = i2c1_rx_buf;
        transferMCfg.txBuff = i2c1_tx_buf;
        transferMCfg.txSz = 3;
        transferMCfg.rxSz = 0;
        i2c1_tx_buf[0] = 0x80;
        i2c1_tx_buf[1] = 0;
        i2c1_tx_buf[2] = 0;
        Chip_I2C_MasterTransfer(I2C1, &transferMCfg);
        status = transferMCfg.status;
        if (status == I2C_STATUS_DONE) {
            humidity_sensor.mode = trigger_fetch1;
            return;
        } else {
            humidity_sensor.mode = read_failure;
        }
    }

After several weeks, it gets stuck in the call to Chip_I2C_MasterTransfer.



Thanks,
Greg Dunn

[lpcware]

Content originally posted in LPCWare by briching on Thu Oct 22 22:42:17 MST 2015
That ribbon cable sounds like it could be a contributing factor.  Have you tried slowing down the I2C bus speed to see if extra capacitance may be the culprit?  Do you have a capacitance meter that you can use to measure each bus wire?  There are some formulas used to select the proper pull-up resistor based on your design criteria, which can be foundhere.

Could you modify the I2C ribbon cable so that ground wires are between I2C bus wires?  This will likely increase the capacitance, but it might help mitigate some of the noise issues you are facing.

As for the I2C chip layer stuff, I took a quick look at what is involved with overriding the default behavior of Chip_I2C_EventHandler with a timeout.  I assume that this was the intention, since there is a routine called "Chip_I2C_SetMasterEventHandler." Looks like it requires access to the (private) i2c_interface struct defined within the c file and not the header.  So good luck overriding the default behavior without extensively modifying the lpcopen code.  Geesh.  NXP fails again.

Also, does it bother you that they issue lpcopen releases that generate several compiler warnings?  I'm about to go fix them myself because I haven't seen a new release for the lpc4357 drivers in a while now, and I'm tired of thinking I caused those problems.

Just some other ideas:
Do you have access to a scope that can help decipher the I2C bus signals?  My preference is an Agilent MSOX scope with the megazoom feature and communication (SPI and I2C) software enabled.  When it gets stuck in the handler, can you lift the SDA line to see if the peripheral is pulling it low, or if it is NXP causing it?

There is also the potential that you are using I2C outside of the scope for which the protocol was designed. 
Quote: "The maximum number of nodes is limited by the address space, and also by the total bus capacitance of 400 pF, which restricts practical communication distances to a few meters." - wikipedia.

If your ribbon cable lengths are longer than this, you are probably on the hairy edge of what is stable/robust.  Have you considered using USARTs instead?

[lpcware]

Content originally posted in LPCWare by gregd on Wed Oct 14 09:06:57 MST 2015
Hello,

Thank you for your suggestions.

I have seen this problem on several different boards.  I believe that it is a problem that can and probably will happen in real world applications at some time sooner or later and should be dealt with in code to allow for a robust application.  Our application uses expansion boards that connect over i2c so this may be a contributing factor since the i2c bus is ran over a ribbon cable off the main board.  I have mostly seen the problem while testing in our environmental test chamber.  As the compressor and heaters kick on and off, I am suspecting that noise pulses get coupled onto the i2c bus.  Each of the expansion boards has its own microcontroller to support the i2c communications.  We are running a higher level protocol over the i2c bus to communicate to the expansion boards to handle any errors that occur.  I have made several updates to the i2c driver to deal with the special cases that can occur. One of the primary things I added is implementing Forced access to the i2c bus as described in section 46.10.6.3 in the user manual UM10503.pdf.  

Section 46.10.6.4 is another area that should be handled.  A slave device that becomes out of sync with the master can hold the SDA line low.  As mentioned in the user guide, sending extra SCL clocks can get the slave device out of this mode.  I can't find any way to do this using the normal i2c registers.  As far as I know, you have to change the SCL pin to a GPIO line and manually toggle the line.  This is easily done with I2C1, but the I2C0 unfortunately does not allow configuring the SCL pin as GPIO.  On our board, we connected another GPIO line to this pin to allow manually giving SCL clocks.

I have added timeouts to any loop that occurs in the driver waiting for some condition to continue.  After a timeout, I try to kick off the i2c by setting forcing access to the bus.  I still need to add support for manually giving the SCL clock pulses if SDA is stuck low.  If the SCL line is held low for some reason then your are just out of luck.  It would be nice if these recovery techniques would be incorporated into the standard LPCOpen driver so everyone can take advantage of them.  I never seem to get any positive feedback for NXP as to any intentions to improve the driver.  They indicate that the endless loops are there to aid in debugging - not much help when your application is running (or actually not running) at a customer.

I am only accessing the i2c peripheral from the M0 core so there should not be any interactions between the two cores.

Thanks again for your response. 

[lpcware]

Content originally posted in LPCWare by briching on Tue Oct 13 20:39:16 MST 2015
I encountered a similar problem with my I2C bus.  My solution was to switch to a different board that had a different LPC4357 on it.  I have not seem the problem since, so my guess is that I had bad hardware.  I recall doing something accidentally to some of the I2C pins (accidentally connecting them to ground or VCC for a while), which may have originally caused the failure.  The problem was that it was intermittent, and was very difficult to debug.  If you haven't already, I suggest you switch to another hardware setup and see if that fixes the problem.

There is also the chance that the other core is intercepting the event and invoking its own I2C handler, leaving your current core to sit there and wait for an event that never arrives.  Check to make sure that your second core isn't doing anything with that peripheral.

Hope that helps.