uart driver, best design for async messages, mqx missing char's in blocking mode

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

uart driver, best design for async messages, mqx missing char's in blocking mode

Jump to solution
4,649 Views
Cdn_aye
Senior Contributor I

K20, MQX 4.01

We are talking to a serial uart that may asynchronously receive messages. We can't just read in non blocking mode because we may miss the message window or pick up parts of the message for the same reason. So we do a fstatus check for a character in non blocking mode and use a timer when a message has started to arrive, then guess at how long to delay to receive a complete message. Herein is the problem. We have to empirically guess at how long between message requests and how long are the message responses because we don't know the length of the message. This is a poor way to work obviously.

Is there any way to just read a full message where a message is defined as the first character received and the last character of the message is a null, for example, and then have an alert/irq/event fires that says. "A message exists"?

Is there anyway to set the driver control so that the serial stream is buffered and the buffer is determined to be complete when a null or other char is received in mqx? We have tried doing an fstatus check followed by a blocking read but char's are missed in the stream, even at 9600 baud.

Any advice would be appreciated.

Thanks

Robert

1 Solution
2,052 Views
matthewkendall
Contributor V

Is there any way to just read a full message where a message is defined as the first character received and the last character of the message is a null, for example, and then have an alert/irq/event fires that says. "A message exists"?

There are two separate things going on here and it is usually best to deal with them separately. One is dealing with the serial stream of characters, identifying the beginning and end of packets, and extracting the messages (a framing layer). The other is interpreting the messages and acting on them (a protocol layer).


This would be done with two tasks. One is a low-level, high-priority task that handles the framing. It reads characters from the input port with a blocking read, accumulates them in a buffer and does whatever simple parsing you need to identify the start and end of a packet and extract the message. When it has got a complete message it sends it to a high-level, lower priority task which handles the protocol. The most straightforward way to send it would be via a MQX message queue. The protocol task blocks reading the message queue, or just polls it at its convenience.

View solution in original post

19 Replies
2,053 Views
matthewkendall
Contributor V

Is there any way to just read a full message where a message is defined as the first character received and the last character of the message is a null, for example, and then have an alert/irq/event fires that says. "A message exists"?

There are two separate things going on here and it is usually best to deal with them separately. One is dealing with the serial stream of characters, identifying the beginning and end of packets, and extracting the messages (a framing layer). The other is interpreting the messages and acting on them (a protocol layer).


This would be done with two tasks. One is a low-level, high-priority task that handles the framing. It reads characters from the input port with a blocking read, accumulates them in a buffer and does whatever simple parsing you need to identify the start and end of a packet and extract the message. When it has got a complete message it sends it to a high-level, lower priority task which handles the protocol. The most straightforward way to send it would be via a MQX message queue. The protocol task blocks reading the message queue, or just polls it at its convenience.

2,052 Views
Cdn_aye
Senior Contributor I

Hi Matthew

That is a good explanation and delineates the issues clearly, thank you.

I think that is close to what I have done  apart from using a message on the framing layer to talk with the protocol layer.

I have a task monitoring the messages and driving a state machine (protocol layer) and a driver that pulls in the messages (framing layer). The read of the driver is set to timeout if no messages are received in a certain timeout. The problem is the blocking read (framing layer) delays everything and slows down the responses because of the accumulated timeouts.

I don't know when a char will arrive, so the only way I knew to do this is with a non blocking read and with a fstatus check with a timeout. When the first char is detected, then the driver loops reading the char's until we timeout on the last char; that is assumed to be the frame. However this also blocks the task that issued the driver call until the driver returns with a buffer or not.  It works but in my view is a non deterministic way to do things. I would like a driver that tells me that a buffer is received.

David suggested using interrupt driven serial driver with a terminating char. I haven't used the ittya driver so I was wondering if this is the way to go, or maybe I am not understanding the principle clearly. I am assuming that if I set the driver terminating char that mqx would notify me when the driver found the terminating char and it had filled in the buffer with char's if any. The others would be discarded, then I could send a message to the protocol layer saying the message is complete without any timeouts.

Regards

Robert

0 Kudos
2,052 Views
matthewkendall
Contributor V

If you want to distinguish incoming packets by observing the gap between them, then you have the problem of wanting your low-level task to wait for two different things: a character or a timeout. That is non-trivial because MQX only provides a mechanism to block on a single thing.

One approach to this problem is to turn all the things you want to wait for into events. A task can block waiting for any event in a group.

For example (omitting boilerplate details for clarity):

int c;

LWEVENT_STRUCT evt;

mainTask(...) {

  _lwevent_create(&evt, LWEVENT_AUTO_CLEAR);

  _task_create(...); /* protTask */

  _task_create(...); /* frmTask */

  _task_create(...); /* charTask */

}

charTask(...) {

  fp = fopen("ittya:");

  while (true) {

    c = fgetc(fp);

    _lwevent_set(&evt, 0x01);

  }

}

frmTask(...) {

  _timer_start_periodic_every(frmNotify, NULL, TIMER_ELAPSED_TIME_MODE, 10);

  while (true) {

    _lwevent_wait(&evt);

    which = _lwevent_get_signalled();

    if (which & 0x01) {

      /* put recently arrived char c in buf */

    }

    if (which & 0x02) {

      /* timeout: packet in buf is complete,

         copy it into a message and send it to MSGQ_PROT */

    }

  }

}

void frmNotify(_timer_id timer, pointer data, uint_32 secs, uint_32 msecs) {

    _lwevent_set(&evt, 0x02);

}

protTask(...) {

  _msgq_open(MSGQ_PROT);

  while (true) {

    pMsg = _msgq_receive(MSGQ_ANY_QUEUE, 0);

    switch (pMsg->msgId) {

      case MESSAGE_FROM_FRM:

        /* parse the packet */

        break;

      case ANOTHER_TYPE_OF_MESSAGE:

         /* do something else */

         break;

    }

  }

}

That's pretty horrid and really needs frmTask to signal to charTask that it has got the character before charTask attempts to read another one, but you get the idea. frmTask waits on events and all the things it can wait for are turned into same; the timer via a callback and received chars via another task.

If you have the freedom to change your low-level format so that packets are distinguished not by gaps but by something in the data (e.g. a terminating character, or a header containing a length that allows you to count subsequent bytes) then your task is easier. If this is the case then you can just do something like this:

mainTask(...) {

  _task_create(...); /* protTask */

  _task_create(...); /* frmTask */

}

frmTask(...) {

  fp = fopen("ittya:");

  while (true) {

    c = fgetc(fp);

    if (c == TERM_CHAR) {

      /* packet in buf is complete,

         copy it into a message and send it to MSGQ_PROT */

    } else {

      /* add c to buf */

    }

  }

}

protTask(...) {

  _msgq_open(MSGQ_PROT);

  while (true) {

    pMsg = _msgq_receive(MSGQ_ANY_QUEUE, 0);

    switch (pMsg->msgId) {

      case MESSAGE_FROM_FRM:

        /* parse the packet */

        break;

      case ANOTHER_TYPE_OF_MESSAGE:

         /* do something else */

         break;

    }

  }

}

What David was suggesting was that you could move the code that looks for the terminating character into the MQX serial driver, thus avoiding the need for the extra task. That seems a little complicated in this case.

2,052 Views
Cdn_aye
Senior Contributor I

Hi Matthew

I agree and have tried something similar, but the whole concept of a driver is going sideways and is becoming a lot of re-inventing the wheel. Other OS's do this as a basic brick and mortar function.

What do you think about implementing the RS-485 spec, that is, using the DTR CTS lines to control flow? The driver has this as an option. Do you think this could solve problem because the driver and hardware would deliver the chars when they are definitely there? One of the major issues right now is as you point out,is  implementing a method to find the character and a second to determine if we have timed out or the stream is ended. I did this in a single task but I think your way of using two tasks with a message queue is a better design.

We are doing a spin of the board so what are your thoughts on using the DTR CTS signals, do you think that could solve the issue? I am trying to avoid further errors due to a lack of understanding on my part.

Thanks for the advice.

Robert

0 Kudos
2,052 Views
dave408
Senior Contributor II

I'm glad I found your post, because it sounds a lot like the process I have gone through with my modbus RTU protocol handler.  I started by handling the protocol in a high priority MQX task.  It kind of works, but I am running into a problem where data that should be in the UART is magically disappearing.  I don't know if it's because of overhead or a bug in the UART driver.

As a result, and with the suggestions from mjbcswitzerland​, I am going to try doing everything in the RX callback that gets called from the UART ISR.  Currently I have the FIFO disabled and have put my previous modbus FSM in the callback.  It assembles the packet, and was planning on sending the packet to my MQX task via a message queue.  I'm glad I came across this when looking for message queue information, as it provides some confirmation that this approach makes sense.

I know this is late relative to your original posting date, but thought I'd provide a link to my other post in case in helps someone else searching the forum.

UART_DRV_ReadDataBlocking seems to allow FIFO data to be lost

0 Kudos
2,052 Views
Cdn_aye
Senior Contributor I

Hi Dave

Thank you for your response.

Our current design has the problem that you and Matthew have correctly determined. The last character of a string cannot be differentiated from a missed character within the string. This causes a long terminating time delay at the end of the string (in our design) for every valid string received. This is a problem when the string is being used for control functions because the entire system is queued to wait for the string time out, which is dead useless time.

To minimize the impact I am changing to the highest reliable baud rate. This reduces the timeout between characters and therefore the long time delay at the end of the string. But again, this is not a good solution in my estimation.

If you find a way that works and could share this I and I think others would appreciate it.

Robert

0 Kudos
2,052 Views
dave408
Senior Contributor II

Hi Cdn_aye​,

This doesn't necessarily answer your question, but as I may have said before, my problem was with implementing the Modbus RTU protocol.  The protocol differentiates between intercharacter and interpacket delays -- 750us max between characters, and 1750us max between packets (if baud rate >= 19200).  This allows you to differentiate between data corruption and packet completion.  Maybe you can use these timings in your protocol to make it easier to make that distinction.

If you look at my thread UART_DRV_ReadDataBlocking seems to allow FIFO data to be lost  you'll see a suggestion by Mark Butcher that is very elegant.  I had originally tried to implement the packet assembly as a FSM, but in the end, his interrupt-driven solution works very well.  I had to modify it a bit to work with KSDK, but the spirit of the solution is there and it's working very well so far.

I should have trusted my instincts and not stayed with a task-based solution once I found out about the timing requirements of Modbus RTU, The OS tick time is 5ms, which is clearly too slow to do anything timing-critical.  All you have to do to implement Mark's (uTasker) Modbus solution is to add an RX callback and 2 PITs.  Once I find the time to do a full writeup of the KSDK-based Modbus RTU implementation, I'll post it here for the community.

0 Kudos
2,052 Views
Cdn_aye
Senior Contributor I

Hi Dave

Thank you for your reply. I will also look into this as well, we have stumbled into another timing trap with the current serial driver. Our method is not catching late portions of the message. We are talking to an Android device and they do not send the frame in a rigorous manner. Now running at a faster baud rate causes all manner of problems.

If you are still able to post a solution I think it would be a help to a lot of people. Thanks for continuing the thread

Robert Lewis

0 Kudos
2,052 Views
dave408
Senior Contributor II

Hi Cdn_aye​, here's my writeup for my solution.  I may have missed a couple of small details, but the majority of the solution and the explanation for the approach is in there.

How to: KSDK-based Modbus RTU Packet Assembler in MQX

0 Kudos
2,052 Views
Cdn_aye
Senior Contributor I

Hi Dave

Thank you for the outline, that looks like a good solution. I have used PE (Processor Expert) with Coldfire, but under Kinetis with MQX there was no working bsp and psp configurations so we had to work under MQX only. That means the install of the vectors, setup of the NVIC, and on and on and on... is all manually coded.

A labyrinth of potential problems if anything is not exactly correct. Now it looks like the SDK does a similiar setup to PE which makes it all much easier. The reason I mention this is, for us to implement this solution we will have to manually convert the SDK device setup or switch to the SDK or do a manual setup. Probably we will do the manual setup because I am not sure of the potential problems splicing part of the SDK system with our manual setup. We have the i2c driver from the forum (because the mqx driver missed char's) so I am guessing that this would have to be integrated as well. A bit off topic, but what I am saying is I will update the thread with Mark and your design based on a non SDK version in the future.

Thanks again for the update and design guideline. I think this will be very helpful.

Robert

0 Kudos
2,052 Views
dave408
Senior Contributor II

Thanks, Robert!  I haven't had any problems going entirely with KDSK and MQX and PEX yet, with the exception of QD which I configured manually.  I'm crossing my fingers.  I did run into problems with the UART driver and hence the writeup.  I2C has worked well for me so far, and so has SPI.  Good luck with your project!  I am looking forward to seeing your followup.

0 Kudos
2,052 Views
Cdn_aye
Senior Contributor I

Hi Dave

I have been checking our uart stream with a logic analyzer and we are having problems at 115200. The analyzer shows a valid stream going and coming but the Rx buffer in the uart driver has missed char's and sometimes even char's from the previous stream.

I don't think I am going to be able to avoid the changes.

Can you suggest how to implement a call back from the uart and to turn off the FIFO please? Do you load and ISR vector for the uart Rx IRQ pointing to an ISR that copies one char? Or was there another way to do this using the MQX serial driver. I would rather not have to write the entire driver interface, or even if you have a post to point to that would be appreciated.

I can clearly see on the logic analyzer what Mark meant by MQX getting in the way. Also the Android device we are talking to, is clearly being taxed to keep up. The inter char spacing is all over the map. The stream is correct but unpredictable at a high baud rate.

Robert

0 Kudos
2,051 Views
dave408
Senior Contributor II

Robert, I think this is consistent with what I have experienced recently.  Please see my previous post:

UART_DRV_ReadDataBlocking seems to allow FIFO data to be lost

Are you calling UART_DRV_ReadDataBlocking for 1 byte at a time?  What I had seen before was that there would be data in the FIFO, but calling this method would never return any data.  In other words, it was as if the method thought the FIFO was empty.  Down in the code, there's a pointer that gets indexed so it knows where to put the next incoming byte, and what I would see is that if I then sent more data, I could see the previous "invisible" byte in memory.

I ran my modbus tests again, but instead the next time around read multiple bytes at a time, and the problem went away.  But of course my modbus implementation was flawed, so it didn't really matter.  Smiley Happy

Here is my code to disable the FIFO using the HAL functions:

void DisableFifo( uint32_t instance)

{

    UART_Type *base = g_uartBase[instance];

    UART_HAL_DisableReceiver( base);

    UART_HAL_FlushRxFifo( base);

    UART_PFIFO_REG( base) &= ~UART_PFIFO_RXFE_MASK;

    UART_HAL_FlushRxFifo( base);

    UART_HAL_EnableReceiver( base);

}

To be clear, I rely on using Processor Expert and the KSDK to implement everything.  I haven't done anything manual in my projects except slight massaging around components like the FlexTimer.  I have to support 3 different Kinetis processors that interface to a very similar pool of peripherals, so I leverage PE and the C++ wrappers I wrote around each device.  Therefore, to implement the RX callback as you have inquired about, I just have to check the box to use the RX callback function, generate the code, and then fill in the blanks.

Under the hood, PE generates code like this:

/*! Device instance number */

#define FSL_RS485_COMP UART1_IDX

/*! Driver state structure without DMA */

extern uart_state_t rs485_comp_State;

/*! @brief UART configuration declaration */

extern const uart_user_config_t rs485_comp_InitConfig0;

/*! Interrupt service routine (ISR) prototype */   

void rs485_comp_IRQHandler(void);   

/*! rs485_comp configuration structure */

const uart_user_config_t rs485_comp_InitConfig0 = {

  .baudRate = 115200U,

  .parityMode = kUartParityDisabled,

  .stopBitCount = kUartOneStopBit,

  .bitCountPerChar = kUart8BitsPerChar,

};

OSA_InstallIntHandler(UART1_RX_TX_IRQn, rs485_comp_IRQHandler);

UART_DRV_Init(FSL_RS485_COMP,&rs485_comp_State,&rs485_comp_InitConfig0);

UART_DRV_InstallRxCallback(FSL_RS485_COMP, rs485_comp_RxCallback, rx_callback_buff, NULL, true);

void rs485_comp_RxCallback(uint32_t instance, void * uartState)

{

  /* Write your code here ... */

    UtaskerModbusImpl( instance, uartState);

}

Because the FIFO was disabled, the callback function gets called for every byte.  So far, I see very reliable communication at 115.2kbps.  I tried to bump it up to 230.4kbps, but the reliability dropped off significantly.  38400baud was also fine.

Hopefully this helps!

0 Kudos
2,052 Views
dave408
Senior Contributor II

Sure, I will share what I can of my design when I get it working.  It would be good to have more eyes on it anyway and get feedback from the community.  I could post the RX callback and the task listener code.  Hopefully I will find time to work on this today.  I am very anxious to see how that approach works out.

0 Kudos
2,052 Views
DavidS
NXP Employee
NXP Employee


Hi Matthew,

Thank for your great insight and guidance.

Much appreciated.

Hi Robert,

Will this help guide you?

Regards,

David

0 Kudos
2,052 Views
DavidS
NXP Employee
NXP Employee

Hi Robert,

Did you come up with a solution?

Regards,

David

0 Kudos
2,052 Views
Cdn_aye
Senior Contributor I

Hi David

Thank you for checking. No we did not get a solution beyond the ones we had already tried.

I tried a number of different ways, but in the end just checked for a first char, and if found did a blocking read and implemented a timeout check for the last char as a termination mechanism. Also for the first char check, I check the fstatus in a loop with a timeout calculation. if No char then exit.

It works, after a fashion, but this is a lame way to implement a real time device driver imo. It is non deterministic having to guess at the time factors and does not account for changes in the sending clients messages.

The correct way would be to have a read of the buffer by an MQX driver where MQX returns a complete buffer based on a break in the Tx or a termination char or a number of characters. This is how I have seen this done in other OS's.

Regards

Robert Lewis

0 Kudos
2,052 Views
DavidS
NXP Employee
NXP Employee

Hi Robert,

For your implementation, what is maximum bytes that would be in a UART packet you receive?

I'd think using interrupt UART driver and making a few modifications/additions might do the trick.

The interrupt UART driver grabs bytes and sticks them into a buffer.  The UART ISR could filter the received byte for your termination character and then set an event to your task waiting for a whole packet/message.

I might be over simplifying as not taking into consideration a corrupted packet.

Regards,

David

0 Kudos
2,052 Views
Cdn_aye
Senior Contributor I

Hi David

The maximum bytes is 50.

I think you are right, can you recommend a method or just use the example code?

Regards

Robert Lewis

0 Kudos