topic Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1) in MQX Software Solutions

UART FIFO appears empty even though it is not (K64 & MQX 4.1)

tmeyer — Thu, 12 Feb 2015 18:47:47 GMT

Hi,

I have a somewhat rare issue with my UART RX FIFO. The K64 UART is connected to a half duplex RS485. I haven't pinned down exactly how to cause the issue yet, but it seems to occur more frequently when I send lots of contiguous serial data. If I had to guess, it would be caused by a bit r framing error on the UART bus.

Setup:

- K64 & MQX 4.1

- UART1 interrupt mode (8N1 @ 115200)

- RTS pin controls RS485 flow control (flag: IO_SERIAL_HW_485_FLOW_CONTROL)

The symptom:

Once the issue occurs, it never seems to go away until I reset the K64. As an example: the master will send a packet of say 10 bytes to the k64 slaves on the rs485 bus. Then the k64 will read one byte at a time from the UART driver. The following function is used to determine if data is ready to be read:

_io_ioctl(port, IO_IOCTL_CHAR_AVAIL, &ready);

When the issue occurs, I can read say 6 bytes from the UART buffer, and then ready becomes FALSE, even though there is still 4 more bytes that should be ready. This function will continue so indicate no data is ready until an additional byte is sent by the master, in which case I will be able to ready 1 more byte from the UART driver.

It seems as if the data is stuck in the FIFO, but is not accessible. I have put a sniffer on the bus to confirm that the data wasn't being cached in the master side UART and can see all the data on the bus. I've also had multiple k64's on the same bus all receiving the same data and some of the K64s will receive all the data properly and others will get into this erroneous state.

Any help would be appreciated.

Thanks,

Tim

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

matthewkendall — Fri, 13 Feb 2015 19:30:44 GMT

This issue maybe?

Re: Bug Report: In MQX 4.1 change of size of bool can cause stack corruption

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

soledad — Fri, 13 Feb 2015 22:51:08 GMT

Hello Tim,

Could you please confirm if the Matthew's suggestion helps???

Have a great day,
Sol

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

tmeyer — Tue, 17 Feb 2015 16:20:43 GMT

Hi,

Thanks for the response. This doesn't appear to be related to the IOCTL_CHAR_AVAIL. I can reproduce the same issue using a simple polled UART0 connected to the shell. After sending arbitrary data to the shell, it will then start echoing characters that were sent 1-3 keystrokes ago. (I double checked this with a scope to ensure I was sending the correct chars).

Upon debugging it in this state, it is clear that there is nothing to be read since there are 0 chars in the uart driver CHARQ. Also, the UART peripheral registers indicate that the RX FIFO is empty:

Serial Communication Interface (UART0)
	UART0_BDH	0x00	0x4006a000
	UART0_BDL	0x36	0x4006a001
	UART0_C1	0x00	0x4006a002
	UART0_C2	0x0c	0x4006a003
	UART0_S1	0xc0	0x4006a004
	UART0_S2	0x40	0x4006a005
	UART0_C3	0x00	0x4006a006
	UART0_D	0x62	0x4006a007
	UART0_MA1	0x00	0x4006a008
	UART0_MA2	0x00	0x4006a009
	UART0_C4	0x08	0x4006a00a
	UART0_C5	0x00	0x4006a00b
	UART0_ED	0x00	0x4006a00c
	UART0_MODEM	0x00	0x4006a00d
	UART0_IR	0x00	0x4006a00e
	UART0_PFIFO	0xaa	0x4006a010
	UART0_CFIFO	0x00	0x4006a011
	UART0_SFIFO	0xc5	0x4006a012
	UART0_TWFIFO	0x07	0x4006a013
	UART0_TCFIFO	0x00	0x4006a014
	UART0_RWFIFO	0x01	0x4006a015
	UART0_RCFIFO	0x00	0x4006a016
	UART0_C7816	0x00	0x4006a018
	UART0_IE7816	0x00	0x4006a019
	UART0_IS7816	0x00	0x4006a01a
	UART0_WP7816T0	0x0a	0x4006a01b
	UART0_WP7816T1	0x0a	0x4006a01b
	UART0_WN7816	0x00	0x4006a01c
	UART0_WF7816	0x01	0x4006a01d
	UART0_ET7816	0x00	0x4006a01e
	UART0_TL7816	0x00	0x4006a01f

What I also find interesting is that if I physically unplug the TTL level UART cable from the processor and plug it back in - the characters will start coming in correctly. It is as if, the

Thanks,

Tim

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

matthewkendall — Tue, 17 Feb 2015 17:59:53 GMT

there are 0 chars in the uart driver CHARQ

The CHARQ struct (as defined by psp/PSP_Generic/include/charq.h) includes not only head and tail indexes (.HEAD and .TAIL) but also a count of the number of chars in the queue (.CURRENT_SIZE). This is clearly redundant and is likely included as an optimisation.

In other contexts I have had trouble with CHARQ where the count became inconsistent with the head and tail indexes. This should not be possible if the CHARQ is always accessed via the provided macros. But there is no flush macro for example, so it is possible that someone decided to directly access the struct. It might be worth investigating if something in the serial driver can cause a problem, for example by directly setting CURRENT_SIZE to zero to flush the queue without making corresponding change to HEAD and TAIL. In your error state it might be worth checking to see if the CHARQ is self-consistent.

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

tmeyer — Tue, 17 Feb 2015 22:35:38 GMT

Thanks Matthew for the replay.

I looked into the CHARQ and it appears that it is behaving as expected. I can step through the code and see the old character(and not the one most recently typed).

ex:

Sequence of events:

Type "12" into the shell
The final character enqueue via _CHARQ_ENQUEUE in the_io_serial_polled_read() function is the '1' character. (Not the '2' as expected).
_kuart_polled_status() indicate that the RX FIFO is empty: UART_SFIFO_RXEMPT_MASK is set, and RXCOUNT in the RCFIFO register is also 0.
"1" is echoed to the shell console
Type "3" into shell
The '2' character is now enqueued via _CHARQ_ENQUEUE in the_io_serial_polled_read() function.
_kuart_polled_status() indicate that the RX FIFO is empty: UART_SFIFO_RXEMPT_MASK is set
"2" is echoed to the shell console

This appears to me that the serial peripheral is somehow caching these characters in it's FIFO or elsewhere. It seems incorrect to me that the FIFO indicates it is empty, however a byte that was sent to the UART prior shows up after sending a new byte. Is there a chance there is an errata relating to the hardware peripheral?

Thanks,

Tim

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

johannesschock — Wed, 18 Feb 2015 14:46:54 GMT

Hello, I had a similar Problem with other Kinetis (K70/60/10) and after some heavy debugging I think there's a hardware bug in the Kinetis Uart module.The Kinetis Uart itself has a internal FIFO (some chips more deep, some chips less deep). After a overrun of this FIFO occurs (a character isn't fetched by IRQ when it is already full and a new one arrives) this FIFO stops functioning: Write and read pointers of that Hardware FIFO will have an offset afterwards, which means you will get your following characters with a delay. The Interrupt for character n+1 lets you read character n, the Interrupt for character n+2 lets you read character n+1 and so on.

After messing around with that problem I found the following solution (more or less empirically):

Locate all occurences of

   if(stat & UART_S1_OR_MASK) {
      ++sci_info_ptr->RX_OVERRUNS;
   }

and replace it with

  if (stat & UART_S1_OR_MASK) {
    sci_ptr->CFIFO |= UART_CFIFO_RXFLUSH_MASK;
    sci_ptr->PFIFO &= ~UART_PFIFO_RXFE_MASK;
    c = sci_ptr->D;
    sci_ptr->PFIFO |= UART_PFIFO_RXFE_MASK;
    ++sci_info_ptr->RX_OVERRUNS;
  }

The locations are in

serl_pol_kuart.c and in serl_int_kuart.c

Hope it helps.

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

tmeyer — Wed, 18 Feb 2015 15:29:24 GMT

That fixed it.

Thanks.

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

yb — Tue, 05 May 2015 09:46:47 GMT

Fixed for me too, thanks a lot !!!

It was a long long problem for me...

Yvan

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

tmeyer — Fri, 17 Jul 2015 16:45:39 GMT

Does anyone know if this was ever documented as an Errata for the kinetis parts? I have encountered this same issue again.

I would like to determine if the newer rev's of certain processors have fixed this.

Tim

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

yb — Tue, 18 Aug 2015 07:23:23 GMT

Be careful !

Under MQX 4.2, the bug is not fixed :-(

Yvan

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

cgarcia1 — Thu, 20 Aug 2015 15:15:03 GMT

I have detected the same problem in KDS v3.0.0 with SDK v1.2.0 using MQX.

The UART driver is totally different, but the same solution applies :-)

I find very interesting that Freescale does not confirm if this is a hardware-related issue or not. And I find even more interesting that Freescale does not solve the problem, as this existed for a very very long time...

Thank you, Johannes, for your answer :-)

EDIT: oh, I forgot to say that I found the problem using a K64. The same problem happened in a K10 using CodeWarrior and MQX 4.1

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

dave408 — Mon, 09 Nov 2015 05:35:26 GMT

Hi johannesschock, is there any chance you have used the KSDK and have seen something similar? I'm using MQX for KSDK 1.2 and have a very similar sort of behavior where over time, something the RX interrupt causes me to miss a byte, but rather than ending up with one bad packet, every packet is bad. If I look at my buffer, it seems like data from the previous partially received packet is read as part of the next packet.

For example, let's say I normally get something like this:

0B 04 08 00 00 00 00 00 04 FA 3F 46 A4

But at the failure point, the packet could look like this:

15 0B 04 08 00 00 00 00 00 04 FA 3F 46

so the 15 comes from a previous packet's CRC, and therefore the last byte of the CRC for the present packet gets cut off. And of course, this just completely messes up everything.

I know I'm asking for a miracle, but I am hoping that you or someone else has seen this with the KSDK as well. I'm going to take the information you have provided to tmeyer and see if it helps me find anything that can be fixed in the KSDK, if applicable.

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

cgarcia1 — Mon, 09 Nov 2015 06:30:16 GMT

Hi dave408,

I was forced to switch to KDS and KSDK, and as I could see, the same issue happens in KSDK 1.2 and the newest KSDK 1.3.

To solve this problem you will have to apply Johannes' solution. In KSDK the UART driver is different, but the same concept applies. When you detect a RX overrun, you will have to perform the same steps shown in Johannes' solution. This worked for me, almost magically :-)

I'm not allowed to share my code, but it's not difficult to do.

I don't understand how Freescale have not pronounced on this issue yet... Everybody uses a UART for almost all projects, and it has to be robust...

EDIT: Please, note that when you fetch a character from UART, status flags are cleared. You will have to check for kUartRxOverrun before UART_HAL_Getchar is called.

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

dave408 — Mon, 09 Nov 2015 12:33:17 GMT

Thanks, Carlos, I hadn't scrolled down all the way and read your response from August. I appreciate your replying to my needless post. :smileyhappy: I'll work on investigating a fix today!

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

dave408 — Mon, 09 Nov 2015 14:38:09 GMT

cgarcia thanks for the edit. At least now I'm seeing some problems I hadn't caught before. I can see that I am getting RX overruns and framing errors as well. Now I just have to solve the issue. I took johannesschock's solution and applied it to my application like this:

Modification to fsl_uart_driver.c in UART_DRV_IRQHandler:

bool detected_overrun = false;
if (UART_HAL_GetStatusFlag(base, kUartRxOverrun)) {
    // Current test doesn't have this commented out, but it really should be because this function reads the D register.
    //UART_HAL_ClearStatusFlag(base, kUartRxOverrun); // Clear the flag, OR the rxDataRegFull will not be set any more */
    base->CFIFO |= UART_CFIFO_RXFLUSH_MASK;
    base->PFIFO &= ~UART_PFIFO_RXFE_MASK;
    uartState->rxBuff[0] = base->D;
    base->PFIFO |= UART_PFIFO_RXFE_MASK;
    detected_overrun = true;
}
if (UART_HAL_GetStatusFlag(base, kUartFrameErr)) {
    // TODO: I have verified framing errors, need to address after RX overrun
    UART_HAL_ClearStatusFlag(base, kUartFrameErr);
}
if (UART_HAL_GetStatusFlag(base, kUartNoiseDetect)) {
    UART_HAL_ClearStatusFlag(base, kUartNoiseDetect);
}
/* Get data and put into receive buffer */
if( !detected_overrun) {
    UART_HAL_Getchar(base, uartState->rxBuff);
}
/* Invoke callback if there is one */
if (uartState->rxCallback != NULL)
{
    uartState->rxCallback(instance, uartState);
}
else...

If I understand correctly, the main idea here is to correctly detect the overrun condition, save the data in the RX buffer, then flush it out so that incoming bytes no longer get offset.

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

dave408 — Mon, 09 Nov 2015 17:29:19 GMT

tmeyer did you ever try your application at lower baud rates? While I agree that hardware could be at the root of this UART problem, I don't understand why I get an RX overrun in the first place. Although I have the FIFO disabled, I would still expect the K64F to be able to keep up with pulling a byte of data from the UART at 115200. However, I've lowered my speed to 38400 and still get overrun errors. The funny thing is that I have two other identical board configurations talking to identical slave devices at 115200, and neither of them exhibits these RX overrun problems. So I'm just wondering if you've tried your test at lower speeds, as I have to wonder if there's also something else at play here.

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

tmeyer — Wed, 11 Nov 2015 21:02:35 GMT

Hev dave408,

It has been a while since I dealt with this issue, but I think I was able to see it at both 9600 and 115200. What I don’t remember is what, other than handling a UART interrupt, the code busy doing. At 100MHz, the k64 should be able to keep up to the 115200 baud. This works out to an incoming byte every 80us (roughly).. so as long as you have your system setup to allow the UART ISR to run (or polling loop) at least this frequent you should be okay!??

I am not confident that this issue is caused by the overrun, perhaps when the issue occurs a overrun is signalled?! The reason I think this, is that I have in the past reproduced this issue on a 115200 command line shell by my slow typing. I suppose I cannot rule out that I may have copy/pasted a command into the shell.

Sorry I couldn’t be more help.

Tim

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

AlexJLennon — Sat, 12 Dec 2015 23:37:45 GMT

Many thanks dave408 and johannesschock for taking the time to post on this.

We made the mistake of assuming the FIFOs on some of the K22 parts were 16C550 style and belatedly found they are only 2 bytes deep.

I've been fighting weird NULLs coming out of the fsl_ drivers for some time now and your suggestions have enabled me to fix

The only comment I'd make is that the (really helpful!) example snippet probably doesn't want to be calling the callback on an error path.

Re: UART FIFO appears empty even though it is not (K64 & MQX 4.1)

dave408 — Mon, 14 Dec 2015 14:01:41 GMT

Yes, I should probably have put the callback check into the if statement right before it... thanks for catching that. :smileyhappy: