InterNiche Lite UDP performance

[wyliek]

Hi there

I am running an application on the M52233DEMO board. It Checks pin states in the main loop. It also listens on a UDP port for packets from A PC and responds with data. At the moment I am polling the device over a small LAN. It responds rather quickly to the UDP polling (takes about .5 milliseconds). However, from 4 to 8 times a minute it seems miss a packet. (see attached Ethereal capture -> packet # 5061. My PC is 192.1.3.204 and the DEMO board is 192.1.2.42).

I know that the packets are arriving at the DEMO board as I am using a hub and ethereal to monitor this.

I assume that they are being dropped somewhere in the DEMO board's hardware or in the software. I have checked the fec_isr() function and it seems that the packets are not arriving here and are therefore not passed to the stack. I therefore assume that it is a FEC driver problem or just a limitation on the chip's performance. I have tried to modify some FEC settings like NUMBIGBUFS, NUMLILBUFS, NUM_RXBDS, NUM_TXBDS as shown below:

/* define number and sizes of free buffers */
#define NUMBIGBUFS   8 //10 //20
#define NUMLILBUFS   6 //10 //20

/* FEC buffer descriptors */
#define NUM_RXBDS    2  //4  //8
#define NUM_TXBDS    2  //4  //8

However I find that I cannot increase NUMBIGBUFS and NUMLILBUFS above 8 each without the board freezing on start-up. Does anyone know why?

Also if I increase NUM_RXBDS and NUM_TXBDS above 8 each the board freezes on start up.

Increasing them by small amounts however, does not seem to make a difference to the frequency of lost packets. Are there any other setting I could try modify to get better performance? Any ideas one how I can increase the values of NUMBIGBUFS, NUMLILBUFS, NUM_RXBDS and NUM_TXBDS further? Would this even help?

Can anyone explain to me how this FEC works simply. I have read the RM but am still a bit lost.

I'll take any suggestions

Thanks

[cHeller]

I am also having trouble with dropped packets, and they appear to be non octet; errors also.

Perhaps this code in ifec.c is wrong? In the ISR fec_isr, the driver checks the FEC_RxBD register for receive errors:

/* while next receive buffer has packet do receive processing */
while ((bdp->bd_cstatus & MCF_FEC_RxBD_E) == 0)
{
/* if receive had an error, just count the error and leave the
* buffer in the BD ring
*/
if (bdp->bd_cstatus & (MCF_FEC_RxBD_ERRS | 0x0400))
{
if (bdp->bd_cstatus & MCF_FEC_RxBD_LG)
fecstats.fec_lgerrs++;
if (bdp->bd_cstatus & MCF_FEC_RxBD_NO)
fecstats.fec_noerrs++;
if (bdp->bd_cstatus & MCF_FEC_RxBD_CR)
fecstats.fec_crerrs++;
if (bdp->bd_cstatus & MCF_FEC_RxBD_OV)
fecstats.fec_overrs++;
if (bdp->bd_cstatus & MCF_FEC_RxBD_TR)
fecstats.fec_trerrs++;
}
else /* we received a good BD */
{

But hang on here. According to the MCF52235 Ref Manual, bit 0x0400
is undefined, and errors _LG, _NO, _CR, and _OV need to be qualified
by the _L bit. (Page 18-47 of the RM). So I changed the code to:


/* while next receive buffer has packet do receive processing */
while ((bdp->bd_cstatus & MCF_FEC_RxBD_E) == 0)
{
/* if receive had an error, just count the error and leave the
* buffer in the BD ring
*/

// Modified to reflect the Freescale MCF52235 Reference Manual
// page 18-48. _LG, NO, CR, and OV are to be qualified by _L.

if ((bdp->bd_cstatus & MCF_FEC_RxBD_TR) ||
((bdp->bd_cstatus & MCF_FEC_RxBD_L) AND (bdp->bd_cstatus
&MCF_FEC_RxBD_ERRS))) {
if (bdp->bd_cstatus & MCF_FEC_RxBD_LG)
fecstats.fec_lgerrs++;
if (bdp->bd_cstatus & MCF_FEC_RxBD_NO)
fecstats.fec_noerrs++;
if (bdp->bd_cstatus & MCF_FEC_RxBD_CR)
fecstats.fec_crerrs++;
if (bdp->bd_cstatus & MCF_FEC_RxBD_OV)
fecstats.fec_overrs++;
if (bdp->bd_cstatus & MCF_FEC_RxBD_TR)
fecstats.fec_trerrs++;
}
else /* we received a good BD */
{


This did not help my problem a great deal, but I thought I'd pass it along, as it may well be a bug.

BTW, my dropped packet problem improved a great deal (but did not completely go away) when I changed from a no-name hub to a name brand switch. I was suspecting that when the client sent two packets to my TCP server within 150uS of each other (on 10baseT) that the second packet was garbled (octet error) by the M52233DEMO board. (Another server based upon a PowerPC did NOT garble the packets. Ever.)

Ethereal has other complaints about the TCP traffic that NicheLite is sending. Has anyone seen these issues:
[TCP Dup ACK]
[TCP Out-Of-Order]
[TCP Illegal Segment]

Thanks for any help.

Chuck

[mjbcswitzerland]

Hi

If you haven't done it yet, stop the debugger after a few frames have been lost and check the MIBRAM registers in the registers view. These are displayed very nicelly in CW6.3 and such things as overruns will be seen as counted events.

In a normal network environment it is extremely unusual to loose a frame unless the input is being overrun, which can be decreased/avoided using adequate buffers. It also only happens when frames are received very quickly after each other. You didn't say at what speed the UDP polling frames are arriving.

Reference: I have been working on a project where 512 byte UDP frames are received and sent every 50ms. In parallel 512 bytes are sent and received at 250k over one UART, also at 50ms repetition rate. At the same time a web server and or FTP server transfers are active (of a random nature of course) which are sometimes writing data to internal FLASH. Dropped frames have never been experience during tests of many hours so the device itself has no inherent problems under such load conditions.

I have published some UDP response times for the uTasker OS and TCP/IP package:
http://www.utasker.com/docs/uTasker/uTaskerBenchmarks.PDF

It is dependent on the UDP frame length and whether UDP Check sum is active or not. In addition there is quite a useful gain to be made when using DMA support for any data copies in the process.

Regards

[wyliek]

Hi there

Mark,

The packets are being sent almost every 22ms. The DEMO board responds to each in less than 1ms. I cant see how it can respond so quickly most of the time but on a rare occasion not pick up the packet at all. Do you think 22ms is a bit to quick?

How do you view the MIBRAM register? When I look at View->Registers all I see is:
CACR
ACR0
ACR1
CBR
MBAR
RAMBAR

and some general purpose registers.

Richly,
You are correct that would be the place to set len. Are we all in agreement that this could be a bug? If so, should I report it?

Thanks for your input guys. If you think of anything else I could try, please let me know. I'm still battling away here to either fix or just diagnose the problem.

Thanks
Kyle

Message Edited by wyliek on 2007-03-0507:03 AM

[mjbcswitzerland]

Hi

22ms is not fast - when serving web pages using frames in a fast local network it is typical to receive three or four TCP frames within less that 100us. In this case it is important to have adequate buffers because the software has no chance of keeping up with the rate with only one or two buffers (5 or 6 usually are adequate for such work). In the case of receiving periodic frames a rate of 2ms should be possible since your response time is in the order of 1ms - unless there is something else in the system which sometimes blocks frame handling for longer periods of time...

To find the registers do VIEW | REGISTERS to open the register window.
Then expand the "FAST ETHERNET CONTROLLERS REGISTERS"
After register EMRBR there are 54 x 32 bit counters (RMON_T_DROP to IEEE_RX_OCTETS_OK). You will have to check their exact meaning in the data sheet but if you break before and after a loss of a frame you may be able to simply identifier a counter which has counted it and then the reason should be clear from the counter description.

I have attached a screen shot for clarity.

Regards

[wyliek]

Hi Mark

Thanks for the reply

I have tried viewing the registers (view->registers) however I do not get the options you have. Please have a look at the attached screenshot...

Is there some setting to enable these registers to be viewed (I am using the M52233DEM with CW V6.3)

Thanks
Kyle

Message Edited by wyliek on 2007-03-0902:09 PM

[mjbcswitzerland]

Kyle

You seem to have only basic registers and no peripheral registers. Have you setup your project so that it knows that you are using the M5223X?

If you don't solve it, it may be an idea to post in the Codewarrior development tools forum. I use the EVB with CW6.3 and assume that the software is the same - if not maybe you could download an update of the SW version?
As a reference, mine displays IDE version 5.7.0 Build 2015.

In debugger settings I have M5223X as target processor.
Target initialisationm file: {Project}cfg\M52235EVB_PnE.cfg
Memory configuration file: {Project}cfg\M52235EVB.mem

Good luck

Regards

Mark

[wyliek]

Thanks Mark

It seems that the .mem file and the target where not set correctly. I can now see all the registers. Will do further debugging using these, however at first look they give me no clue as to what is causing the UDP packets to be dropped. For example if I poll the device until two packets have been dropped and then look at the FEC counters, none of them have incremented by two.

What is quite strange is that when I connect my PC directly to the DEMO board with a crossover cable, no packets are dropped at all. I am pretty sure that the packets are getting to the board as I have done the "Ethereal + HUB" test. SO this leads me to believe that either, the ColdFire FEC or PHY cannot handle the traffic on my network as well as the constant UDP polling. This is not due to buffer overflow, because the FEC counters do not show any buffer overflows. So I'm still a bit stuck on this...

I will do further investigations and keep everyone up to date

Thanks again for your help

[mjbcswitzerland]

Kyle

It may be that the board is answering but the message is being lost due to a collision on the network. Collisions are normal occurences on a network and no indication of a problem, but if the device is configured to work in full duplex mode it will disable its collision detection and there will be no indication of losses due to collisions.

If enabled in half-duplex mode, its collision detector is enabled and it automatically retries when collsions are detected and will not lose frames (except when there is a network problem and the excessive collision counter will be incremented as a result).

Be careful when the device is set up for auto-negotiation because the mode (whether full- or half-duplex) is still valid.

TCR = FDEN; // this sets full duplex in the EMAC (collison detection disabled)
TCR = 0; // this sets half duplex in the EMAC (collision detection enabled)

Full duplex is only advisable for direct connections (eg. cable between device and PC).

Collision rate depends on network activity and also transmission speed. 10M speed results in more collisions because the chance of two transmitters sending at the same time increases since they drive for longer periods of time.

Ensure that collsions are correctly detected and then tx loss is generally not possible. It may explain your observations. If not, please detail your test set up and I can configure the uTasker demo software to do the same - I can send the downloadable and if it works correctly you can then compare all register set ups in the debugger to find a difference.

Regards

[wyliek]

Mark

I know that the board is not answering. This is how:

I set my PC to poll the M52233DEMO board continuously. The timeout for no response was set to 10s. What happens is that the PC polls the MCF52233 and I see it responding until the polling stops because the PC is waiting for a response. It waits like this for 10 seconds. During this time the colfire is also outputting all received and transmitted udp packets on the serial port. Looking at the serial port output I see no received packet and that is why the MCF is not responding.

I also debugged by outputting a character when the input_ippkt(PACKET pkt, int length) function is called. This function is called by the FEC ISR when it wants to enque any received packets. It seems that when I get a timeout there is no such character displayed. This tells me that my problem is not due to transmitted packets being lost but is a receive problem. It is almost as if the FEC is not seeing these packets at all. Could it be some sort of PHY error? Maybe to do with connection speed and or auto negotiation?

The UDP data that is sent in each packet to the DEMO board is rather simple:
p01000000010D250117BF
They consist of a beginning character, a message counter (2 bytes), some other status information, a command identification byte and a 2 byte CRC. The entire message is just 22 bytes long. The Coldfire responds with a message of equal size including its status, a message counter and a CRC.

The DEMO board normally responds to these packets very quickly (sub 1 millisecond) however I do get the occasional packet which just doesnt make it. I am beginning to think more and more that it is the hub I am using. I will switch it for another and let you know if any changes ensue.

Thanks again for everyones help

Kyle

Ps Chuck... I have replaced my error checking code in the fec ISR with your code. It does not fix my problem or even cause any noticeable difference. Is your DEMO board connected to a LAN? If so check to see if any devices are sending malformed packets (use ethereal).

I have also been using TCP/IP and have not come across any of the problems you speak of. If you send me source code I'd be happy to test it on my board and see if I can repeat the errors

[cHeller]

-I am beginning to think more and more that it is the hub I am using. I will switch it for another and let you know if any changes ensue.

Moving from a hub to a switch did help me reduce nonoctet errors.

-Ps Chuck... I have replaced my error checking code in the fec ISR with your code. It does not fix my problem or even cause any noticeable difference. Is your DEMO board connected to a LAN? If so check to see if any devices are sending malformed packets (use ethereal).

Yes, perhaps my hub was, but I have a call into InterNiche to get the real answer on the code in ifec.c.

-I have also been using TCP/IP and have not come across any of the problems you speak of. If you send me source code I'd be happy to test it on my board and see if I can repeat the errors

You are so kind! The client side is too hard to duplicate, but here is the ethereal trace. It's better understood when you decode unknown RPC calls via Edit/Preferences/Protocols/RPC/Dissect unknown RPC program numbers.

This trace shows two (of many) errors that I get. The first is pair of
duplicate ACKs in lines 7 and 13. They are duplicates of 4 and 10. I can make these go away by commenting out this code in TCPOUT.C:

if (win > 0)
{
int adv = (int)win - (int)(tp->rcv_adv - tp->rcv_nxt);

if (so->sendq.sb_cc == 0 && adv >= (int)(tp->t_maxseg * 2)) {
goto send;
}
if ((100 * (unsigned)adv / mt_defrxwin) >= 35) {
goto send;
}
}

This code is different than the code in TCP/IP Illustrated, Wright & Stevens, page 860. I will try and reconcile the differences, and see if I can "fix" this code, if it is indeed broken.

But then also notice at the end, there are two TCP out-of-order errors on lines 29 and 32. I traced these down to calling m_close(so) in my TCP socket callback routine. It appears that when the client (PC) send s the FIN, the coldfire server will ACK the FIN, but first it calls the callback routine with code = M_CLOSED. My understanding is that for case M_CLOSED, I am to call m_close, but that calls tcp_output and sends the ACK. So one ACK gets sent as a direct result of receiving the packet in tcp_rcv, and another gets sent when tcp_rcv calls my socket callback which calls m_close.

In any case, I'm getting a lot of ACK packets!

I tried turning on DO_DELAY_ACKS and actually got a compiler error:
'qhead' is not a member of class 'struct queue'. InterNiche confirmed that this is indeed a bug (should be q_head). In any case, delayed acks didn't seem to fix anything.

Any help would be greatly appreciated. Perhaps m_close should not be calling tcp_output when a server is being closed by the client?

Again, this thread has been a big help, and I appreciate it.

Chuck Heller

[wyliek]

Hi Chuck

You seem to have forgotten to attach the ethereal trace. Attach it and I'll have a look.

I remember at one time I was having a problem with TCP. The DEMO board was advertising a window size of 0 bytes. Thus the client (my PC was waiting a long time between sending data). I never really found out why that was happening. I fixed it by making a clean project and starting from the beginning. It must have been caused by some other task running in my application (I find that NicheTask can be a bit sensitive at times). Or perhaps some of the playing around I had been doing with the Niche Lite code. Anyways, My TCP server works fine now.

Could you detail your test setup? I assume the your ColdFire is acting as a server. In the callback function, if the code is M_CLOSED you should close the passed M_SOCK so, not the listening socket. I ran into some trouble on this when I didn't fully understand the callback function.

Cheers

[cHeller]

Sorry, here is the attachment, and the code for the callback function.

The Coldfire is the server, the PC is the client. I have a very small RPC implementation on top of TCP, as you'll see in the attachment. I believe that I am calling m_close with the real socket, not the half open listening socket, as I understand this.

Thanks again,
Chuck H

[wyliek]

Its been confirmed. The UDP packets I was losing were caused by the Hub that I was using. It seemed that the packets were getting dropped by this hub. I have the DEMO board now connected to a switch and have not lost 1 packet in an entire hour of polling.

Thanks to everyone for all your help

Cheers

[mccPaul]

Hi wyliek,

This may help to explain how the FEC driver works:

NUM_RXBDS and NUM_TXBDS are the number of buffer descriptors allocated for the FEC. These are structures that are used to tell the FEC how the data that is transmitted or received is organised. In some drivers, TX frames will be split into separate buffers containing Ethernet header, protocol header (eg IP + UDP) and payload. In this case three buffers descriptors would be set up to point to the three bits of theframe and then the FEC will be told that there are buffer descriptors ready to consume. After this the FEC takes over and transmits the data.

In the NicheLite stack, there are two BDs for RX and TX, but it has a very simple way to deal with the transmitted and received data.

TX - the NicheLite stack constructs a frame in a single buffer (a BIGBUF) that is sized to be big enough to contain the maximum size of frame. It could just set up a BD to point to this and be done, but the FEC requires buffers to be 32 and preferable 128 bit aligned to improve its DMA efficiency. The way the stack constructs a frame, starting with payload and adding protocol headers, it cannot guarantee the alignment of anything except the payload portion of the buffer. If the frame is not properly aligned it may copy the header to a LILBUF buffer and then set up two BDs to point to the header and the payload in separate buffers.

RX - the Nichelite stack uses rx buffers that are big enough to receive a maximum size frame - I think there may be a #define somewhere that sets this size. The FEC will truncate any frame over 2047 bytes long, but I think that the maximum size is set to be smaller than this in the unmodified stack. Two BDs are set up, pointing to empty rx buffers. When a frame is received, the fec_rx_isr() function replaces the buffer that has just been filled with a new frame with an empty buffer. If there is only a small amount of data it will copy the data to a smaller buffer in an attempt to save RAM. The new frame is then passed to the stack.

Now, the BIGBUFs and LILBUFs are managed by a queue. When the system starts, a queue with NUMBIGBUFS is created, same for NUMLILBUFS. These are used both for TX and RX. If there are no more free buffers available when the RX ISR runs, the received frame may be silently dropped. You may need to examine the code in fec_rx_isr()  to see where this can happen and then add some code to inform you when it happens to your UDP packet.

Unless the RX BDs are not being set up so that the FEC has somewhere to put the data then there is no reason why the hardware should miss a frame. We are able to stream data from a 5282 at wire speed (90mbit/s payload) and I don't see why the 52235 would be different.

Hope this helps.

Paul.

I suspect the reason your system conks out when you increase the number of buffers will be due to a lack of RAM for the buffers.

[wyliek]

Oh, here is the ethernet capture

Thanks Paul

That does clear things up a bit. Another thing I just noticed is that the fec_isr is occasionally detecting a Non Octet aligned error (not as frequent as dropped UDP packets). However this does not seem to be the cause of the dropped packet as it does not occur when a packet is missed. What I'd like to know is:

Is it normal to occasionally get these fec errors or is something seriously wrong?

I have looked at the fec_isr and asked it to output a character everytime a packet is received. When the device does not respond to a udp packet it seems that the fec_isr is not outputting this character. This leads me to believe it isnt even passed to the FEC. I will investigate further.

Any thoughts?

Thanks again  for your help

Kyle

[mccPaul]

The NO error is not a normal occurrence and it implies that something on your network is constructing strange frames. The FEC on the 52235 may be having to deal with traffic that is not unicast - it could be dealing with other broadcast or multicast frames (low end switches & hub may convert multicast frames to broadcast).

Try setting a breakpoint that is triggered when the rx ISR has a faulty frame and then examine the frame headers to see where it cam from. Ethereal is a big help here as you can use it to see how frame and protocol headers are constructed but you may end up having to google to see the spec for a frame.

If the frame tx'd to the FEC is good, it _will_ generate an RXF IRQ _if_ there are RX BDs set up and RDAR is written to tell the FEC that there are available BDs.

Cheers,

Paul.

[wyliek]

It seems your were correct

There are a couple of devices on the network which seem to be constructing strange frames. These are all Lantronix Xport devices. However this is not the route of my lost UDP packet problem, or at least not a direct cause. Is it possible that these packets are screwing up my receive logic?

Wait, it can't be because they are just dropped when this error occurs. So I'm back to square one. It does actually seem that my problem is occurring before the fec_isr as the mentioned UDP packets seem to not even make this interrupt.

Thanks for the pointers Paul

Cheers

[wyliek]

Hi again

I have been analyzing the fec_isr routine and have come across something that puzzles me.

In the receive logic section the isr first declares an rxpkt, then increments some fecstats counters. It then checks for receive errors. If there are no errors a new packet is declared for the receive ring and the variables:
int bd2, len;
BD *bdp2;
Then there is a check to see if the frame is spread over two buffers. In my case it isn't. So the consequent code is not executed. The next runable section checks to see if variable len is lilbufsiz to see whether to allocate a big or small buffer.
Where is variable len set?

Am I missing something?

Thanks
Kyle

see code below:

else /* we received a good BD */
{
PACKET pkt; /* new packet for receive ring */
int bd2, len;
BD *bdp2;


//putchar('*');
bd2 = next_rxbd;
if ((bdp->bd_cstatus & MCF_FEC_RxBD_L) == 0)
{
/* All frames should fit in a single BD. If they
* don't, i.e. RxBD_L is not set, the rest of the
* frame is in the next BD(s), so just combine them. */

do
{
putchar('2');
if ((bd2 = bd2 + 1) >= NUM_RXBDS)
bd2 -= NUM_RXBDS;
bdp2 = &RxBDs[bd2];

if ((bdp2->bd_cstatus & MCF_FEC_RxBD_E) == 0)
{
len = bdp2->bd_length;
/* last BD length is total frame length */
if (bdp2->bd_cstatus & MCF_FEC_RxBD_L)
len -= bdp->bd_length;

if (bdp->bd_length + len = MAX_ETH_PKT)
{
rx_copies++;

/* append the data to the first BD */
MEMCPY((void *)(bdp->bd_addr + bdp->bd_length),
(void *)bdp2->bd_addr, len);
bdp->bd_length += len;
bdp->bd_cstatus = (bdp2->bd_cstatus & ~MCF_FEC_RxBD_W) |
(bdp->bd_cstatus & MCF_FEC_RxBD_W);
bdp2->bd_cstatus |= 0x0400; /* unused bit */
}
else /* too much data to copy */
{

/* TODO: increment some error counter */
putchar('E');
goto next_recv;
}
}
else
{

/* TODO: increment some error counter */
putchar('e');
goto next_recv;
}
}
while ((bdp2->bd_cstatus & MCF_FEC_RxBD_L) == 0);
}

/* First, get a fresh packet for rx ring. If we can't get one then
* we have to dump the current packet by leaving it in the ring,
* since the FEC won't let us have an empty ring entry.
*/

/* If we can fit this packet into a little buffer, then get one
* and copy the packet. This slows performance a bit, but helps
* preserve big buffers in memory-tight systems.
*/
if ((len lilbufsiz) &&
((pkt = pk_alloc(bdp->bd_length)) != NULL))
{
/* copy received data into newly alloced small packet. Leave the
* big packet in the FEC ring BD, and plug the small packet's address
* into the rxpkt variable.
*/
putchar('l');
MEMCPY(pkt->nb_buff, (void*)bdp->bd_addr, bdp->bd_length);
rxpkt = pkt; /* put little pkt in local variable */
rxpkt->net = nets[fec_iface]; /* install net in rxpkt */
input_ippkt(rxpkt, bdp->bd_length); /* pass to IP stack */
}

Message Edited by wyliek on 2007-03-0201:29 PM

Message Edited by wyliek on 2007-03-0201:29 PM

[mccPaul]

Hi Kyle,

That looks like one of the bugs in this driver!

Change len in that line to bdp->bd_length and it should work (a bit better anyway).

Cheers,

Paul.

[Richly]

---

mccp wrote:

...

Change len in that line to bdp->bd_length and it should work (a bit better anyway).

I don't claim to understand the code in fec_isr, but wouldn't it be better to set up len at the top, immediately after it's declared, as follows:

len = bdp->bd_length; // will be changed if more than one BD.

The original post noted that len was not set in case there was only one BD. I presume that in the case of more than one BD, it was set correctly. My proposed patch has the effect of using the newly calculated value if there is one and using the value you suggest otherwise.

Best regards,
Richard

Message Edited by Richly on 2007-03-0505:46 AM

Message Edited by Richly on 2007-03-0505:46 AM