MQX Performance with MQX Bootloader

[stevejanisch]

I am evaluating MQX 4.0 on a TWR-MCF5441x evaluation kit.

For several weeks I have been evaluating MQX 4.0 performance for a future application.  The application is for precision control of some high speed servo motors and to accomplish this I have created a high priority timer interrupt.  I am using the MQX function _time_diff_microseconds to evaluate the jitter for the high priority interrupt.  I log and record peaks and averages for the jitter and display them on a telnet debug screen I have incorporated on a lower priority task.

I am building the project for external Ram Release.

When I launch the application from CodeWarrior (CTRL+F11 Run ) the jitter is minimal.  I do get some, but mostly all are within within +/- 50 microseconds of expected.

My next step was to launch the application via the MQX bootloader included as an example application.  This application runs a small program that accepts binary image files via TFTP and stores them in flash.  It can be configured to automatically read a binary from flash and copy it to RAM and launch it, so is a viable option for production.

I would have expected the two approaches to have identical or at least very close results.  But the application launched by the bootloader show a lot more jitter -- both in frequency and amplitude.  There are many interrupts outside of +/- 500 microsecond.

Has anyone else witnessed this sort of phenomenon, or have any idea why the two environments have drastically different results?

I have the following thoughts but really cannot seem to come up with a good theory, or even a good starting point to correcting the problem:

(1) The bootloader itself runs as a low priority task (lower than any of the tasks in my application).  It exits after copying and launching the application with the mqx function _mqx_exit.  Should it shut down differently?

(2) Is there some "clean-up" required of the MQX bootloader program and the lack of this is messing up the application tasks?

(3) I'm not sure exactly what happens with the CodeWarrior launch, but I assume it uses the ELF file.  The bootloader is copying and launching a binary image -- are these two files not functionally equivalent?

Any ideas, pointers -- even snide remarks -- appreciated...

[GeneGoff]

Hi Steve,

Sorry I can't help with your issue, but I am curious about what you said "...MQX bootloader included as an example application.  This application runs a small program that accepts binary image files via TFTP and stores them in flash."

A couple quick questions:

1) Does this bootloader accept the binary via TFTP through Ethernet and using the stock RTCS?

2) The image is stored in flash, so on the next power-off-on cycle it should execute the image, and so this can work as a flash program updater through Ethernet?

3) Where can this example application be found in MQX?  I know earlier versions of MQX had an Ethernet-based flash bootloader but it was abandoned.

Thanks,

Gene

[stevejanisch]

Tough to get help these days... I've posted a lot of my musings in the hope that they will allow others to be less frustrated with this stuff than I have been.

The example bootloader I refer to does accept binary via TFTP and therefore yes to both (1) and (2).  (3) could be a bit harder... as you no doubt know documentation and examples are obfuscated and clunky (although as obfuscated and clunky as they are, I hope they aren't abandoning the examples).

There are numerous documents and bits of example code on this website... I know because I have a not-so-small stack of printouts sitting next to a nearly empty jar of Motrin as I've tried to put together the various pieces for my project.  Some have worked, many have not... it's hard to piece together your processor, your board and various versions and patches of everything to come up with a working example.

I am using MQX 4.0 and CW 10.3 with a TWR-MCF5441x evaluation board.  I'm not sure whether the examples were loaded as part of MQX or with some type of CW or MQX patch for the evaluation board, but it shows up as one of the basic examples when you get to the "Select Application type and template" portion of the new project wizard.

I can see that for different evaluation boards, the bootloader example isn't listed.  The pdf document that comes with the example (MQXBLDUG Rev. 1.3 07/2011) states that it supports NAND or NOR flash drivers and tested on the following boards:

• TWR-MPC5125 with NAND flash support
• TWR-MCF54418 with NAND flash support
• M5329EVB with NOR flash support
• M53015EVB with NOR flash support
• M54455EVB with NOR flash support

This example code has actually been one of the better example pieces I've come across, although it didn't work out of the box because a misplaced printf seemed to crash the bootloader.  Of course anything is better than nothing, which is what I've had to deal with more than a few times.

But if that doesn't work for you, there are at least two other example sets that might... one you can compile via Linux but I couldn't get any TFTP server to work with it; another uses some other Ethernet library you can get through Freescale, but they don't support the controller I am using so I didn't look too closely at that one.

Feel free to reply back if you need anything more specific.

[GeneGoff]

Thanks for the info Steve.  "Obfuscated" - that's a new word for me, and it describes Freescale's examples well.  For example, the web_hvac example project took me a long time to figure out how it works, basically the XMLHttpRequest AJAX stuff, because there was so much hvac app code in there which obscured the real meat of how the web server works. 

I did some digging and found out the bootloader you are using was indeed the Freescale MQX 3.8/mqx/examples/bootloader sample project which was dropped in MQX 4.0.  I sent in an SR to Freescale and they replied that it was deliberately dropped from 4.0 and they don't have plans at the moment to develop an Ethernet bootloader.  Incidentally, this bootloader is also covered in the app. note MQXBLDUG.  Since I'm using the Kinetis K60, CW10.4, and MQX 4.0, that bootloader won't help me.

However, I discovered FNET which is a fairly comprehensive open-source IP stack which includes a booloader which worked fine on my platform.  Check it out here: Embedded TCP/IP stack: FNET Overview .  Apparently there is a licensing conflict with MQX, so you won't see FNET integrated with MQX, but FNET can run bare metal and the bootloader can run on it's own 48KB boot block by repositioning the vector table and where MQX sits - this is detailed in AN4367 which never mentions the name FNET, but the app note is indeed using FNET 1.0.0.

A second hurdle I'm working on is getting an embedded web server project running where in the client browser you can just type in a local domain name and not have to know the IP address of the web server.  I know the technique is basically using mDNS and a local domain server running on the browser PC, but MQX's DHCP client currently doesn't support Option 12 needed to send out the web server's host name to the PC's DNS server.  Let me know if you have any experience with this.

-Gene

[stevejanisch]

At the time I started my path they were not supporting my processor in FNET, but after visiting the site I see they now are -- thanks for the tip.  It's too bad they dropped the bootloader example... it was one of the better examples I came across (after you fixed the bugs, of course... oh god in heaven, is that how low the bar is set).  Perhaps I will return to it if I still have trouble launching an MQX program from my bootloader (if you review my original postings, my main focus was how poorly MQX performance ran from the bootloader... I think I have a solution although it doesn't give me a lot of confidence).

I don't have specific experience with the Option 12 DHCP client.  I wonder if it would be worth looking into another open source solution and see if you could port it.

[DavidS]

Hi Steve,

If ultimately both images are loaded to DDR space but only the performance is different I'd recommend checking cache settings.

My assumption is the core frequency is same but always good to check clock configuration too.

Regards,

David

[stevejanisch]

Thanks for your reply, David.

I haven't called any MQX cache functions; and the clock seems to be set to the same nominal value.  At any rate the two files seem to be the result of the same build... so are using the same LCF file.

Let me give you an example of the difference in jitter I am seeing.  I have a timer interrupt that pretty much just calls _time_diff_microseconds to evaluate how much time has passed since the last timer interrupt.  My low priority telnet task merely displays these values.

This first group of 100 values is from the CodeWarrior launch.  The timer interrupt is configured for 1 millisecond so a value of 1000 microseconds is nominal.  As you can see, the values are all very close to nominal. My worst case jitter is 5 milliseconds microseconds:

998    1003   998    1000   999    1003   998    1000   997    1003

1001   999    999    1002   997    1001   1000   1000   1004   1000

1005   996    999    998    1002   998    1001   999    1002   998

1001   997    1004   999    1001   998    1002   997    1000   1002

1003   996    1002   998    1004   996    1001   1000   1001   999

998    999    1002   1001   999    998    1006   996    999    1000

1000   1001   1017   985    998    1002   998    1000   997    1003

999    1000   997    1005   999    998    999    1005   1003   995

999    1003   996    1000   999    1004   998    999    999    1002

1000   999    998    1005   998    998    1001   1002   999    1001

This second group comes from the bootloader launch:

858    997    1002   999    1159   842    1001   996    1006   1163

835    997    1001   1003   1168   831    1001   1000   1002   1144

851    1003   999    1001   1158   840    1000   1000   1003   1167

831    998    1000   1005   1170   825    1001   998    1003   1141

856    1000   999    1003   1155   844    1002   1146   856    1004

1199   798    1281   739    1159   820    1178   822    1001   1146

852    998    1002   1003   1157   842    999    999    1004   1164

833    999    1000   1004   1178   819    1000   1001   1002   1143

854    999    999    1004   1158   840    1000   1000   1000   1166

834    998    1003   1002   1178   821    999    1002   1000   1141

Both are fairly typical collections.  For the bootloader launch my jitter is typically +/- 250 milliseconds microseconds -- that is 50 times that of the CodeWarrior launch!  The average over the 100 points is roughly 1000, which leads me to believe it is trying to fire every millisecond -- in fact you can often see a short value following a long value (you can read the queue from left to right... so for example 842 follows 1159 in the first row... the result is 2001.

It's almost as if some other high priority task or interrupt is interfering with the application... but I am using the bootloader code pretty much out of the example, and it doesn't seem to have anything like that (not to mention that the application only runs on the exit handler of the bootloader, so it would appear it is closing).

Edited to replace milliseconds microseconds...

[stevejanisch]

Actually tried a few things that really make me think the whole jitter issue has something to do with the bootloader not finishing up correctly...

First off, I added a couple of printf's to the bootloader program itself.  With these in place it seemed I was actually able to get the boot application launch to hang (I had revised the bootloader to redirect stdin/out to the COM port and the printfs didn't complete).  I added some _time_delay(x) functions in the bootloader and the printfs completed and the application launched once again...

Then I decided to try and add _time_delay(x)'s to the application as well... I noticed I had some test code at the beginning of the task that was assigned with MQX_AUTO_START_TASK that effectively had about 2 seconds of sleep time.  When I took out the sleeps it actually made the jitter WORSE.  I now had what I originally reported... a lot of interrupts that were +/- 500 microseconds -- and even a few worse than that.  I put in sleeps of 5 and then increased up to 10 and it did somewhat appear to get better, although it still wasn't as good as the CW launch (I could run for hours and never see anything out of +/- 100 microseconds).

Seems like I am on to something, but not sure exactly what... always have hated cheesy "fixes" based on arbitrary sleeps... would be nice to know better what is happening when an MQX program is completed and then another one is started -- maybe I should try not ending the bootloader at all, but perpetually putting it in some deep pause state...

Anyone else ever see anything like this?

[stevejanisch]

Might have made an important observation... quite by accident.

Up until this point I had been comparing the CW launch to the example bootloader launch of the RAM/Release builds.  Somewhere along the lines I started building the RAM/Debug versions.

It appears the Release and Debug Versions differ greatly for the bootloader launch (they appear to be pretty much the same if they are launched by CodeWarrior).

Compare these results:

Jitter	CW Launch	Bootloader
		Release	Debug
±10	99.21%	28.81%	99.20%
±25	0.23%	0.38%	0.24%
±50	0.14%	0.12%	0.14%
±100	0.14%	1.99%	0.14%
±200	0.29%	56.82%	0.28%
±400	0.00%	5.17%	0.00%
±500	0.00%	1.36%	0.00%
±1000	0.00%	5.35%	0.00%
±1500	0.00%	0.00%	0.00%
Time (minutes)	20.00	19.92	20.28

The jitter for the CW Launch and the Bootloader launch of the Debug binary are nearly identical; the Release binary shows much more jitter.

The release and debug builds do use different LCF files, but their contents are identical.  I haven't seen any big differences in the build configurations, but have only made a cursory comparison.  One thing is very obvious -- the file size of the binary is about 25% larger for the debug build... I'm going to try and figure out exactly what causes this change.  At any rate I find the results surprising and perplexing.

[stevejanisch]

Continuing the conversation with myself... close examination of the xMAP files created with the release and debug builds shows the code block is 412,112 bytes for the release and 509,440 bytes for the debug build.  If I compare the functions from both builds there are about the same number -- maybe two extra files for the debug version.  My application code is nearly identical, but the stack sizes for the MQX, BSP, and PSP routines are very often larger.  So this explains the size difference in the binaries when building RAM release and debug binaries.

Still don't have a theory for why MQX performance is so terrible with the release version -- or at least the jitter appears much worse with the release version.

For now I will cautiously move forward with the directive to not trust the release builds and hope for the best.

[stevejanisch]

Another useful discovery -- release build runs well as bootloader launch... if application code is optimized to 4.

Build	Release				Debug
Optimization	0		4		0		4
File Size	418k		378k		516k		476k
±10	1057653	88.303%	1212393	99.606%	1194606	99.18%	1376376	99.122%
±25	1110	0.093%	1985	0.163%	2383	0.20%	7618	0.549%
±50	672	0.056%	1165	0.096%	1460	0.12%	1786	0.129%
±100	1515	0.126%	366	0.030%	2331	0.19%	1565	0.113%
±200	109284	9.124%	370	0.030%	3703	0.31%	1219	0.088%
±400	21599	1.803%	691	0.057%	2	0.00%	5	0.000%
±500	2943	0.246%	15	0.001%	0	0.00%	0	0.000%
±1000	2978	0.249%	201	0.017%	0	0.00%	0	0.000%
±1500	0	0.000%	0	0.000%	0	0.00%	0	0.000%
±2000	0	0.000%	0	0.000%	0	0.00%	0	0.000%
±3000	0	0.000%	0	0.000%	0	0.00%	0	0.000%
>3000	0	0.000%	0	0.000%	0	0.00%	0	0.000%
Time	1197754	19.96 min	1217186	20.29 min	1204485	20.07 min	1388569	23.14 min
±10 to ±50	88.452%		99.865%		99.499%		99.799%
±100 to ±200	9.251%		0.060%		0.501%		0.200%
>200	2.298%		0.075%		0.000%		0.000%

Note that the MQX release libraries are compiled with optimization 4 (default of wsd).  When the application code is also compiled for release with optimization 4 it runs with much less jitter than if compiled with optimization 0.  The debug builds also showed improvement with optimization 4, although there wasn't as large a change (the MQX debug libraries are compiled with optimization 0... also default of wsd).

At this point in time, I would say that either debug or the optimized released will work, although the worst case jitters will need to be examined a bit closer.