topic Re: File writing tasks silently failing on K60 and K64F in MQX Software Solutions

File writing tasks silently failing on K60 and K64F

ironsean — Mon, 03 Nov 2014 14:33:28 GMT

Hello, I've experienced an issue using MFS and a file writing task on

both a K60 custom board, and a K64F FRDM board: When the fopen() or

write() function is called the function will never return and the task will

cease operating. There are no errors thrown and I don't know where to

look to figure out why the function ceased. If I step through line by line

in the debugger eventually the debugger will just stop responding as

well.

It happened first with a custom K60 based board, using MFS on the

Internal Flash (to facilitate developing the file logic until a new board with

better storage was developed). When a specialized test task wrote to

Flash it worked, and when ONLY the file writing task (not the webserver,

telnet server, sensor reading, or data uploading tasks) it also worked.

But when all the tasks were running the file writing would silently stop

the task. By lowering the priority of the file writing task to the lowest

priority it began to function properly.

Now, on the K64F FRDM board we are attempting to refactor that task

to use the SD card. The internal flash writing still works. We have also

used the SD card example application to verify the SD card writing

works. I moved the SD card initialization into our full project and it

installs and uninstalls the SD card appropriately, but whenever we try to

write to it the task silently fails again.

Any idea where or how I could try and Troubleshoot this, or what might

be happening? Because of the task priority and the way it just ceases, I

wondered if there's some sort of deadlock that occurs with the file

writing task and one of the others. Priority doesn't seem to be helping

the SD card case though.

We're using CodeWarrior Version: 10.5

Build Id:130916

Re: File writing tasks silently failing on K60 and K64F

soledad — Mon, 03 Nov 2014 21:14:19 GMT

Hello Sean,

which MQX version are you using??

Regards

Sol

Re: File writing tasks silently failing on K60 and K64F

ironsean — Mon, 03 Nov 2014 21:19:52 GMT

On the K60 custom board it was MQX 4.0.x (we updated to .4 or .5 as the latest I believe), and with the K64F we are using MQX 4.1.0, from the FRDM board's specific MQX release package.

I'm sorry, this was copied from a service ticket where the MQX versions were already listed as part of the ticket metadata and I forgot to include them here.

Thanks,

Sean

Re: File writing tasks silently failing on K60 and K64F

RadekS — Wed, 05 Nov 2014 15:52:20 GMT

Could you please specify which priorities you used for your tasks?

General recommendation is do not use priority levels below 8 for tasks, because they could block some internal interrupts typically used for io operations.

Could you please specify your parameters for fopen()?

Unfortunately MQX up to 4.1.1 contains bug in append mode. Unfortunately, append mode is implemented wrongly.

As workaround, please use r+/w/w+ modes instead of append mode and seeks manually according your needs.

Next MQX release will contain complete redesigned MFS subsystem which will solve this issue.

There is also one additional bug in MFS specific only for MQX for FRDMK64F/TWRK64F120M and MQX 4.1.1:

https://community.freescale.com/message/441581

Have a great day,
RadekS

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

Re: File writing tasks silently failing on K60 and K64F

ironsean — Wed, 05 Nov 2014 16:56:37 GMT

No priorities below 8, and in fact the file writing only worked on the K60 with the lowest priority of the tasks.

{ MAIN_TASK,	main_task,	1200,	9,	"Main",	MQX_AUTO_START_TASK,	0,	0},
{ HTTP_TASK,	http_task,	1200,	16,	"http",	0,	0,	0},
{ SONAR_TASK,	sonar_task,	1200,	11,	"sonar",	MQX_FLOATING_POINT_TASK,	0,	0},
{ TEST_TASK,	test_task,	1000,	13,	"test",	0,	0,	0},
{ TELNET_TASK,	telnet_task,	1000,	14,	"telnet",	0,	0,	0},
{ SENDER_TASK,	sender_task,	1000,	12,	"sender",	0,	0,	0},
{ READ_TASK,	read_task,	1000,	12,	"read",	0,	0,	0},
{ LOG_TASK,	log_task,	1000,	18,	"log",	0,	0,	0},

I am in fact using the append modes, with this call:

logfile = fopen("MFS:level_log.dat", "a+");

Weirdly, the append mode seems to work fine on the internal flash, and on the previous chip, just not with the SD card. Also, the task doesn't stop on the fopen() call but on the write() call. Do you have any more information about this append bug so I can find out what's going on and what to expect, and if this may be the cause of my issue? Also, any ETA on the next MQX release? With that K64 specific bug in fseek, and append mode not working properly I might need to just wait.

Re: File writing tasks silently failing on K60 and K64F

RadekS — Wed, 05 Nov 2014 17:33:23 GMT

Thank you for additional information.

MQX 4.0.2 has problem in flashX (I am not sure if you use it) with data in cache. Write followed by read caused that you read wrong data. MQX4.1 contains fix with invalidate this cache.

Problem with append mode appears when file size is multiply of cluster size. See also:

https://community.freescale.com/message/441854

So, it typically works for few hundreds writes and after that it stops.

Re: File writing tasks silently failing on K60 and K64F

ironsean — Wed, 05 Nov 2014 21:19:22 GMT

Thank you for more information. I can confirm that my issues are not related to the append functionality (writing either works, or fails completely if I'm using the SD card), and I haven't written far enough to find the cluster size append error. I have even changed to "w" mode to overwrite new files and the issue is still the same.

The caches are all set to write-through for my application, as it's more resilient to power loss or SD card removal. But the functionality fails before writing anyway.

I am testing again now, and once the fopen() executes on the MFS:, which is installed without error on the sdcard:, the LED beside the card on the K64 turns red and blue, and the task ceases.

Re: File writing tasks silently failing on K60 and K64F

ironsean — Fri, 21 Nov 2014 14:18:07 GMT

Anyone else have any ideas where to even look to debug this? How to figure out what happened to the task that ceased operation?

Re: File writing tasks silently failing on K60 and K64F

pbanta — Fri, 21 Nov 2014 15:38:42 GMT

If you pause the debugger when the task is stuck what does the call stack for the stuck task look like?

Re: File writing tasks silently failing on K60 and K64F

ironsean — Mon, 24 Nov 2014 20:39:16 GMT

The Thread which is my logging task (0x10006) is this:

Thread [ID: 0x10006] (Suspended: Signal 'Process Suspended' received. Description: Process Suspended.)
	7 (AsmSection)() dispatch.S:178 0x0000044a
	6 _time_delay_internal() time.c:554 0x000299c0
	5 _time_delay_for() time.c:484 0x000298fc
	4 _lwevent_wait_internal() lwevent.c:1163 0x0002dd54
	3 _lwevent_wait_ticks() lwevent.c:1345 0x0002dde4
	2 _esdhc_send_command() esdhc.c:557 0x0000fada
	1 _esdhc_read() esdhc.c:1509 0x000107da

The Log_task is no longer even showed as the root task, and my debugger ends up looping through dispatch.S

Re: File writing tasks silently failing on K60 and K64F

asmith — Tue, 25 Nov 2014 12:30:03 GMT

I see something similar to this with MQX 4.1.1 with a K70.

It is my beleif that this has something to do with MQX kernel memory being overwritten.

I have debugged into the write functions and I am "hanging" when the indirect function is being called suguesting that the function address in kernel memory has been zapped somehow.

I am actually using arm-none-eabi-gdb --tui to debug rather than CW but see the same debugger hanging issue.

I can often trace further by stepping by assembler instruction rather than C instruction.

Re: File writing tasks silently failing on K60 and K64F

ironsean — Tue, 25 Nov 2014 21:59:29 GMT

I took a quick look myself, and found this for an approximate code flow:

log_task.c

io_fopen.c:48-97

mfs_init.c:323-418

mfs_create.c:58

mfs_dir_entry.c:462

mfs_dir_entry.c:58-93

mfs_search.c:121

mfs_rw.c:437 MFS_Read_device_sector

mfs_rw.c:311

mfs_rw:256 read()

io_read.c:53

io_read.c:78 res = (*dev_ptr->IO_READ)(file_ptr, data, num);

part_mgr.c:1420 read() io_read, sdcard.c:285

sdcard_esdhc.c:363-408

io_read.c:74 in _mqx_int _io_read:

if (file_ptr->HAVE_UNGOT_CHARACTER) {

res = (*dev_ptr->IO_READ)(file_ptr, data+1, num-1);

}

else {

res = (*dev_ptr->IO_READ)(file_ptr, data, num);

I get the debugger to hang on that final call to io_read.c:74 after getting through the sdcard_esdhc.c code, and it gets to the if statement at line 74 where if (file_prt->HAVE_UNGOT_CHARACTER) can be seen as false through the debugger, but stepping over or into that statement doesn't lead to either of the res = calls. I'll see if I can step through assembly code and get any further.

Re: File writing tasks silently failing on K60 and K64F

asmith — Wed, 26 Nov 2014 14:32:52 GMT

You appear to be failing mush as I am at the indirect call

res = (*dev_ptr->xxxx)

My guess is that the memory where dev_ptr points has been trashed.

I have been trying to chase down MQX memory corruption thus far without success.

Re: File writing tasks silently failing on K60 and K64F

ironsean — Thu, 27 Nov 2014 15:24:31 GMT

You might be right, after some further testing the exact moment it loses it is inconsistent, but it's usually at one of those DEV_PTR calls. And a couple times when I've checked the variable right before losing it the contents of the struct are almost all 0x59D.

Re: File writing tasks silently failing on K60 and K64F

RadekS — Thu, 04 Dec 2014 14:55:22 GMT

Unfortunately it doesn’t exactly fit to any of known issues.

Is it possible place watchpoint at this DEV_PTR (and address where it points) and check whether dev pointer is modified somewhere between fopen on fclose?

Did you try different SD card? I know about one old and slow SD card here which has similar issue (it just stop working after few writes). We suppose that there is some issue due to timeouts during card transfers, but we didn’t have time for investigation till now.

Re: File writing tasks silently failing on K60 and K64F

asmith — Thu, 04 Dec 2014 21:01:10 GMT

In my case I am using a K70. I am not using an SD.

I am building MQX using GCC from the command line under Ubuntu 14.04. I have used both the ubuntu Mutli-arch arm gcc compiler an the recomended kinetis GCC toolchain.

I have built for DDR and SDRAM all with the same results.

Eventually I execute and indirect call inside MQX and things go to hell.

I can play games with the code and delay this, but ultimately it is happening.

I am looking for a clue where to look to try to find this.

I am also trying to build my application with KDS to see if somehow that makes a difference, but it is a fair amount of effort creating a project file.

Re: File writing tasks silently failing on K60 and K64F

ironsean — Thu, 04 Dec 2014 21:12:21 GMT

My experience on a K60 board writing to internal flash via MFS failing also supports that it isn't an SD card issue. In that case, I could write to the flash when only the testing task was running, but I could not write while my other tasks were running until I changed the file writing to have a lower task priority (possibly causing whatever task was wiping the memory to not run during the call anymore?)

As well, if I simply compile an SD card example project I can read and write to the SD card fine. but when incorporating the same code in my project it fails. I will try and set the watchpoints and see if I can find it changing for sure.

Sean

Re: File writing tasks silently failing on K60 and K64F

ironsean — Fri, 05 Dec 2014 20:41:08 GMT

David, what stack size are you allocating for the task doing file writing? I found out I was getting a stack overflow during the file write call (actually debugging another task). When I bumped my allocation up from 1000 to 2000, I see my stack usage jump up to 71% during file writing, which is well above what the old limit would have been. Now writing works properly.

Re: File writing tasks silently failing on K60 and K64F

asmith — Mon, 08 Dec 2014 16:42:34 GMT

I have tried so many things it is hard to recall details about all of them.

I beleive I have had failures with single tasking examples and 2000+ stacks.

Right now I am stalled because something caused USB to quit working entirely, and I can not get back to where it was.

Need to push to the SCM more frequently

Re: File writing tasks silently failing on K60 and K64F

asmith — Mon, 08 Dec 2014 16:46:35 GMT

I would have little problem beleiving this was a stack issue,

though there is still the question of why kernel data is getting corrupted ?

Stacks are supposed to come from free memory. Why would a stack overflow encroach on kernel data ?

I am really not happy with the magic numbers in the link files.

Explicitly placing one kind of memory at an absolute address while allocating other memory out of a pool that overlaps seems like a recipe for disaster to me.