MKW41Z Thread commissioning issue with fsci bootloader

[michaellundberg]

Greetings:
I am building a Thread based sensor network using the MKW41Z parts.
There are SED endpoints, and a Host Controlled Device attached to a Linux
host via the serial port. I am using the 2.2.2 version of the SDK,
though I first noticed this issue with 2.2.1.

Generally, things are working the way they should. The HCD is
pretty close to the vanilla example provided in the projects. But
I want to make the HCD upgradeable, using the fsci bootloader.
I modified the HCD loader file, and was able to successfully upgrade
the firmware. My SED devices that were previously commissioned, continued
to work after the HCD firmware was upgraded. However, if I tried to
commission a new device, the code did not work.

I have traced this down, I think, to the thread library, which makes a
read cycle to the interrupt vector table. With the default HCD code
(which runs without a bootloader), the read (from address 0x0000001c)
returns a value of 0, which seems to satisfy the library, and everthing
works. With the fsci bootloader, the IVT is filled with default
vectors, and the value returned is not 0, and the library does not
seem to start the DTLS session necessary to commision the new endpoint.

I set a breakpoint on the read to 0x000001c, and it is coming from a
routine named "IP6_Service". From the assembly language, it looks like perhaps
it is expecting a non-zero pointer from the "IP_IF_GetAddressDataStruct6" function,
and indexing 0x1c (28) off of that value. But when it gets a 0 pointer, it ends
up reading from the IVT (my speculation, can't really tell without the
library source).

Even when the code works (i.e. no boot loader), the read is still happening,
but the HCD code plugs "0x0" into the IVT. I hacked the fsci bootloader
to place 0xffffffff into the 0x1c address, and things work okay as well.
For now, I'll make the IVT at 0x1c be 0, to match the non-bootloader hcd code

Just wondering if anyone else has observed this issue? My thinking is
that the library should protect itself better against NULL pointers, of
course. But is there some configuration I should use to bypass this issue?
Seems somehow related to IPV6, should I disable (worried other things
might break)?

Thanks!
Mike Lundberg

[estephania_mart]

Hello,

Sorry no, I just added the wrong name of the file, I meant the Kinetis Thread Stack Application Developer's Guide as you can see here.

Sorry for putting the wrong name of file, but in the Thread developers guide you can find the information I referred in the last post about the linker files when using the Fsci_bootloader.

Sorry for the inconveniences cause by the name error.

Regards,

Estephania

[estephania_mart]

Hello,

Just to confirm, when you deploy your example with the HCD, did you make sure that you made the linker file changes mentioned in the BLE Application Developer's Guide? When using the fsci_bootloader you must change the linker file configuration of the project, if you did not make those changes there might be some memory issues , could you please check that and confirm ?

Regards,

Estephania

[michaellundberg]

Hi Estephania!
Again, thanks for your patience and your help.

I'm working with Thread, not BLE (though I understand that much of
the information can be applicable to both). So I followed
"Kinetis Thread Stack and FSCI Bootloader Quick Start Guide".
In section 1.4, it says "The user is requested to overwrite
MKW41Z512xxx4_connectivity.ld with the one under
boards\frdmkw41z\wireless_examples\thread\reed_ota_client\freertos."
That is what I did. I've attached the original (no_bl...) and the
modified (bl...) loader files.

My understanding is that the fsci bootloader, when working normally,
jumps to the application, and consumes 0 Ram space (i.e. any ram used
by the bootloader is overwritten by the application). Of course,
the bootloader does take some flash space, and the HCD application is
displaced to make room. But even then, I think there is still 100K
of extra flash space.

The difference that I see is that in main.c, the fsci boot loader fills the
the IVT with pointers to various interrupt handlers (for address
0x1c, that would be the default handler). When the HCD application
is built without allowing for the boot loader, the "reserved" IVT
entries get "0" because of the values in startup/startup_MKW41Z4.S
This changes the thread library behavior, when it reads from the
IVT area.

Thanks!
Mike Lundberg

[michaellundberg]

Hi Estephania!
Thank you for your response to my issue.

I do understand that you don't share the Thread Library source, no problem.

While your understanding of what I described is correct, I only mentioned
that previously commissioned nodes worked to show that the HCD was
generally operational, outside of commissioning. If I start with no
existing commissioned endpoints, and try only to commission the first one,
it fails to commission.

When I first hit this problem, I spent more days than I'd like to admit
looking at memory, because to build the upgradeable HCD, I had to relocate
the HCD application to make room for the fsci bootloader. Eventually,
I discovered that if I erased the device, then only installed the relocated
HCD code from MCUExpresso, and ran it, commissioning worked correctly.
If I installed the bootloader along with the HCD application, then
commissioning would not work, even though I started from MCUExpresso,
and the bootloader never ran.

I installed smaller and smaller pieces of the fsci bootloader, until
I was only installing a very small piece of the IVT. It narrowed
down to that 0x0000001c location: If 0xf1 at 0x0000001c, commissioning
would fail. If 0x0, or 0xff at 0x0000001c, commissioning would work.

This failure can be reproduced with the sample code. If you build
a router-eligible-device, and a end-device, and put them each
on a FRDM_KW41Z (I actually use the Rigado R41Z), you can use
"thr create" on the REED and "thr join" on the ED, and the endpoint
joins. But if you modify that 0x0000001c location on the REED, by writing
0xf1 into it (just like the fsci boot loader has), then the
endpoint will not be able to join the thread network, because commissioning
won't work.  You can add a "watchpoint" at 0x0000001c after creating the network
(make sure to check "read", not write), in MCUExpresso, to see the issue.

The Thread Library appears to me to be mistakenly accessing the
Interrupt Vector Table, and behaving differently depending upon what
it reads back. If I'm right that this is a bug, I guess I'm looking for two things:
1) If possible, someone can confirm the Thread Library bug, and maybe fix it.
2) Let others know about this issue, in case they face the same problem.

Thank you for your attention, and your suggestions. I want to say that
I am very impressed with the capabilities of the source code that NXP/Kinetis
provides. Also, having these forums to discuss issues is great!

Thanks!
Mike Lundberg

[estephania_mart]

Hello,

If I understand correctly you have already a network with some nodes , but at a certain point when you have your network and try to add a new device this fails, is this correct?

If this is the case , have you checked the memory ? It is possible that device is getting out of memory depending on you application size and how many devices you have already in your network, in the release notes of the SDK you will fin the footprint for most of the examples, considering that you have the library footprint + the fsci bootloader + your application + the routing table of the network it can be possible that the device is going out of memory.

About the source code of the libraries, I apologize but we do not share the source code of the pre-compiled libraries

regards,
Estephania