A quick overview: I have a product that has been shipping for a number of years now and I've come across a problem that has occured in the field but I've never been able to replicate in the lab. It has only occured 2 or 3 times out of several hundered units but the problem is consistant enough to make me think that it's not just a flaky part. This product is used in an automotive environment and I have several other similar (but different) products that have never displayed the problem.
The product has an in-house developed serial bootloader, field upgradable firmware and a user adjustable set of parameters (via serial port) stored in flash. In these few cases of failure when the unit has been powered down after running for a time (and I'm not certain this is true in all cases) the system will not startup again properly. I connect the BDM to find that one sector (always the same sector, second sector of page $3E) has been erased. This page is used to store the field upgradable firmware and should only be written or erased when upgrading the firmware (through the bootloader). The customer could if properly informed reload the unit's firmware by using the bootloader power-on button and doing a normal firmware update with the current firmware and the user parameter data even remains intact.
I found an issue with the value in FCLKDIV (Flash clock was set too fast). I fixed that problem, but the unit I currently have still has the problem inspite of having the correct value in FCLKDIV.
I now suspect an issue with transient input voltage but I'd like to know if anybody else has a suggestion as to what could cause a (seemingly) specific sector to be spontaneously erased.