Programming help needed, storing data

khumphri · ‎01-27-2006

This message contains an entire topic ported from a separate forum. The original message and all replies are in this single message. We have seeded this new forum with selected information that we expect will be of value to you as you search for answers to your questions.

Posted: Sat Nov 5, 2005 6:26 pm

For a MC9S12DP256B

I'm using this board:

http://www.axman.com/education/edc_hc12.html

I now need to now store data variables, so if there is a power off situation, the data is stored -> a back up power supply will not suffice. I adapted a program provided by Axiom (written in C), to take in data from a sensor, and time stamp the data and save it in an array.

However, I am having an ENORMOUS difficulty understanding how to:

--> save data from this array, in the event power is lost

I do not know assembly. I cannot find any books to help me with this problem.

Where can I find a solution - not necessarily the "answer" but somewhere that will explain to a non-EE how this sort of data storage works? I gather there are places to write data to, but haven't the slightest clue where to read about how to write to those portions.

Thanks!

Posted: Sat Nov 5, 2005 8:28 pm

In the MC9S12DP256 there are two different memories that retain data when power is off.

The terminology used is the result of historical accident, and not very intuitive.

The "flash" memory is an "Electrically Erasable Programmable Memory" or "EEPROM" that is optimized for fast block writing at the expense of erase - write cycles. The "EEPROM" is a similar memory, but optimized for many erase-write cycles.

You probably will find it easier to use the "EEPROM", unless you accumulate more data than will fit in the EEPROM, in which case you will have to use "flash" memory.

Your compiler may provide library routines for writing to EEPROM or flash. Search the compiler and library documentation.

If you can't find the appropriate library routines, it relatively easy to write your own functions in C to do this.

I recommend studying the Freescale AN2400/D, an application note that describes flash and EEPROM and the usage considerations. At the end it gives well commented examples of flash and EEPROM programming functions in both C and assembler.

If your data will fit, it is much easier to use EEPROM than flash because of the special considerations for flash memory usage I describe below.

There is a problem that the flash page visible in the "program page window" with 16 bit addresses form 0x8000 through 0xBFFF is controlled by the PPAGE register, which is changed by the CALL and RTC instructions. (These are the instructions used for calling and returning from "far" functions in paged memory.) A consequence of this is that you should handle reading and writing of data in paged flash from functions that are NOT in paged flash.

When the flash memory is being programmed, it cannot be read. This means that you cannot execute instructions out of a flash memory that is being programmed.

On the MC9S12DP256, the flash memory actually has 4 independent sections. This allows you to program one section with code operating out of a different section.

Because of the problems with the paged window, this code also needs to be in one of the "fixed" flash pages, either page 0x3F, which is always mapped to 16-bit addresses 0xC000 through 0xFFFF or page 0x3E which may be mapped to 16-bit addresses 0x4000 through 0x7FFF.

(On some of the less expensive HC(S)-12 parts there is only one section of flash, so you must transfer some of the programming code into RAM to reprogram the flash.)

These sections of flash hold several flash "pages". The Freescale documentation mostly calls the sections "flash arrays", but some of the documentation calls them "flash banks". In any case you need to recognize that a bank or array contains several pages.

There ware provisions for protecting parts of flash and EEPROM from further programming, and there is also a provision for "securing" the entire microcontroller to keep the contents of the microcontroller secret.

I recommend that you avoid doing anything with these features until you have the basic flash and/or EEPROM working reliably. If possible, avoid them completely, because errors in handling them hobble the microcontroller and are hard to debug.

Hope this helps. Please tell me if anything is still confusing, or if you need more information.

Posted: Wed Nov 9, 2005 5:37 pm

> In the MC9S12DP256 there are two different memories that retain data when

> power is off.

>

> The terminology used is the result of historical accident, and not very

> intuitive.

>

> The "flash" memory is an "Electrically Erasable Programmable Memory" or

> "EEPROM" that is optimized for fast block writing at the expense of erase -

> write cycles. The "EEPROM" is a similar memory, but optimized for many

> erase-write cycles.

The EEPROM - not flash - is what I was thinking about using.

> Your compiler may provide library routines for writing to EEPROM or

> flash. Search the compiler and library documentation.

It looks like the compiler I am using:

http://www.gnu-m68hc11.org/

has a persistent storage example:

http://stephane.carrez.free.fr/examples/persist.php#persist

> I recommend studying the Freescale AN2400/D, an application note that

> describes flash and EEPROM and the usage considerations. At the end it

> gives well commented examples of flash and EEPROM programming

> functions in both C and assembler.

I'll re-read it.

> If your data will fit, it is much easier to use EEPROM than flash because

> of the special considerations for flash memory usage I describe below.

I believe it will. I'm restructuring an array/struct, but it will contain the following data:

hour (char, 0-23) <- this is actually the index of the array/struct

Year (char, 00-99)

Month (char, 1-12)

Day (char, 1-31)

Count (int, 0-10000)

> Hope this helps. Please tell me if anything is still confusing, or if you need

> more information.

My biggest problem isn't the programming (for now), it's trying to visualize the data storage. See, I'm used to directory structures (like in Windows), and I'm not an EE. So, writing to a specific location is very confusing for me. In addition, I'm relatively new to Hex/binary and other notations.

If I knew how to draw a visual map of the data, I think it would help me understand the whole memory location/addressing properly. I also need a good thorough primer on terminology used, I'll probably google this later today.

Thanks!

Posted: Wed Nov 9, 2005 11:10 pm

I'm confused about what you are confused by, but it sounds like C structures, arrays and indexing are not things that you are comfortable with yet.

Are you trying to code this problem in C style or C++ style?

If you are trying to program in C++ style, you will need to understand classes and templates to understand the "persistent" class template in the example you cite.

Try reading up on them again, and then try to answer the questions I ask below.

> > In the MC9S12DP256 there are two different memories that retain data when

> > power is off.

> >

> > The terminology used is the result of historical accident, and not very

> > intuitive.

> >

> > The "flash" memory is an "Electrically Erasable Programmable Memory" or

> > "EEPROM" that is optimized for fast block writing at the expense of erase -

> > write cycles. The "EEPROM" is a similar memory, but optimized for many

> > erase-write cycles.

>

>The EEPROM - not flash - is what I was thinking about using.

>

> > Your compiler may provide library routines for writing to EEPROM or

> > flash. Search the compiler and library documentation.

>

>It looks like the compiler I am using:

>

>http://www.gnu-m68hc11.org/

>

>has a persistent storage example:

>

>http://stephane.carrez.free.fr/examples/persist.php#persist

My reading of this example is that the work of using EEPROM is all hidden in the C++ "template<class type> class persistent< type >", which is defined elsewhere.

> > I recommend studying the Freescale AN2400/D, an application note that

> > describes flash and EEPROM and the usage considerations. At the end it

> > gives well commented examples of flash and EEPROM programming

> > functions in both C and assembler.

>

>I'll re-read it.

>

> > If your data will fit, it is much easier to use EEPROM than flash because

> > of the special considerations for flash memory usage I describe below.

>

>I believe it will. I'm restructuring an array/struct, but it will contain

>the following data:

>

>hour (char, 0-23) <- this is actually the index of the array/struct

I don't understand how you intend this. Can you explain why this item only should be an index?

>Year (char, 00-99)

>Month (char, 1-12)

>Day (char, 1-31)

>Count (int, 0-10000)

Do you have a problem understanding this C fragment?

typedef struct

{

char Hour ;

char Year ;

char Month ;

char Day ;

int Count ;

} DATA_POINT ;

DATA_POINT SavedData[ 100 ] ;

...

SavedData[ i ].Hour = ThisHour ;

SavedData[ i ].Year = ThisYear ;

SavedData[ i ].Month = ThisMonth ;

SavedData[ i ].Day = ThisDay ;

SavedData[ i++ ].Count = ThisCount ;

...

Do you have any problem understanding this diagram?

+------------+

| Hour[ 1 ] |

+------------+

| Year[ 1 ] |

+------------+

| Month[ 1 ] |

+------------+

| Day[ 1 ] |

+------------+----------+

| Count[ 1 ] |

+------------+----------+

| Hour[ 2 ] |

+------------+

| Year[ 2 ] |

+------------+

| Month[ 2 ] |

+------------+

| Day[ 2 ] |

+------------+----------+

| Count[ 2 ] |

+-----------------------+

...

I think that you will find things a little easier to understand and program

if you re-arrange the item in the entry like this:

typedef struct

{

char Year ;

char Month ;

char Day ;

char Hour ;

int Count ;

} DATA_POINT ;

This has the data in a logical order like the digits of a number. The most rapidly changing variables are at the end, and the least rapidly changing variables are at the beginning.

Posted: Thu Nov 10, 2005 1:51 am

> I'm confused about what you are confused by, but it sounds like C structures,

> arrays and indexing are not things that you are comfortable with yet.

>

> Are you trying to code this problem in C style or C++ style?

C

I'm comfortable with arrays (though I do not know the terminology properly), structs are something I was introduced to only recently.

> >has a persistent storage example:

> >

> >http://stephane.carrez.free.fr/examples/persist.php#persist

>

> My reading of this example is that the work of using EEPROM is all hidden

> in the C++ "template<class type> class persistent< type >", which is

> defined elsewhere.

Ok, sorry!

> >hour (char, 0-23) <- this is actually the index of the array/struct

>

> I don't understand how you intend this. Can you explain why this item only

> should be an index?

I guess I wasn't clear. If I use an array, I mean to say the pointer/reference, ie:

arrayData[hour]

So the array is 24 rows long. As the day progresses, the array is written over from the beginning.

> Do you have a problem understanding this C fragment?

>

> typedef struct

> {

> char Hour ;

> char Year ;

> char Month ;

> char Day ;

> int Count ;

> } DATA_POINT ;

>

> DATA_POINT SavedData[ 100 ] ;

> ...

> SavedData[ i ].Hour = ThisHour ;

> SavedData[ i ].Year = ThisYear ;

> SavedData[ i ].Month = ThisMonth ;

> SavedData[ i ].Day = ThisDay ;

> SavedData[ i++ ].Count = ThisCount ;

> ...

I'm pretty sure I understand it.

> Do you have any problem understanding this diagram?

>

> +------------+

> | Hour[ 1 ] |

> +------------+

> | Year[ 1 ] |

> +------------+

> | Month[ 1 ] |

> +------------+

> | Day[ 1 ] |

> +------------+----------+

> | Count[ 1 ] |

> +------------+----------+

> | Hour[ 2 ] |

> +------------+

> | Year[ 2 ] |

> +------------+

> | Month[ 2 ] |

> +------------+

> | Day[ 2 ] |

> +------------+----------+

> | Count[ 2 ] |

> +-----------------------+

> ...

Not completely - is this just showing how the data is arranged?

> I think that you will find things a little easier to understand and program

> if you re-arrange the item in the entry like this:

>

> typedef struct

> {

> char Year ;

> char Month ;

> char Day ;

> char Hour ;

> int Count ;

> } DATA_POINT ;

>

> This has the data in a logical order like the digits of a number. The most

> rapidly changing variables are at the end, and the least rapidly changing

> variables are at the beginning.

That does make a bit more sense. However, I want to drop Hour:

typedef struct

{

char Year ;

char Month ;

char Day ;

int Count ;

} DATA_POINT datalog[datasize];

I would get the current time, and use the hour value to locate my record:

datalog[hour].Count = datalog[hour].Count + accumulated_pulses;

The array/struct is sent via serial port to a serial->ethernet bridge (over 802.11b, existing implementation), and a computer will be taking the data in and determining when the counts occurred.

Did I confuse the issue further?

Posted: Thu Nov 10, 2005 2:36 am

As I understand it, you are accumulating a count, and every hour you want to record the count.

As you say you will wrap around on the hour array, I gather that you expect to be able to upload the array to the PC at least once per day.

It seems to me that this will not deal with power failures too well.

The situations that I have in mind are:

The power fails for an hour or more starting at half past the hour. In this case the counts for at least two hours will be incorrect, but there is no provision for indicating this.

It seems to me that in this case using the hour as the index in a table that is 24 entries long will give you yesterday's data in this case without any indication.

For some reason the upload fails to be recorded on some day.

In this case you would just miss data.

You haven't made clear exactly what the count is recording, and how you want to handle these cases.

The reason that I chose the format where the array index was just the serial number of the entry, was that this would make it clear when observations were missing, and by changing the recording interval you could either fill up the EEPROM with several days data or get finer resolution around a power failure.

If this takes up too much space in EEPROM, you could pack the entries into an int taking full advantage of the limited range of the numbers.

Alternatively, you could treat the EEPROM as a log with different kinds of entries.

For example a few times a day you could enter the date time and the interval between count records. The normal entry would be just the count. These would be the majority of the entries. When the microcontroller resets after a power failure, it could add a "reset" entry followed by the date time entry.

Instead of arranging for the microcontroller to know the date and time, you could just record the relative time since the last reset, modulo some suitable period, like a week. The PC program that processes the uploading can calculate the date of the entries by using the date & time of the upload as the date & time of the latest entry in the table, and calculating the date & time of the earlier entries backward form the last entry.

Could you write something more about what you mean by "visualizing the data"?

A note below.

Hope this helps.

[snip]

> > Do you have any problem understanding this diagram?

> >

> > +------------+

> > | Hour[ 1 ] |

> > +------------+

> > | Year[ 1 ] |

> > +------------+

> > | Month[ 1 ] |

> > +------------+

> > | Day[ 1 ] |

> > +------------+----------+

> > | Count[ 1 ] |

> > +------------+----------+

> > | Hour[ 2 ] |

> > +------------+

> > | Year[ 2 ] |

> > +------------+

> > | Month[ 2 ] |

> > +------------+

> > | Day[ 2 ] |

> > +------------+----------+

> > | Count[ 2 ] |

> > +-----------------------+

> > ...

>

>Not completely - is this just showing how the data is arranged?

Yes, I was trying to understand what you meant about having trouble "visualize the data"

Posted: Wed Nov 16, 2005 4:38 am

> As I understand it, you are accumulating a count, and every hour you want

> to record the count.

Yes - and no.

It will record the counts every so often - minute, etc...:

Lets say for 11/20/2005, 13:00, 25 pulses were recorded

I could still record the data every 15 minutes:

11/20/2005, 13:00, 2 pulses were recorded

11/20/2005, 13:00, 5 pulses were recorded

11/20/2005, 13:00, 20 pulses were recorded

11/20/2005, 13:00, 25 pulses were recorded

11/20/2005, 14:00, 1 pulses were recorded

On a power failure, it will run through an accumulation of the last count routine, and save the data to EEPROM.

So the resolution is essentially 1 hours worth of data, but it is accumulated every 15 minutes or minute.

> As you say you will wrap around on the hour array, I gather that you expect

> to be able to upload the array to the PC at least once per day.

Actually, it will attempt to send data to the PC every 15 minutes, with the hope that the PC will be running 24/7. If the PC crashes, it would be very good to have more than 24 hours worth of data, in case the PC is not monitored exactly as I'm expecting it to be.

> It seems to me that this will not deal with power failures too well.

>

> The situations that I have in mind are:

>

> The power fails for an hour or more starting at half past the hour. In

> this case the counts for at least two hours will be incorrect, but there is

> no provision for indicating this.

If the power fails, there is nothing to count. The last accumulation would be added up, and then the table will be written to EEPROM.

> It seems to me that in this case using the hour as the index in a table that is 24

> entries long will give you yesterday's data in this case without any indication.

No - the table will also have the day and month in it, and the computer can infer which row was earlier. However, I am rethinking this, as I may have to provide other information as well, which will require the table to be larger than I planned.

> For some reason the upload fails to be recorded on some day.

>

> In this case you would just miss data.

This would be a problem, which is one thing I did not consider. Technically, the computer will always be on -> however, if it gets shut off, then there will be some trouble. I'll have to think about this problem a bit, thought it really won't have an impact on what I am doing. Increasing the table to 72 hours would address this situation.

> You haven't made clear exactly what the count is recording, and how you

> want to handle these cases.

The count is recording the pulse of a magnet passing by a hall sensor, and time stamping it (using a DS1305).

> The reason that I chose the format where the array index was just the serial

> number of the entry, was that this would make it clear when observations

> were missing, and by changing the recording interval you could either fill up the

> EEPROM with several days data or get finer resolution around a power failure.

I think I'll switch my thinking around to what you say here (ie, use the serial number of the entry). However, once a power failure hits, there will be no pulses (as they are tied to power consumption).

> Alternatively, you could treat the EEPROM as a log with different kinds of entries.

Hmm, right now the last 24-48 hours would be the most important data, 72 would give a bit of room for problems (if there is room in the EEPROM, I believe there is, haven't done the math). So, in a power failure, it writes the last 72 hours worth (72 rows worth) of data to the EEPROM.

> Instead of arranging for the microcontroller to know the date and time, you

> could just record the relative time since the last reset, modulo some

> suitable period, like a week. The PC program that processes the uploading

> can calculate the date of the entries by using the date & time of the

> upload as the date & time of the latest entry in the table, and calculating

> the date & time of the earlier entries backward form the last entry.

Hmmm. At this point it's too late to try this, but I will give it some thought. I never liked the idea of the microcontroller knowing the date, as that means the microcontroller has to have the date set physically (keypad/LCD) or via serial -> I only have data coming out, no provision for data coming in via serial port. In a power failure, the DS1305 has enough power for 24 hours to keep the date and time, which can be increased.

> Could you write something more about what you mean by "visualizing the data"?

I was thinking of a map of the EEPROM, kinda like the one on page 19 of TechNote AN2400:

http://www.freescale.com/files/microcontrollers/doc/app_note/AN2400.pdf

The graphic helps me understand what is being done inside the chip. (I believe it's for 'Flash EEPROM', not 'EEPROM')

It's like this: I tend to be a bit more visual in my thinking. So, I like to use Mathematica over matlab/octave, CAD programs are nice but I work just a bit quicker in Adobe Illustrator, I like GUI's rather than Unix terminal(s). A TI-89 was a big step up from my TI-83 as the 89 had "Pretty Print" (it shows equations and fractions as you'd write them on paper).

So, with memory, I read "bits, bytes, word, block" and have trouble imagining the structure and how the data is laid out and addressed. When someone says, "You can write from this address to that address" I'd like to be able to visually think of the data. I can't, and at the very most, have a serious amount of trouble. Or, when the instructions say, "You need to erase a sector from A to B", I don't know what I'm erasing, even if the code is clearly laid out on how to do the erasure.

Posted: Sun Nov 6, 2005 7:52 am

To store data so it is not lost during loss of power, you will need to write the data into the internal EEPROM of the S12DP256B device. Writing to the EEPROM is not done by simply writing to its memory locations, but is rather done by a special series of actions that finally writes the data in the EEPROM. This write action is slower that regular writes, so you would probably execute it only when you really need to. Also the EEPROM has a limited lifetime of typically 10,000 - 100,000 cycles of erase/program. This means that you don't want to write to a specific EEPROM location more than 10,000 as otherwise, it may wear off, and fail to store your data reliably. This also means you will need to write to the EEPROM only occasionally, or every several minutes to create a backup data (so if your system looses power, the data since the last backup that was stored in RAM is lost, but the last backup data from the EEPROM can be recovered).

You can read more information on how to write to the EEPROM on the S12DP256B part, in the following Freescale document:

http://www.freescale.com/files/microcontrollers/doc/ref_manual/S12EETS4KV2.pdf

Posted: Wed Nov 9, 2005 5:04 pm

> To store data so it is not lost during loss of power, you will need to

> write the data into the internal EEPROM of the S12DP256B device.

> Writing to the EEPROM is not done by simply writing to its memory

> locations, but is rather done by a special series of actions that finally

> writes the data in the EEPROM. This write action is slower that regular

> writes, so you would probably execute it only when you really need to. Also

> the EEPROM has a limited lifetime of typically 10,000 - 100,000 cycles of

> erase/program. This means that you don't want to write to a specific EEPROM

> location more than 10,000 as otherwise, it may wear off, and fail to store

> your data reliably. This also means you will need to write to the EEPROM

> only occasionally, or every several minutes to create a backup data (so if

> your system looses power, the data since the last backup that was stored in

> RAM is lost, but the last backup data from the EEPROM can be recovered).

I plan (tentatively!) to use a backup battery I'm working on to trigger a routine that will write the data, then stop the program. The backup battery (maxim sells several power management chips) will only need to power the chip for 20 minutes after power is out. However, I'm first trying to figure out writing to the EEPROM, which I will give App note AN2400/D and the link you posted a better read through.

Message Edited by khumphri on 01-27-2006 08:36 AM

Programming help needed, storing data

Programming help needed, storing data

General