Hi Marti,
Couple of things.
1) You don't want to turn on the timer system in you main for loop, once should be sufficient.
2) You should re-enable interrupts before you calculate frequency, just after you save the globals to locals. The idea is to have interrupts disabled for a short a time as possible. You should also save second and first locally.
3) Even though you have declared time as a float, countend, countstart, second and first are unsigned ints. You will discover that C will calculate the line
time = ((65536*(countend-countstart))+second) - first; /* Calculate total count */
as an unsigned int, and then promote the result to float. This will probably result in truncation. Read you C book for how it handles promotion of types.
4) In you code
interrupt void tim0_overflow_isr(void)
{
if(a == TRUE){
TFLG2 = 0x80; /* Clear overflow flag ch5 */
....
If 'a' is not true, TFLG2 will never be cleared, and the interrupt will never be cleared, and the interrupt routine will keep being called for ever and ever. You must clear the flag allways.
5) Your code to check for interrupt priority is probably not going to work. Firstly the example I gave when TC5 was zero, was only one case. It may be, depending on other interrupts in you system, and how long you disable interrupts for, that TC5 may be three or four. I generally check the other interrupts flags to see if we have a priority problem.
I generally do this by checking if the value of TC5 was between 0 and 0x7fff (simple branch on negative will work in assembler), and if so, then check to see the Timer Over Flag is set. If the Timer Over Flow flag is set, then timer overlow interrupt has not executed, so I increment the timer overflow count (in the TC5 interrupt) and clear the Timer OverFlow flag.
6) If you are measuring 50 Hz, you really don't need two interrupts. One interrupt set for either rising or falling edge will work fine. You need to measure the time between two rising edges, or two falling edges, not the time between rising and falling edges, if you want to measure the frequency. If you measure between a rising and falling edge, the frequency will change with the duty cycle.
Don't give up. Once you get your head around the interrupts its pretty easy to measure the frequency with and HC12.
Cheers
David