MaxY,
I tried this AN2880SW and found two bugs and something surprising. Results:
1) First of all there are at least two bugs in the code.
1.1) First problem (a bug) is in main.c at line 50:
const unsigned int keys [] = {0x1111, 0x2222, 0x3333, 0x4444}; /* Backdoor keys */
This looks nice but const in this CW project means flash. Ant DoOnStack routine can't read these values from flash because at the time flash is "disconnected" (KEYACC is set). I tried to step through code and found that my D64 (I modified just PRM file) returns zeros from keys[] when stepping through DoOnStack.
1.2) 2nd bug is in AN2720.asm file:
;; 130: *(unsigned int *)(0xBF00) = keys[0]; /* Write key 1 */
LDX keys
STX $BF00
;; 131: *(unsigned int *)(0xBF02) = keys[1]; /* Write key 2 */
LDY keys + 1
STY $BF02
;; 132: *(unsigned int *)(0xBF04) = keys[2]; /* Write key 3 */
LDD keys + 2
STD $BF04
;; 133: *(unsigned int *)(0xBF06) = keys[3]; /* Write key 4 */
LDD keys + 3
STD $BF06
Keys is array of ints, not chars so there must be keys+2 instead of keys+1, keys+4 instead of keys+2 etc:
;; 130: *(unsigned int *)(0xBF00) = keys[0]; /* Write key 1 */
LDX keys
STX $BF00
;; 131: *(unsigned int *)(0xBF02) = keys[1]; /* Write key 2 */
LDY keys + 2
STY $BF02
;; 132: *(unsigned int *)(0xBF04) = keys[2]; /* Write key 3 */
LDD keys + 4
STD $BF04
;; 133: *(unsigned int *)(0xBF06) = keys[3]; /* Write key 4 */
LDD keys + 6
STD $BF06
2) What's interesting - unmodified code (except PRM) was working on D64! After I moved keys[] from flash to RAM program started to say "Cannot unsecure". And after second fix it started to work again.
It seems that D64 2L86D and 0M89C masksets do accept zeros as backdoor keys!
I tried the same with DG128, it refused zeros as backdoor keys. Maybe at the time of writing this appnote author had buggy Dx256 that was also accepting zeros as backdoor keys?