I have tried the same with SH16 powered from 3.3V. It works. The first visible difference is that when SH16 is powered blue LED in USBDM interface doesn't blink. Only yellow lights up. OK, when I program SOPT1 and security, PROGHCS08 requires power down but then starts normally asking for algorithm.
SH8 powered from 5V on another board behaves different. After powering blue LED gets off for circa a second then both LEDs light up. But there is no communication.
I have tried newer version of USB BDM with newer programming software. One time I have managed to get a normal start of program but there was a refusal to load programming algorithm.
The command I send is :
LDA #%00000100
STA SOPT1 (SOPT1 is located at $1802)
I need to change a bit software in existing boards but I completely cannot enter to programming procedures. You are right, I also have never encoutered such a phenomenon. If chip is secured then it asks for erasure, OK, but then I can normally reprogram it.
In QY4 there was an internal error in MCU, I had the same. I have observed very strange activity on IRQ pin when there was attend to enter programming procedure. I was putting 10nF there to cut these oscillations and it was helping.
regards
J