Hello,
Monitor mode entry on those old HC08 parts was always troublesome.
1) You have the circuit in the datasheet.
1b) Keep all cables as short as possible
2) Hold the part reset with the reset button on ~RST, hold it down with the power off.
3) Apply power, keeping ~RST low.
4) Make sure that there is voltage Vtst on the ~IRQ pin. It should be VDD+2.5 volts to 9 volts. If you are running at 5.0 volts, Vtst should be at least 7.5 volts.
5) Now release ~RST and the part will enter monitor mode.
You are not using CodeWarrior, and so what program are you using to talk to the MCU over the serial port?
Good luck
Mark