How are you interfacing to your target? I use an in-house designed MON08 Interface and it generates the required Vtst, MOD0, MOD1 and DIV4 signals. If the Vtst is on the IRQ pin(1), PTB1 (MOD0) pin(20) is tied to Vdd with a 10K Resistor, PTB2 (MOD1) pin (18) is tied to Ground with a 10K resistor, and PTB3 (DIV4) Pin(17) is tied either high or low with a 10K resistor depending on what your clock is. If you use a 9.8304 MHz clock source and PTB3 is high your Baud rate will be 9600 Baud. If PTB3 is Low your Baud Rate will be 19200. If you use a 4.9125MHz and PTB3 is LOW you get a 9600 Baud rate.
Double check your voltage levels on the required MON08 pins. Make sure your Vtst is the correct voltage also. Vtst according to the datasheet should be between 7.5V and 8.5V with a 5V Vdd supply.
I hope this helps.
Brian
