Does the RX pin need to be pulled up or pulled down when not using? My guess for TX not producing a signal greater than 600mV peak to peak and having a nasty slope down (like a cap discharge) on the high of the "square wave" is because I left the RX pin floating. This is the only thing I did differently from my other sc16is750 which is also on the board and works fine. We are using the chip as just a transmitter.
Did you try disabling the Rx when not using it, writing a 1 to RXDISABLE to the Extra Features Control Register (ECFR),
if you did a pull-up resistor should work, around 10k.