Hmmmmmmmm....That is odd.
I have not encountered this type of behavior. Without eyeballing everything it's difficult to point somewhere!
I would recommend going back to the start, assume nothing, review the HW and pin-out connectivity to where you are measuring. Then move on to review IOMUX settings to ensure there are no errors - and finally in the kernel init.c make sure you all is in order. Typically I've found that something simple is staring you in the face while you look hard for deeper complexities.
mike