Have a look at 'Figure 9-2. Port Interrupt Block Diagram" in the Reference Manual. That explains how it is wired.
This is a KEYBOARD input port, so it is only meant for detecting single edges on one of the input pins at a time.
To do what you want there are two choices - flip the polarity like you're doing or disable that pin with the KBIPEn bit. The problem there is that you probably want to re-enable the interrupt when the pin changes state again, and the easiest way to do that is to do what you're doing - flip the edge.
If you get two pins changing close together you'll have to have some pretty smart code so it doesn't lock up or lose interrupts.
If the pins aren't changing all that often (and if you don't need microsecond-level timing) it might be simpler to just poll the pins from a timer interrupt or from your main event loop and detect changes in software.
If you do want fully independent interrupt capability and nanosecond-level timing you're better off using the TPM to capture the input times and generate interrupts.
Tom