Hello,
How the A53 core is the principal core, the A53 is the one that is going to use. Probably the problem could be because you have a multi-master conflict between your A53 core and the other device that you have. Try applying the patch and see if that solves the problem.
Furthermore, can you please pass me your modified device tree. Because disabling the I2C bus could be another solution, but apparently is not being done in your case.
Best regards,
Diego.