Hello Stephen,
If the clock parent is the same for A53 and M4, then by modifying the clock speed on Linux, will also modify at M4 side, for example, if both domains are fed by SYSTEM_PLL1, by changing the PLL1 speed, changes all the clock tree tied to this source.
There are 2 methods to change the clock speed of one domain without changing the other side:
- Each domain uses a different clock source
- Change the speed by selecting a different clock divider, instead of modifying the PLL parameters, for example.
If not, how would you reliably synchronize the two clocks to run in step with each other?
To syncronize the clocks, the application may select the same clock source for both cores, for example in the table below, the ARM_A53_CLK_ROOT may derive from SYSTEM_PLL1_CLK and ARM_M4_CLK_ROOT may derive from SYSTEM_PLL1_DIV3. In this situation, the speed for M4 is 1/3 of A53 and they will be in synchronization.

Just pay attention for the maximum frequency speed for each domains.
Regards