I found this issue with my sabre-sdb board for all kernels.
I ported my patches from 5.4 boundary devices kernel
https://github.com/boundarydevices/linux/commit/531de831b3c465f2b8602f26cbdef25afe47af5d
https://github.com/boundarydevices/linux/commit/d1c959481f4a6381996cd370c3ca214c62a307f5
https://github.com/boundarydevices/linux/commit/6709ca50fcd66c3ac5dfe4d19e8423815c0552f3
which fixes this issue.
I found this issue with my sabre-sdb board for all kernels.
I ported my patches from 5.4 boundary devices kernel
https://github.com/boundarydevices/linux/commit/531de831b3c465f2b8602f26cbdef25afe47af5d
https://github.com/boundarydevices/linux/commit/d1c959481f4a6381996cd370c3ca214c62a307f5
https://github.com/boundarydevices/linux/commit/6709ca50fcd66c3ac5dfe4d19e8423815c0552f3
which fixes this issue.