Hi Lorenzo,
Connecting the debugger can work in my side, as follows:

For gdb-multiarch, customer need to "set architecture aarch64" firstly,then "target remote /dev/ttyUSBx".

Another method:
If you want to extending the terminals, they can use agent-proxy. The steps is as follows:

in another terminal:

Then in the third terminal, executing "gdb-multiarch vmlinux" or "aarch64-poky-linux-gdb vmlinux" to connect target.

Note the last picture, "target remote localhost:5551" when connecting target.
Best Regards,
Israel H.