Hi,
The problem is the one I said above: the graphics libraries version do not match the kernel driver since you keep the same rootfs with 2 different kernels.
I'm pretty sure your xorg.conf is configured to use vivante which is the 3D/2D GPU for rendering, so it fails.
Also, yes the frame buffer works with both, try that:
$ cat /dev/urandom > /dev/fb0
If it displays something it proves the framebuffer works, but doesn't mean your X setup will.
You can try modifying xorg.conf not to use the GPU but I'm not sure of the result.
Regards,
Gary