Can you confirm if SDL_VIDEO_DRIVER_X11 is defined.
If so, X11 client need to communicate with X server by some address. Can you try following:
export DISPLAY=:0
then run your sdl application again.
You can find your X server address by
ps -ef | grep X
On my board, the result is
root 5295 5195 0 Sep24 tty7 00:00:24 /usr/bin/X :0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
root 7074 5503 0 09:43 ttymxc1 00:00:00 grep --color=auto X