Hi,
We try to integrate a custom wayland compositor on the i.MX8M and having some issues with the gbm buffer handling.
The library we use manually creates a gbm bo with either gbm_bo_create or gbm_bo_create_with_modifiers depending on if modifiers are present on the format previously selected. In both cases the fd is read with gbm_bo_get_fd. The buffer is afterwards (re-)imported with either gbo_bo_import(gbm_device, GBM_BO_IMPORT_FD, gbm_import_fd_data, SCANOUT) or gbm_bo_import(gbm_device, GBM_BO_IMPORT_FD_MODIFIER, gbm_import_fd_modifier_data, SCANOUT) depending on if the buffer has modifiers set other than linear/invalid. This export/import is done to support multi output/render devices.
Default the AR24 format with modifier Vivante_super_tiled is used. This will create a buffer with modifiers, where the import through gbm_bo_import will fail with an error of ENOENT.
As the libgbm is closed source I could not have look what happens there, but a run an strace and found that the ioclt DRM_IOCTL_PRIME_FD_TO_HANDLE is called in the kernel. I tried to follow the calls in the kernel and it seems that the call chain is:
DRM_IOCTL_PRIME_FD_TO_HANDLE -> drivers/gpu/drm/drm_prime.c:drm_prime_fd_to_handle_ioctl -> driver.prime_fd_to_handle -> drivers/gpu/drm/imx/imx-drm-core.c.prime_fd_to_handle -> drivers/gpu/drm/drm_prime.c:drm_gem_prime_handle_to_fd -> drm_gem_object_lookup -> -ENOENT
So it seems that the lookup of the handle fails with -ENOENT.
When using Linear or Invalid the buffer can be imported successfully.
I am a bit out of options, so maybe someone could help us and tell us what we are doing wrong.
Thanks in advance,
Br,
Christian
