Making it an ICD

[twaik]

Hello.
I tried to get rid of all the vulkan-layer stuff used here and make it plain ICD.
But there are some problems.
It looks like vulkaninfo exits too early and does not finish to print info.
vkcube stucks on the last frame, it seems like flip_thread is finished too early and does not post the last semaphore signal.
Stacktrace of hung

#0  0x0000007ff5e8f0dc in syscall () from /apex/com.android.runtime/lib64/bionic/libc.so
#1  0x0000007ff5e938bc in __futex_wait_ex(void volatile*, bool, int, bool, timespec const*) () from /apex/com.android.runtime/lib64/bionic/libc.so
#2  0x0000007ff5ef6998 in pthread_cond_wait () from /apex/com.android.runtime/lib64/bionic/libc.so
#3  0x0000007f752e27bc in util::timed_semaphore::wait (this=0x7ff5925734, timeout=18446744073709551615) at /home/twaik/.termux-build/sysvk-x11/src/util/timed_semaphore.cpp:108
#4  0x0000007f752e69b0 in wsi::swapchain_base::wait_for_free_buffer (this=this@entry=0x7ff5925500, timeout=137) at /home/twaik/.termux-build/sysvk-x11/src/wsi/swapchain_base.cpp:639
#5  0x0000007f752e5f0c in wsi::swapchain_base::wait_for_pending_buffers (this=this@entry=0x7ff5925500) at /home/twaik/.termux-build/sysvk-x11/src/wsi/swapchain_base.cpp:608
#6  0x0000007f752e665c in wsi::swapchain_base::teardown (this=0x7ff5925500) at /home/twaik/.termux-build/sysvk-x11/src/wsi/swapchain_base.cpp:317
#7  0x0000007f752eae14 in wsi::x11::swapchain::~swapchain (this=0x7ff5925500) at /home/twaik/.termux-build/sysvk-x11/src/wsi/x11/swapchain.cpp:178
#8  0x0000007f752e2530 in util::allocator::destroy<wsi::swapchain_base> (this=<optimized out>, num_objects=1, objects=0x7ff5925500) at /home/twaik/.termux-build/sysvk-x11/src/util/custom_allocator.hpp:252
#9  0x0000007f752e8208 in wsi::destroy_surface_swapchain (swapchain=swapchain@entry=0x7ff5925500, dev_data=..., pAllocator=<optimized out>, pAllocator@entry=0x0) at /home/twaik/.termux-build/sysvk-x11/src/wsi/wsi_factory.cpp:228
#10 0x0000007f752e11b0 in wsi_layer_vkDestroySwapchainKHR (device=<optimized out>, swapc=0x7ff5925500, pAllocator=0x0) at /home/twaik/.termux-build/sysvk-x11/src/layer/swapchain_api.cpp:86
#11 0x000000555556d32c in ?? ()
#12 0x0000007ff5e8b19c in __libc_init () from /apex/com.android.runtime/lib64/bionic/libc.so

Changes:

1. CmakeLists is modified to build libsysvk_x11.so and install icd's json to needed place.
2. Moved stuff from sysvk.c to this project.
3. vkGetInstanceProcAddr is extracted only the first time vk_icdGetInstanceProcAddr is invoked.
4. vkGetInstanceProcAddr can not extract vkGetDeviceProcAddr and vkCreateDevice if instance argument is NULL, so they are extracted when instance is available in vk_icdGetInstanceProcAddr.
5. Implemented vkEnumerateInstanceExtensionProperties vkEnumerateDeviceExtensionProperties with injecting of needed extensions.
6. In the original project identifying device by it's queue is done using extracting dispatchable object which is only possible with using some Loader's stuff for layers. Currently I am extracting all possible queues pointers and let code use it as key for device (in unordered_map).
7. Removed invoking CreateXcbSurfaceKHR since it mostly was a noop with allocating VkIcdSurfaceXcb.

Thank you.

[xMeM]

* thread #1, name = 'vulkaninfo', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x0)                                             
* frame #0: 0x0000000000000000
frame #1: 0x0000007f745cf268 libsysvk_x11.so`wsi_layer_vkDestroySurfaceKHR(instance=0xb400007ff48e1650, surface=0xb400007ff4af0258, pAllocator=0x0000000000000000) at surface_api.cpp:168:4                                               
frame #2: 0x0000007f746a641c libvulkan.so.1.3.289`___lldb_unnamed_symbol1057 + 144                                                                          
frame #3: 0x0000005555605670 vulkaninfo`___lldb_unnamed_symbol690 + 2660
frame #4: 0x0000007ff56c55fc libc.so`__libc_init + 112                   
(lldb)  f 1
frame #1: 0x0000007f745cf268 libsysvk_x11.so`wsi_layer_vkDestroySurfaceKHR(instance=0xb400007ff48e1650, surface=0xb400007ff4af0258, pAllocator=0x0000000000000000) at surface_api.cpp:168:4                                                  
   165  {
   166     auto &instance_data = layer::instance_private_data::get(instance);    
   167
-> 168     instance_data.disp.DestroySurfaceKHR(instance, surface, pAllocator);
   169                                                                           
   170     instance_data.remove_surface(
   171        surface, util::allocator{ instance_data.get_allocator(), VK_SYSTEM_ALLOCATION_SCOPE_OBJECT, pAllocator });

There is no need to pass the DestroySurfaceKHR call to the proprietary driver.

[twaik]

How did you get detailed backtrace?

[twaik]

It still hangs in vkcube after trying to close window. Maybe there is something else?

[xMeM]

How did you get detailed backtrace?

DISPLAY=:0 VK_ICD_FILENAMES=~/sysvk_icd/build/sysvk_icd.json lldb vulkaninfo

[xMeM]

$ DISPLAY=:0 VK_ICD_FILENAMES=~/sysvk_icd/build/sysvk_icd.json vulkaninfo
(/data/data/com.termux/files/home/sysvk_icd/layer/layer.cpp:196): Failed to create the instance
WARNING: [Loader Message] Code 0 : terminator_CreateInstance: Received return code -7 from call to vkCreateInstance in ICD /data/data/com.termux/files/home/sysvk_icd/build/./libsysvk_x11.so. Skipping this driver.

Extensions implemented by the WSI layer need to be excluded in the ppEnabledExtensionNames passed to the proprietary driver.
VK_KHR_surface and VK_KHR_swapchain.

[twaik]

How did you get detailed backtrace?

DISPLAY=:0 VK_ICD_FILENAMES=~/sysvk_icd/build/sysvk_icd.json lldb vulkaninfo

You build it on device, right?

WARNING: [Loader Message] Code 0 : terminator_CreateInstance: Received return code -7 from call to vkCreateInstance in ICD /data/data/com.termux/files/home/sysvk_icd/build/./libsysvk_x11.so. Skipping this driver.

Do you use any other layers?

[twaik]

I removed it but it does not seem to help. I still see this backtrace after sending SIGINT to hung process.

0x0000007ff39570dc in syscall () from /apex/com.android.runtime/lib64/bionic/libc.so
(gdb) bt
#0  0x0000007ff39570dc in syscall () from /apex/com.android.runtime/lib64/bionic/libc.so
#1  0x0000007ff395b8bc in __futex_wait_ex(void volatile*, bool, int, bool, timespec const*) () from /apex/com.android.runtime/lib64/bionic/libc.so
#2  0x0000007ff39be998 in pthread_cond_wait () from /apex/com.android.runtime/lib64/bionic/libc.so
#3  0x0000007f72d1e774 in util::timed_semaphore::wait (this=0x7ff3325734, timeout=18446744073709551615) at /home/twaik/.termux-build/sysvk-x11/src/util/timed_semaphore.cpp:108
#4  0x0000007f72d22968 in wsi::swapchain_base::wait_for_free_buffer (this=this@entry=0x7ff3325500, timeout=137) at /home/twaik/.termux-build/sysvk-x11/src/wsi/swapchain_base.cpp:639
#5  0x0000007f72d21ec4 in wsi::swapchain_base::wait_for_pending_buffers (this=this@entry=0x7ff3325500) at /home/twaik/.termux-build/sysvk-x11/src/wsi/swapchain_base.cpp:608
#6  0x0000007f72d22614 in wsi::swapchain_base::teardown (this=0x7ff3325500) at /home/twaik/.termux-build/sysvk-x11/src/wsi/swapchain_base.cpp:317
#7  0x0000007f72d26dcc in wsi::x11::swapchain::~swapchain (this=0x7ff3325500) at /home/twaik/.termux-build/sysvk-x11/src/wsi/x11/swapchain.cpp:178
#8  0x0000007f72d1e4e8 in util::allocator::destroy<wsi::swapchain_base> (this=<optimized out>, num_objects=1, objects=0x7ff3325500) at /home/twaik/.termux-build/sysvk-x11/src/util/custom_allocator.hpp:252
#9  0x0000007f72d241c0 in wsi::destroy_surface_swapchain (swapchain=swapchain@entry=0x7ff3325500, dev_data=..., pAllocator=<optimized out>, pAllocator@entry=0x0) at /home/twaik/.termux-build/sysvk-x11/src/wsi/wsi_factory.cpp:228
#10 0x0000007f72d1d168 in wsi_layer_vkDestroySwapchainKHR (device=<optimized out>, swapc=0x7ff3325500, pAllocator=0x0) at /home/twaik/.termux-build/sysvk-x11/src/layer/swapchain_api.cpp:86
#11 0x000000555556d32c in ?? ()
#12 0x0000007ff395319c in __libc_init () from /apex/com.android.runtime/lib64/bionic/libc.so

[xMeM]

You build it on device, right?

Yes.

Do you use any other layers?

I installed vulkan-validation-layers,which is disabled by default.

fix-layer.patch.txt
I need this patch to work.
fix-deadlock.patch.txt
This patch fixes deadlock.

[twaik]

AFAIK you have write access to my fork. So you can push there commits.
I can add these patches later.

[twaik]

One more question. What about rebasing to latest commit of original project? Issue is already opened.

[xMeM]

One more question. What about rebasing to latest commit of original project? Issue is already opened.

Okay, I'll rebase to the latest commit when I have time.