zdbfs crashes when the underlying disk goes out of space

[OmarElawady]

last logs from the qsfs container:

[     137.151184][-] data write: Quota exceeded                                                                                                                                                            
[-] cache flush: blockid mismatch                                                                                                                                                                          
[-] fatal: segmentation fault                                                                                                                                                                              
[+] blocks cache size: 4096 KB                                                                                                                                                                             
[+] fuse: ready, waiting events: /mnt                                                                                                                                                                      
[-] zdb: zdbfs-data: set: error: Cannot write data right now                                                                                                                                               
0x4058c7: (zdbfs_system_sighandler+0x37)                                                                                                                                                                   
0x43e8af: (sigprocmask+0x21)                                                                                                                                                                               

This happen on a zos node, testing locally with a limit disk size doesn't crash zdbfs.

[maxux]

Which version do you use ? It should reoport Input/output error in this case and not crash.
Which files were you writing at that time ? Can you reproduce ?

I can't reproduce the crash but I found lot of memory issue regarding cache when zdb is out of space. I'm investigating.

Locally I try by mounting a small tmpfs and put zdb on, which will quickly reach No space left on device :)

[OmarElawady]

zdbfs version is 0.1.10. I couldn't reproduce locally (created a limited disk, and it failed with an io error but didn't crash).

But it seems to happen consistently on zos nodes. This is what I was trying when it crashed: threefoldtech/quantum-storage#19

[OmarElawady]

zdbfs gdb's stack trace:

Program received signal SIGSEGV, Segmentation fault.
sdscatlen (s=0x7f329eea0f39 "*3\r\n$3\r\nSET\r\n$8\r\n\341\017", t=0x0, len=len@entry=131072) at sds.c:383
383     sds.c: No such file or directory.
(gdb) bt                                                                                                                                                                                                  
#0  sdscatlen (s=0x7f329eea0f39 "*3\r\n$3\r\nSET\r\n$8\r\n\341\017", t=0x0, len=len@entry=131072) at sds.c:383
#1  0x00000000004290e9 in redisFormatSdsCommandArgv (target=target@entry=0x7fffb30f1fd0, argc=3, argv=0x7fffb30f2040, argvlen=0x7fffb30f2060) at hiredis.c:605
#2  0x0000000000429f0c in redisAppendCommandArgv (c=c@entry=0x7f329f240960, argc=<optimized out>, argv=<optimized out>, argvlen=<optimized out>) at hiredis.c:1118
#3  0x000000000042a059 in redisCommandArgv (c=0x7f329f240960, argc=<optimized out>, argv=<optimized out>, argvlen=<optimized out>) at hiredis.c:1170
#4  0x000000000040653a in zdb_set (remote=0x7f329f219570, id=id@entry=4065, buffer=<optimized out>, length=<optimized out>) at zdb.c:384
#5  0x0000000000401688 in zdbfs_cache_block_release (fs=0x7fffb30f2390, cache=0x7f329f219b60) at cache.c:555
#6  0x0000000000402222 in zdbfs_cache_block_release (cache=<optimized out>, fs=<optimized out>) at cache.c:635
#7  0x0000000000406f2f in zdbfs_fuse_release (req=0x7f329f240190, ino=<optimized out>, fi=<optimized out>) at zdbfs.c:1022
#8  0x000000000041d286 in do_release (req=<optimized out>, nodeid=<optimized out>, inarg=<optimized out>) at ../lib/fuse_lowlevel.c:1455
#9  0x000000000041f443 in fuse_session_process_buf_int (se=se@entry=0x7f329f219c20, buf=buf@entry=0x7fffb30f22d0, ch=ch@entry=0x0) at ../lib/fuse_lowlevel.c:2666
#10 0x000000000041fad7 in fuse_session_process_buf (se=se@entry=0x7f329f219c20, buf=buf@entry=0x7fffb30f22d0) at ../lib/fuse_lowlevel.c:2538
#11 0x00000000004096e6 in zdbfs_fuse_session_loop (se=se@entry=0x7f329f219c20, fs=fs@entry=0x7fffb30f2390, timeout=timeout@entry=1000) at zdbfs.c:1248
#12 0x00000000004011c3 in main (argc=<optimized out>, argv=<optimized out>) at zdbfs.c:1337