<jemalloc>: Error in munmap(): Cannot allocate memory #1871
Description
Hello,
I am trying to run cactus
$DOCKER run --rm=true \
--memory=${MEM}g \
-v /home/exacloud/gscratch:/home/exacloud/gscratch \
-w ${PWD} \
quay.io/comparative-genomics-toolkit/cactus:v3.1.2 \
cactus-pangenome \
./js \
--mgCores 23\
--mgMemory 650G \
$GENOME_FILE \
--outDir $OUTPUT \
--outName $OUTPUT \
--reference $REF_NAME \
--vcf \
--giraffe \
--gfa \
--gbz
After ~24H of running, it logged a lot of these errors:
RuntimeError: Command /usr/bin/time -f "CACTUS-LOGGED-MEMORY-IN-KB: %M" bash -c 'set -eo pipefail && vg deconstruct /tmp/toilwf-9eb2ee1719df55e8bdfe7990fe12165e/9da7/job/tmpoihrgfqu/rm-pg.NC_133406.1.clip.vg -P T2T-MMU8v2 -C -a -t 31 | bgzip --threads 31' exited 139: stderr=vg [warning]: System's vm.overcommit_memory setting is 2 (never overcommit). vg does not work well under these conditions; you may appear to run out of memory with plenty of memory left. Attempting to unsafely reconfigure jemalloc to deal better with this situation.
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.70.0 "Zebedassi"
Caught signal 6 raised at address 0x23d333c; tracing with backward-cpp
Stack trace (most recent call last):
#18 Object "/home/cactus/bin/vg", at 0x6824b4, in
#17 Object "/home/cactus/bin/vg", at 0x238f606, in
#16 Object "/home/cactus/bin/vg", at 0x238dd69, in
#15 Object "/home/cactus/bin/vg", at 0x6830cb, in
#14 Object "/home/cactus/bin/vg", at 0xfb4ddb, in
#13 Object "/home/cactus/bin/vg", at 0xe49c0a, in
#12 Object "/home/cactus/bin/vg", at 0x13029f4, in
#11 Object "/home/cactus/bin/vg", at 0x159f4cc, in
#10 Object "/home/cactus/bin/vg", at 0x159e91b, in
#9 Object "/home/cactus/bin/vg", at 0x15a02a9, in
#8 Object "/home/cactus/bin/vg", at 0x22b6ad5, in
#7 Object "/home/cactus/bin/vg", at 0x63ea8f, in
#6 Object "/home/cactus/bin/vg", at 0x22c7b08, in
#5 Object "/home/cactus/bin/vg", at 0x22c79a6, in
#4 Object "/home/cactus/bin/vg", at 0x22c793b, in
#3 Object "/home/cactus/bin/vg", at 0x63cfe7, in
#2 Object "/home/cactus/bin/vg", at 0x63f72f, in
#1 Object "/home/cactus/bin/vg", at 0x23a67e5, in
#0 Object "/home/cactus/bin/vg", at 0x23d333c, in
and
<jemalloc>: Error in munmap(): Cannot allocate memory
<jemalloc>: Error in munmap(): Cannot allocate memory
Library locations:<jemalloc>: Error in munmap(): Cannot allocate memory
#
<jemalloc>: Error in munmap(): Cannot allocate memory
Please include this entire error log in your bug report! Object "━0 ━━
━ERROR: Signal 0
Object "━━0x6━
occurred. VG has crashed. ━━━ Object "━━
━", at Library locations:Visit ", at 0x22c7089━, in
━#0x5
Object "#https://github.com/vgteam/vg/issues/new/chooseERROR: Signal to report a bug.━", at ━━0x23615b8, in
0x23d333c
#", at ━0x23d333c0x6━━━, in ━ occurred. VG has crashed. ━━━━━━, in
0x4━ Object "0x2━
Object "Visit
Context dump:Library locations:
https://github.com/vgteam/vg/issues/new/choose━ to report a bug.━
━━━
Library locations: Thread ━━ERROR: Signal ━0━: ━
━━0x6Starting 'gbwt' subcommand occurred. VG has crashed.
", at ━Found ━", at ━Visit 0x63f72f━0x22c793b━ERROR: Signal ━
Context dump:0x1━https://github.com/vgteam/vg/issues/new/choose to report a bug., in
0x6, in ━━━━━━━Command exited with non-zero status 134
CACTUS-LOGGED-MEMORY-IN-KB: 59012052
The job is still active though (maybe other workers are completing?). Are there any debugging steps or information that would help diagnose what went wrong here? Thanks for any help.