Bigint leads to index corruption

[patrikhermansson]

I'm doing an import of a fairly large dataset into Manticore with columnar. The table has several number fields and when these are set as type integer the import finishes without a hitch. However, if i change these to bigint the import crashes after a few minutes. Not instantly, but consistently after about 1M inserts and without any SQL errors but the log says:

malloc(): unsorted double linked list corrupted
Crash!!! Handling signal 6
malloc(): invalid size (unsorted)
Crash!!! Handling signal 11

The bigint's are negative and would overflow 32bit, so using integer to start with was a mistake from my side. I can then not continue the import and the table is corrupted. When I drop the table and recreate it I get the message:

Create schema index err: Error 1064 (42000): error adding table 'TABLENAME': directory is not empty: /var/lib/manticore/TABLENAME

Meaning I have to clear the whole manticore directory and start over.

[tomatolog]

could you provide full crash log output?

It should have query that cause the crash and versions of daemon along with MCL

[sanikolaev]

@patrikhermansson We also need to know your Manticore and MCL (Manticore Columnar Library) version.

[patrikhermansson]

I don't see anything in the log except for the lines posted above. Line before is just a standard: rt: table TABLENAME: diskchunk 903(387), segments 10 saved in 2.893498 (7.905310) sec, RAM saved/new 90985767/297426799 ratio 0.333333 (soft limit 44739197, conf limit 134217728)

I'm running a batch import via the json api. I've checked the query log and not getting anything there either. Is there anywhere else?

Here are the version details: Manticore 6.0.4 1a3a4ea82@230314 (columnar 2.0.4 5a49bd7@230306) (secondary 2.0.4 5a49bd7@230306)

[sanikolaev]

@patrikhermansson please provide the output of cat /proc/cpuinfo

[patrikhermansson]

Sure this is what Im on:

processor : 1
vendor_id : AuthenticAMD
cpu family : 25
model : 1
model name : AMD EPYC 7313 16-Core Processor
stepping : 1
microcode : 0xa001144
cpu MHz : 1500.000
cache size : 512 KB
physical id : 0
siblings : 32
core id : 1
cpu cores : 16
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 16
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca sme sev sev_es
bugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass
bogomips : 6000.55
TLB size : 2560 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

[sanikolaev]

Thanks. OK, you've got sse4_2. It's just that Crash!!! Handling signal 6 could happen because the CPU is too old, but it's not your case.

[patrikhermansson]

Yes, it's a rather up to date CPU.

I'm running this in docker with memlock / ipc lock. If that's relevant. My docker-compose config:

manticore:
image: manticoresearch/manticore
environment:
- EXTRA=1
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
memlock:
soft: -1
hard: -1
cap_add:
- IPC_LOCK
environment:
- EXTRA=1
- QUERY_LOG_TO_STDOUT=true

[sanikolaev]

I can't reproduce the issue after inserting 100M docs with a bigint with high concurrency

root@perf ~ # mysql -P9306 -h0 -e "select count(*) from big; desc big; select * from big"
+-----------+
| count(*)  |
+-----------+
| 100000000 |
+-----------+
+-------+--------+---------------------+
| Field | Type   | Properties          |
+-------+--------+---------------------+
| id    | bigint | columnar fast_fetch |
| a     | bigint | columnar fast_fetch |
+-------+--------+---------------------+
+----------+---------------------+
| id       | a                   |
+----------+---------------------+
| 10960000 | 4338315660539059073 |
|  6180000 | 2887223011121785419 |
|  1170000 | 2317195558393372589 |
| 42390000 | 2855216435071718328 |
| 76690000 |  467402649807378195 |
| 16230000 |  163829614606304563 |
| 74220000 |  151550084550947092 |
| 20850000 |  301749822751561457 |
| 25620000 | 2663671494022688385 |
| 28270000 | 3257520370875634595 |
| 32810000 | 1867070823271238253 |
|  3710000 |  523367845367911992 |
| 37580000 | 3130416205645549845 |
| 88030000 | 2675725742403853785 |
|  8560000 | 2444432541463393430 |
| 69320000 | 3838349613530557320 |
| 66770000 | 3118735865690233652 |
| 49820000 | 1526823756460321103 |
| 98630000 | 3273029932253883017 |
| 13570000 | 1978262314688973982 |
+----------+---------------------+

The version is:

root@perf ~ # searchd -v
Manticore 6.0.4 1a3a4ea82@230314 (columnar 2.0.4 5a49bd7@230306) (secondary 2.0.4 5a49bd7@230306)

@patrikhermansson please provide more details:

• your schema
• sample document

[sanikolaev]

I've discussed this issue with the developer and chances are the issue has been already fixed in the MCL. Can you try out the latest dev packages of Manticore Search / MCL - https://mnt.cr/nightly