Fixes for the qwen support #130

sanikolaev · 2026-01-29T16:12:51Z

No description provided.

This reverts commit 974bd51.

Implement Qwen local embedding model + tokenizer sanitization Fix attention/weight loading quirks for Qwen weights Update embeddings lib version to 1.1.1

CLAassistant · 2026-01-29T16:12:57Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ sanikolaev
❌ donhardman
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

github-actions · 2026-01-29T17:06:27Z

Linux debug test results

8 files 8 suites 13m 27s ⏱️
504 tests 482 ✅ 22 💤 0 ❌
518 runs 496 ✅ 22 💤 0 ❌

Results for commit 0a3b8d8.

♻️ This comment has been updated with latest results.

github-actions · 2026-01-29T17:06:29Z

Windows test results

5 files 5 suites 18m 6s ⏱️
485 tests 470 ✅ 15 💤 0 ❌
493 runs 478 ✅ 15 💤 0 ❌

Results for commit 0a3b8d8.

♻️ This comment has been updated with latest results.

github-actions · 2026-01-29T17:13:21Z

Linux release test results

8 files 8 suites 6m 29s ⏱️
504 tests 489 ✅ 15 💤 0 ❌
518 runs 503 ✅ 15 💤 0 ❌

Results for commit 0a3b8d8.

♻️ This comment has been updated with latest results.

- Remove QwenModel variant and custom implementation - LocalModel now handles Qwen, Llama, Mistral, Gemma via auto-detection - Add ModelArch enum for BERT vs causal architecture detection - Implement CausalEmbeddingModel for supported architectures - Consolidate embedding logic into unified LocalModel - Remove redundant qwen.rs implementation - Update create_model to route non-API models through LocalModel - Add architecture detection and integration tests

donhardman

I implemented the proper way. Now it should support Llama, Qwen, Mistral, and BERT models. It’s still good to check with Manticore together, but things are covered by tests and implemented in the right way now.

github-actions · 2026-01-30T18:32:31Z

clt

❌ CLT tests in test/clt-tests/mcl/
✅ OK: 14
❌ Failed: 1
⏳ Duration: 522s
👉 Check Action Results for commit d3205e4

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/mcl/auto-embeddings-qwen.rec

test/clt-tests/mcl/auto-embeddings-qwen.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_qwen (title TEXT, vec FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME='Qwen/Qwen3-Embedding-0.6B' FROM='title')"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: error adding table 'test_qwen': prealloc: Failed to create an instance of the model
+ 1
––– input –––
mysql -h0 -P9306 -E -e "SHOW CREATE TABLE test_qwen"
––– output –––
- *************************** 1. row ***************************
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
-        Table: test_qwen
- Create Table: CREATE TABLE test_qwen (
- id bigint,
- title text,
- vec float_vector knn_type='hnsw' hnsw_similarity='L2' model_name='Qwen/Qwen3-Embedding-0.6B' FROM='title'
- )
––– input –––
mysql -h0 -P9306 -e "insert into test_qwen(id, title) values(1, 'book'),(2, 'bread');"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as total_records FROM test_qwen"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "select id, title, knn_dist() from test_qwen where knn(vec, 3, 'loaf')"
––– output –––
- +------+-------+------------+
+ ERROR 1064 (42000) at line 1: table test_qwen: requested KNN search attribute 'vec' not found
- | id   | title | knn_dist() |
- +------+-------+------------+
- |    2 | bread | #!/0\.111[0-9]*/!# |
- |    1 | book  | #!/0\.118[0-9]*/!# |
- +------+-------+------------+

- Fix weight prefix remapping for Qwen3-Embedding models - Correct tensor dtype handling in embedding computation - Enable tests to run instead of skipping on load failure - Update candle dependencies to version 0.9.2 - Align hf-hub revision for compatibility

- Use manticoresoftware candle fork with clear_kv_cache() - Explicitly clear cache to prevent stale state between inferences

- Add test_cache_path helper using CARGO_MANIFEST_DIR - Replace hardcoded paths across all test cases - Ensure consistent and portable cache directory handling

…lInfo

- Downgrade hf-hub to 0.3.2 - Downgrade dirs, dirs-sys, redox_users - Align ureq HTTP client dependencies - Add windows-sys 0.48.x targets

github-actions · 2026-01-30T22:01:09Z

clt

❌ CLT tests in test/clt-tests/mcl/
✅ OK: 14
❌ Failed: 1
⏳ Duration: 498s
👉 Check Action Results for commit cf2a147

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/mcl/auto-embeddings-qwen.rec

test/clt-tests/mcl/auto-embeddings-qwen.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_qwen (title TEXT, vec FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME='Qwen/Qwen3-Embedding-0.6B' FROM='title')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -E -e "SHOW CREATE TABLE test_qwen"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "insert into test_qwen(id, title) values(1, 'book'),(2, 'bread');"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as total_records FROM test_qwen"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "select id, title, knn_dist() from test_qwen where knn(vec, 3, 'loaf')"
––– output –––
+------+-------+------------+
| id   | title | knn_dist() |
+------+-------+------------+
- |    2 | bread | #!/0\.111[0-9]*/!# |
+ |    2 | bread | 0.31665143 |
- |    1 | book  | #!/0\.118[0-9]*/!# |
+ |    1 | book  | 0.50007039 |
+------+-------+------------+

- Expand CausalEmbeddingKind enum for Qwen, Llama, Mistral, Gemma - Extend model type detection for gemma2 and gemma3 variants - Support both torch_dtype and dtype config fields for tensor type - Add integration tests for TinyLlama, TinyMistral, and Gemma models

- Add integration tests for embedding models - Cover loading and initialization paths - Test encoding functionality with various inputs - Verify output consistency and format

…fork - Pin candle-core, candle-nn, candle-transformers to specific git revision - Move test helper functions to dedicated test module - Minor loop variable refactor for clarity

github-actions · 2026-01-31T09:59:24Z

clt

❌ CLT tests in test/clt-tests/mcl/
✅ OK: 14
❌ Failed: 1
⏳ Duration: 512s
👉 Check Action Results for commit 448837c

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/mcl/auto-embeddings-qwen.rec

test/clt-tests/mcl/auto-embeddings-qwen.rec

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd ${SEARCHD_ARGS:-} > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Accepting connections!'; else echo 'Timeout or failed!'; fi
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "CREATE TABLE test_qwen (title TEXT, vec FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME='Qwen/Qwen3-Embedding-0.6B' FROM='title')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -E -e "SHOW CREATE TABLE test_qwen"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "insert into test_qwen(id, title) values(1, 'book'),(2, 'bread');"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "SELECT COUNT(*) as total_records FROM test_qwen"
––– output –––
OK
––– input –––
mysql -h0 -P9306 -e "select id, title, knn_dist() from test_qwen where knn(vec, 3, 'loaf')"
––– output –––
+------+-------+------------+
| id   | title | knn_dist() |
+------+-------+------------+
- |    2 | bread | #!/0\.111[0-9]*/!# |
+ |    2 | bread | 0.31665143 |
- |    1 | book  | #!/0\.118[0-9]*/!# |
+ |    1 | book  | 0.50007039 |
+------+-------+------------+

donhardman · 2026-01-31T10:02:15Z

Also those model supported:

Locutusque/TinyMistral-248M-v2
TinyLlama/TinyLlama-1.1B-Chat-v1.0
h2oai/embeddinggemma-300m

sanikolaev and others added 3 commits January 29, 2026 23:05

Revert "feat: Qwen local embeddings models support"

8b51ee2

This reverts commit 974bd51.

feat: add Qwen embeddings models support

dc9ad9d

Implement Qwen local embedding model + tokenizer sanitization Fix attention/weight loading quirks for Qwen weights Update embeddings lib version to 1.1.1

chore(ci): bump Rust version to 1.92.0 in embedding build workflow

07c5374

sanikolaev added 2 commits January 29, 2026 23:33

fix: code style

b0ae69c

fix: satisfy clippy unnecessary_map_or in qwen.rs

2fcae88

sanikolaev and others added 3 commits January 30, 2026 00:23

refactoring

ace7446

fix: build issue

ed7d665

donhardman self-requested a review January 30, 2026 18:13

donhardman approved these changes Jan 30, 2026

View reviewed changes

donhardman force-pushed the qwen-new branch from 5131583 to dc6d0ce Compare January 30, 2026 19:40

donhardman added 4 commits January 31, 2026 03:14

refactor(embeddings): clear KV cache before model forward pass

c95d5b1

- Use manticoresoftware candle fork with clear_kv_cache() - Explicitly clear cache to prevent stale state between inferences

refactor(local): centralize cache path resolution for tests

b4d6750

- Add test_cache_path helper using CARGO_MANIFEST_DIR - Replace hardcoded paths across all test cases - Ensure consistent and portable cache directory handling

refactor(embeddings): simplify model constructors to accept LocalMode…

a75d5cc

…lInfo

chore(embeddings): downgrade hf-hub and dependencies

dbd422d

- Downgrade hf-hub to 0.3.2 - Downgrade dirs, dirs-sys, redox_users - Align ureq HTTP client dependencies - Add windows-sys 0.48.x targets

donhardman added 3 commits January 31, 2026 16:22

test(embeddings): add integration tests for embedding models

54b21e6

- Add integration tests for embedding models - Cover loading and initialization paths - Test encoding functionality with various inputs - Verify output consistency and format

chore(embeddings): update candle crates to use manticoresoftware git …

0a3b8d8

…fork - Pin candle-core, candle-nn, candle-transformers to specific git revision - Move test helper functions to dedicated test module - Minor loop variable refactor for clarity

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for the qwen support #130

Fixes for the qwen support #130

Uh oh!

sanikolaev commented Jan 29, 2026

Uh oh!

CLAassistant commented Jan 29, 2026

Uh oh!

github-actions bot commented Jan 29, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 29, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 29, 2026 •

edited

Loading

Uh oh!

donhardman left a comment

Uh oh!

github-actions bot commented Jan 30, 2026

Uh oh!

github-actions bot commented Jan 30, 2026

Uh oh!

github-actions bot commented Jan 31, 2026

Uh oh!

donhardman commented Jan 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Fixes for the qwen support #130

Are you sure you want to change the base?

Fixes for the qwen support #130

Uh oh!

Conversation

sanikolaev commented Jan 29, 2026

Uh oh!

CLAassistant commented Jan 29, 2026

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linux debug test results

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Windows test results

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linux release test results

Uh oh!

donhardman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 30, 2026

clt

Failed tests:

Uh oh!

github-actions bot commented Jan 30, 2026

clt

Failed tests:

Uh oh!

github-actions bot commented Jan 31, 2026

clt

Failed tests:

Uh oh!

donhardman commented Jan 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Jan 29, 2026 •

edited

Loading

github-actions bot commented Jan 29, 2026 •

edited

Loading

github-actions bot commented Jan 29, 2026 •

edited

Loading