Skip to content

Conversation

@kzjeef
Copy link
Collaborator

@kzjeef kzjeef commented May 23, 2025

  • add qwen vl 2.5 model support.
  • Qwen VL2.5 only support 'transformers' as vit engine, (trt not support yet.)
  • upgrade package version to make sure VL2.5 code is added.

Resolves #79

test command:

server:
dashinfer_vlm_serve --model qwen/Qwen2.5-VL-3B-Instruct --vision_engine transformers --port 8000 --host=127.0.0.1

client:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d \ '{"model": "qwen/Qwen2.5-VL-3B-Instruct", "messages": [{"role": "user", "content": [{ "type": "text", "text": "Describe the image." }, {"type": "image_url", "image_url": {"url": "https://farm4.staticflickr.com/3075/3168662394_7d7103de7d_z_d.jpg"}}]}], "max_completion_tokens": 1024, "top_p": 0.5, "temperature": 0.1, "frequency_penalty": 1.05 }'

result:

{"id":"chatcmpl-rxqDiCQEJweEeeB7FADiER","object":"chat.completion",
"created":1747992522,"model":"model","choices":[{"index":0,"message":{"role":"assistant","content":"The
image features a small hummingbird perched on a branch. The bird is positioned in the center of the scene,
with its vibrant colors and delicate features clearly visible. The hummingbird appears to be enjoying its
time in nature, possibly searching for food or simply resting on the branch. \n\nThere are no other birds
or animals present in the image, making it a solitary moment captured in this natural
setting."},"finish_reason":"stop"}],"usage":{"prompt_tokens":382,"total_tokens":95,"completion_tokens":81}}

- add qwen vl 2.5 model support.
- Qwen VL2.5 only support 'transformers' as vit engine, (trt not support
  yet.)
- upgrade package version to make sure VL2.5 code is added.

test command:

server:
`dashinfer_vlm_serve --model qwen/Qwen2.5-VL-3B-Instruct --vision_engine transformers --port 8000 --host=127.0.0.1`

client:
```
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d \ '{"model": "qwen/Qwen2.5-VL-3B-Instruct", "messages": [{"role": "user", "content": [{ "type": "text", "text": "Describe the image." }, {"type": "image_url", "image_url": {"url": "https://farm4.staticflickr.com/3075/3168662394_7d7103de7d_z_d.jpg"}}]}], "max_completion_tokens": 1024, "top_p": 0.5, "temperature": 0.1, "frequency_penalty": 1.05 }'
```

result:
```
{"id":"chatcmpl-rxqDiCQEJweEeeB7FADiER","object":"chat.completion",
"created":1747992522,"model":"model","choices":[{"index":0,"message":{"role":"assistant","content":"The
image features a small hummingbird perched on a branch. The bird is positioned in the center of the scene,
with its vibrant colors and delicate features clearly visible. The hummingbird appears to be enjoying its
time in nature, possibly searching for food or simply resting on the branch. \n\nThere are no other birds
or animals present in the image, making it a solitary moment captured in this natural
setting."},"finish_reason":"stop"}],"usage":{"prompt_tokens":382,"total_tokens":95,"completion_tokens":81}}
```
@kzjeef kzjeef requested a review from x574chen May 23, 2025 10:16
@kzjeef kzjeef added this pull request to the merge queue May 28, 2025
Merged via the queue into main with commit a259d62 May 28, 2025
0 of 12 checks passed
@kzjeef kzjeef mentioned this pull request May 28, 2025
github-merge-queue bot pushed a commit that referenced this pull request Jul 28, 2025
* Update build-check-share-runner.yml

* ci: adjust code clone for build test.

* thridparty: reduce mkl size, remove unused so.

* xformers: replace link with source.

* cutlass: try use local tar rather than git submodule.

* ci: remove shared runner

* Update README.md

* mm: add qwen vl2.5 model support. (#86)

- add qwen vl 2.5 model support.
- Qwen VL2.5 only support 'transformers' as vit engine, (trt not support
  yet.)
- upgrade package version to make sure VL2.5 code is added.

test command:

server:
`dashinfer_vlm_serve --model qwen/Qwen2.5-VL-3B-Instruct --vision_engine transformers --port 8000 --host=127.0.0.1`

client:
```
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d \ '{"model": "qwen/Qwen2.5-VL-3B-Instruct", "messages": [{"role": "user", "content": [{ "type": "text", "text": "Describe the image." }, {"type": "image_url", "image_url": {"url": "https://farm4.staticflickr.com/3075/3168662394_7d7103de7d_z_d.jpg"}}]}], "max_completion_tokens": 1024, "top_p": 0.5, "temperature": 0.1, "frequency_penalty": 1.05 }'
```

result:
```
{"id":"chatcmpl-rxqDiCQEJweEeeB7FADiER","object":"chat.completion",
"created":1747992522,"model":"model","choices":[{"index":0,"message":{"role":"assistant","content":"The
image features a small hummingbird perched on a branch. The bird is positioned in the center of the scene,
with its vibrant colors and delicate features clearly visible. The hummingbird appears to be enjoying its
time in nature, possibly searching for food or simply resting on the branch. \n\nThere are no other birds
or animals present in the image, making it a solitary moment captured in this natural
setting."},"finish_reason":"stop"}],"usage":{"prompt_tokens":382,"total_tokens":95,"completion_tokens":81}}
```

* Update build-check.yml

* Update build-check.yml

* ci: fix release script. (#88)

* cmake: nccl find lib without version.

* flash attn: less memory footprint for compile.

* workflow: fix typo.

* ci: fix git safe dir issue.

* multimodel: minor fix for benchmark

* build: support 90a by default.

* [Build]: upgrade pybind11 for python 3.12, also remove cutlass folder.

* docker: fine centos docker and ubi8 docker.

* add build script for ubi8 support.

* github: action use ubi8 image rather than centos image.

* github: try use github provided machine.

* Revert "github: try use github provided machine."

This reverts commit 6705c60.

* github: remove prefix of docker image.

* github: change workflow image to ubi8 images.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen2.5-VL兼容性

3 participants