-
Notifications
You must be signed in to change notification settings - Fork 74
Open
Description
通过TGI托管的模型,启动方式为:
BNB_CUDA_VERSION=122 CUDA_VISIBLE_DEVICES=0,1 text-generation-launcher --model-id /data/llms/codeshell-7b-chat --tokenizer-config-path /data/llms/codeshell-7b-chat/tokenizer_config.json --sharded true --trust-remote-code --port=8080
CUDA_VISIBLE_DEVICES=0,1 和 --sharded true 设置后报错:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
Metadata
Metadata
Assignees
Labels
No labels