Skip to content

Why does my GPU not getting utilized, encodec only utilizes CPU backend #215

@naren200

Description

@naren200

bark.cpp/build/examples/main/main binary doesn't use GPU, it only uses CPU as far as i can see.
I am not sure, why my GPU is not utilized to any extent...

Image

root@thom:~/app/bark.cpp# ./build/examples/main/main -m ./models/bark-small/ggml_weights.bin -p "this is an audio generated by bark.cpp" -t 4

██████╗  █████╗ ██████╗ ██╗  ██╗    ██████╗██████╗ ██████╗ 
██╔══██╗██╔══██╗██╔══██╗██║ ██╔╝   ██╔════╝██╔══██╗██╔══██╗
██████╔╝███████║██████╔╝█████╔╝    ██║     ██████╔╝██████╔╝
██╔══██╗██╔══██║██╔══██╗██╔═██╗    ██║     ██╔═══╝ ██╔═══╝ 
██████╔╝██║  ██║██║  ██║██║  ██╗██╗╚██████╗██║     ██║     
╚═════╝ ╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝ ╚═════╝╚═╝     ╚═╝     

encodec_load_model_weights: using CPU backend
encodec_load_model_weights: model size = 44.36 MB

bark_tokenize_input: prompt: 'this is an audio generated by bark.cpp'
bark_tokenize_input: number of tokens in prompt = 513, first 8 tokens: 20579 20172 20199 33733 58966 20203 28169 20222 

Generating semantic tokens... 78%

bark_print_statistics:   sample time =    75.15 ms / 600 tokens
bark_print_statistics:  predict time =  5390.09 ms / 8.98 ms per token
bark_print_statistics:    total time =  5476.48 ms

Building process for GPU access; btw yet GPU is not working

Also, I changed two lines manually in the following files.

  1. File: bark.cpp/bark.cpp
    You also need to modify bark.cpp line 826
    model.backend = ggml_backend_cuda_init();
    by model.backend = ggml_backend_cuda_init(0);

  2. File: bark.cpp/encodec.cpp/encodec.cpp
    You also need to modify encodec.cpp line 215
    model.backend = ggml_backend_cuda_init();
    by model.backend = ggml_backend_cuda_init(0);

After the above changes I built through the following GPU command,

mkdir build
cd build
cmake -DGGML_CUBLAS=ON -DGGML_CUDA=1 -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 ..
cmake --build . --config Release

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions