CUDA OOM when FineTuning on T4 15 GB

[mellahysf]

@juyongjiang Thank you for this great work!

How to finetune the model using less memory?

I'm facing CUDA OOM while trying to finetune on google colab pro with T4 15 GB...

Thanks!

[juyongjiang]

@mellahysf Hi, essentially, we use the low-rank adaptation (LoRA) to fine-tune the LLMs, making it feasible on a single GPU with less memory. So if only 15GB memory is available, I suggest reducing the rank parameter --lora_r=4 while it may decrease the model performance, and setting the smaller --batch_size=64. Have a try. : )

[mellahysf]

@juyongjiang it didn't work with less memory... I tested with GPU 25 GB and it works. thanks.