Whoa! 45 HOURS to train 8 tracks at 1 minute each - 1000 epochs total??? WHUUUUT?

[LIQUIDMIND111]

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• LORA TRAINING for 8 songs on an RTX 5060 Ti 16GB VRAM - QUESTION in PRE-PROCESSING #429

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[ChuxiJ]

It shouldn’t take that long. Could you let me know your device and GPU?
Also, please provide your model initialization options, as they might have an impact.

[LIQUIDMIND111]

It shouldn’t take that long. Could you let me know your device and GPU? Also, please provide your model initialization options, as they might have an impact.

Thanks for the quick follow-up. Here are my details:

Device & GPU: PC with NVIDIA GeForce RTX 5060 Ti (16GB VRAM).

Operating System: Windows 10.

Model Initialization Options (from my Service Configuration):

Main Model: acestep-v15-base

5Hz LM Model: acestep-5Hz-lm-4B.

Initialization Settings: Initialize 5Hz LM, Use Flash Attention, but Offload to CPU, Offload DiT to CPU, Compile Model, and INT8 Quantization were all disabled

Training Context: I was training a LoRA with 8 tracks (approx. 1 minute each). I was currently seeing an ETA of 47-52 hours for 1000 epochs, with each epoch taking about 180 seconds.

i had to reduce the 8 tracks to 30 seconds!!! at 200 epochs total.

now it takes 4 hours,

[LIQUIDMIND111]

It shouldn’t take that long. Could you let me know your device and GPU? Also, please provide your model initialization options, as they might have an impact.

i am having issues with the LOSS, using LR at 0.0001 i get an average loss of 0.6, should i do 0.00005 to try to go lower? or what LR do you recommend for 8 tracks at 30 second each and a total of 200 epochs?

[ChuxiJ]

Just use the default hyperparameters. I didn’t tweak the settings much for most of my LoRAs either.

I might not be up to date, so I don’t know the compute capability of the RTX 5060 Ti.
For a 3090 Ti, 200–400 epochs should be enough.

You can also try the latest main branch and see if your issue is fixed.