Fine-tuning Data Mix

[quannguyenminh103]

Hi, thanks for sharing the great work. I would love to adapt your mechanism into my project. However, I have some questions that I hope you can help address.

1. You mentioned "For robust training, include the new data in a large data mix (e.g., our provided SFT blend)." in the fine-tuning example. I am unsure what is the large data mix over here? Do you mean we have to add my training data with your pretrain data? However, isn't your data private?
2. How much data should be sufficient (number of samples) for the domain specific fine-tuning phase?
3. I have a small specific dataset with text labels (which can be used for the fine-tuning) but I also have a much bigger medical image data source (w/o labels). Do you think we could leverage the the big data source (w/o labels) somewhere in your pipeline to make the model more power?

Thank you!