Replicating Eval Results

[artemisp]

Hello, I was wondering what are the inference hyperparameters used to replicate the results mentioned in the arxiv paper? Using the demo hyper-parameters seems to lead to significantly lower CIDEr score on Clotho and NoCaps.

[artemisp]

No I have gotten significantly worse results in all datasets I tried. We should wait for a response from the authors. Vicuna models are notoriously sensitive to prompting and generation hyperparameters so that might be at fault.