Include Prompts in Evaluation?

[haikangdeng]

Hi, I'm curious about the intuition to include or not include prompts during evaluation. In your toxicity evaluation, only the generated sequence is evaluated. However, in your sentiment evaluation, prompts are prepended to the generated sequences, and the entire sequence is evaluated. Would it be better to keep the settings consistent in these experiments? Which do you recommend, keep or remove prompts? Thanks

Toxicity

DExperts/scripts/run_toxicity_experiment.py

Lines 171 to 173 in 30e2af5

	for i, gen in enumerate(generations_iter):
	generations.append(gen)
	perspective(f'generation-{i}', gen)

Sentiment

DExperts/scripts/run_sentiment_experiment.py

Lines 165 to 171 in 30e2af5

	for i, p in tqdm(enumerate(prompts), total=len(prompts), desc='Scoring generations'):
	sentences_for_prompt = []
	for j in range(n):
	gen = generations[i*n + j]
	sentences_for_prompt.append(f'{p}{gen}')
	try:
	predictions_for_prompt = classifier(sentences_for_prompt)