import config file missedFilefile_config.json of safety_llm

[ZJunBo]

file_config.json of safety_llm is missed.

[Y0oMu]

Hi @ZJunBo,

Thank you for reporting this issue. The file_config.json for safety_llm has been uploaded to the URL specified in download.py. Could you please try downloading it again?

Let me know if you encounter any further issues.

[ZJunBo]

When generating the model response in the safety_llm section, although there is no error reported in the code, the generated JSON file does not have "responses". Why is this the case?

[ZJunBo]

[nauyisu022]

Thank you for reporting this issue! The problem occurs because of inconsistent key naming in the JSON file. To resolve it, please replace all instances of "jailbreak_prompt" with "prompt" in the file.

For example:

{
"prompt": "Your content here" // Updated key (previously "jailbreak_prompt")
}
If you need further assistance or encounter other issues, feel free to ask!
This fix has already been implemented in the latest codebase.

[nauyisu022]

Thank you for reporting this issue! The problem occurs because of inconsistent key naming in the JSON file. To resolve it, please replace all instances of "jailbreak_prompt" with "prompt" in the file.

For example:

{ "prompt": "Your content here" // Updated key (previously "jailbreak_prompt") } If you need further assistance or encounter other issues, feel free to ask! This fix has already been implemented in the latest codebase.

Another Solution:

await generate_responses(
    data_folder=data_folder,
    request_type=request_type,
    prompt_key='prompt',  # [name of prompt key]
    async_list=async_list,
    sync_list=sync_list,
)

There is another solution: we can add a prompt_key parameter to generate_responses where you can pass in the name of the prompt key. In the screenshot's case, this would be 'jailbreak_prompt'.

[ZJunBo]

Thank you for reporting this issue! The problem occurs because of inconsistent key naming in the JSON file. To resolve it, please replace all instances of "jailbreak_prompt" with "prompt" in the file.
For example:
{ "prompt": "Your content here" // Updated key (previously "jailbreak_prompt") } If you need further assistance or encounter other issues, feel free to ask! This fix has already been implemented in the latest codebase.

Another Solution:

await generate_responses(
data_folder=data_folder,
request_type=request_type,
prompt_key='prompt', # [name of prompt key]
async_list=async_list,
sync_list=sync_list,
)
There is another solution: we can add a prompt_key parameter to generate_responses where you can pass in the name of the prompt key. In the screenshot's case, this would be 'jailbreak_prompt'.

Thanks for your reply. In your way, the problem can be solved.

[ZJunBo]

I'd like to know if the code for obtaining the Safety_llm evaluation results is correct. It seems that the safety evaluation code is the same as that for truthfulness. However, I encounter an error during execution. It prompts me that the qa.csv file is required, but the files I've generated are only all_jailbreak_prompts.csv and exaggerate_safety.csv.
In safety_llm.ipynb, code is shown as:

The error prompts me that the qa.csv file is required:

The files I've generated:

[nauyisu022]

I'd like to know if the code for obtaining the Safety_llm evaluation results is correct. It seems that the safety evaluation code is the same as that for truthfulness. However, I encounter an error during execution. It prompts me that the qa.csv file is required, but the files I've generated are only all_jailbreak_prompts.csv and exaggerate_safety.csv. In safety_llm.ipynb, code is shown as:

The error prompts me that the qa.csv file is required:

The files I've generated:

Thank you for pointing that out! You’re right—this was a bug where the safety evaluation incorrectly reused truthfulness code, leading to an expectation for qa.csv.

I’ve rewritten the logic for the safety_llm metrics calculation, and this has been fixed in the latest commit. It works well on my end.

To resolve the issue, please run git pull to ensure you have the latest code.
You will then see the output of the metrics results in a DataFrame table.

Make sure to check the correct file paths as well.

Let me know if you encounter any further issues or need clarification!

[ZJunBo]

hi, when I run fairness_llm.ipynb, in contexual variator part, it prompts me some files are missed. LLM_fairness_disparagement.json and LLM_fairness_preference.json. Why causes this problem?

[nauyisu022]

hi, when I run fairness_llm.ipynb, in contexual variator part, it prompts me some files are missed. LLM_fairness_disparagement.json and LLM_fairness_preference.json. Why causes this problem?

Hi,

Apologies for the inconvenience. The issue you’re encountering is due to the fact that the generation for preference and disparagement was mistakenly commented out. We have fixed this bug now. Please pull the latest code, and try running it again.

If the stereotype data has already been generated, you can also pass the subset parameter when calling the function:

from trusteval.dimension.fairness.fairness_llm import pipeline
pipeline.run(base_dir=base_dir, subset=['preference', 'disparagement'])

This should resolve the problem. Let me know if you encounter any further issues, and again, we apologize for the oversight.

[ZJunBo]

hi, when I run fairness_llm.ipynb, in contexual variator part, it prompts me some files are missed. LLM_fairness_disparagement.json and LLM_fairness_preference.json. Why causes this problem?

Hi,

Apologies for the inconvenience. The issue you’re encountering is due to the fact that the generation for preference and disparagement was mistakenly commented out. We have fixed this bug now. Please pull the latest code, and try running it again.

If the stereotype data has already been generated, you can also pass the subset parameter when calling the function:

from trusteval.dimension.fairness.fairness_llm import pipeline
pipeline.run(base_dir=base_dir, subset=['preference', 'disparagement'])
This should resolve the problem. Let me know if you encounter any further issues, and again, we apologize for the oversight.

Thanks for your reply, now fairness llm data can be correctly processed. But when I run lm_metric function, it shows Aspect fairness_llm not found in aspect_dict. I check the code, it seems that in aspect_dict (in lm_evaluator.py) has no links to fairness_llm.

[ZJunBo]

hi, when I run fairness_llm.ipynb, in contexual variator part, it prompts me some files are missed. LLM_fairness_disparagement.json and LLM_fairness_preference.json. Why causes this problem?

Hi,
Apologies for the inconvenience. The issue you’re encountering is due to the fact that the generation for preference and disparagement was mistakenly commented out. We have fixed this bug now. Please pull the latest code, and try running it again.
If the stereotype data has already been generated, you can also pass the subset parameter when calling the function:
from trusteval.dimension.fairness.fairness_llm import pipeline
pipeline.run(base_dir=base_dir, subset=['preference', 'disparagement'])
This should resolve the problem. Let me know if you encounter any further issues, and again, we apologize for the oversight.

Thanks for your reply, now fairness llm data can be correctly processed. But when I run lm_metric function, it shows Aspect fairness_llm not found in aspect_dict. I check the code, it seems that in aspect_dict (in lm_evaluator.py) has no links to fairness_llm.

This problem is solved. I added the links to fairness_llm.

[ZJunBo]

When evaluating the fairness_llm, although the code can be executed correctly, I found that it seems the model didn't generate the expected responses. The content is shown in the figure.

[Y0oMu]

When evaluating the fairness_llm, although the code can be executed correctly, I found that it seems the model didn't generate the expected responses. The content is shown in the figure.

We appreciate your feedback. The defect has been resolved. Please pull the latest code to apply the fix.

[ZJunBo]

When evaluating the fairness_llm, although the code can be executed correctly, I found that it seems the model didn't generate the expected responses. The content is shown in the figure.

We appreciate your feedback. The defect has been resolved. Please pull the latest code to apply the fix.

Thank you for your reply. I've run the latest code, and there are new issues:

1. the response content of the model is irrelevant to the prompt, and the correspondence seems inconsistent.
2. the judge_responses cannot generate corresponding judgment responses.
   Thank you for your assistance.