SWE-Bench Multimodal

[juanmichelini]

When we run eval_infer in swebenchmultimodal, if no instance was created it crashes.
It shouldn't crash. Instead it should run as usual, letting the harness be responsible for the next course of action.

Log Tail:[01/17/26 21:07:36] INFO     Model name:                       eval_infer.py:270                             litellm_proxy/minimax/minimax-m2                   [01/17/26 21:07:36] INFO     Converting                         eval_infer.py:50                             eval_outputs/princeton-nlp__SWE-be                                              nch_Multimodal-dev/litellm_proxy/m                                              inimax/minimax-m2_sdk_0f4bbd5_maxi                                              ter_500_N_litellm_proxy-minimax-mi                                              nimax-m2/output.jsonl to SWE-Bench                                              format:                                                                         eval_outputs/princeton-nlp__SWE-be                                              nch_Multimodal-dev/litellm_proxy/m                                              inimax/minimax-m2_sdk_0f4bbd5_maxi                                              ter_500_N_litellm_proxy-minimax-mi                                              nimax-m2/output.swebench.jsonl                     [01/17/26 21:07:36] INFO     Conversion complete: 0 entries    eval_infer.py:104                             converted, 0 errors                                [01/17/26 21:07:36] ERROR    Script failed: No valid entries   eval_infer.py:288                             were converted                                     /workspace/benchmarks/.venv/lib/python3.12/site-packages/litellm/llms/custom_httpx/async_client_cleanup.py:66: DeprecationWarning: There is no current event loop  loop = asyncio.get_event_loop()

[juanmichelini]

@OpenHands can you fix this? please be minimal

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary

Fixed issue #338 with a minimal change. The PR is now open at: #339

Changes Made

1. Removed the exception in benchmarks/swebenchmultimodal/eval_infer.py (lines 109-110) that was causing the crash when no entries were converted:

   # Removed:
   if converted_count == 0:
       raise ValueError("No valid entries were converted")

2. Added a test in tests/test_swebenchmultimodal.py to verify that empty input files don't raise exceptions.

Checklist

• ✅ Issue addressed: Script no longer crashes when no entries are converted
• ✅ Minimal change: Only removed 3 lines of code
• ✅ Test added to verify the fix
• ✅ Test passes
• ✅ Branch created and pushed
• ✅ PR opened with reference to issue SWE-Bench Multimodal #338

View full conversation