GeminiClient crashes with incomplete API response metadata in humanities_data_benchmark

[MHindermann]

GeminiClient crashes with incomplete API response metadata in humanities_data_benchmark

From RISE-UNIBAS/humanities_data_benchmark#84

Description

The GeminiClient crashes when using gemini-3-pro-preview due to incomplete response metadata. This occurred during the humanities_data_benchmark run (T0416) on 2026-01-24, causing all retries to fail with division and subscript errors.

Complete Error Log

2026-01-25 10:10:30,065 INFO:root:Running T0416
2026-01-25 10:10:30,072 INFO:root:Loaded business_letters from ../benchmarks\business_letters\benchmark.py
2026-01-25 10:10:30,762 INFO:root:Running business_letters (genai/gemini-3-pro-preview)...
2026-01-25 10:10:30,795 INFO:root:Found 57 objects to process.
2026-01-25 10:10:30,795 INFO:root:Skipping T0416, letter01...
[...]
2026-01-25 10:10:30,804 INFO:root:Skipping T0416, letter33...
2026-01-25 10:10:30,804 INFO:root:Processing T0416, letter34...
2026-01-25 10:10:30,928 INFO:ai_client.utils:Resized image letter34_p1.jpg from (2479, 3508) to (1447, 2048)
2026-01-25 10:10:30,945 INFO:google_genai.models:AFC is enabled with max remote calls: 10.
2026-01-25 10:19:31,031 INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-preview:generateContent "HTTP/1.1 200 OK"
2026-01-25 10:19:31,032 WARNING:ai_client.utils:Attempt 1/4 failed for _do_prompt: unsupported operand type(s) for /: 'NoneType' and 'int'. Retrying in 1.00s...
2026-01-25 10:19:32,035 INFO:google_genai.models:AFC is enabled with max remote calls: 10.
2026-01-25 10:28:33,789 INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-preview:generateContent "HTTP/1.1 200 OK"
2026-01-25 10:28:33,790 WARNING:ai_client.utils:Attempt 2/4 failed for _do_prompt: unsupported operand type(s) for /: 'NoneType' and 'int'. Retrying in 2.00s...
2026-01-25 10:28:35,793 INFO:google_genai.models:AFC is enabled with max remote calls: 10.
2026-01-25 10:37:59,632 INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-preview:generateContent "HTTP/1.1 200 OK"
2026-01-25 10:37:59,633 WARNING:ai_client.utils:Attempt 3/4 failed for _do_prompt: unsupported operand type(s) for /: 'NoneType' and 'int'. Retrying in 4.00s...
2026-01-25 10:38:03,636 INFO:google_genai.models:AFC is enabled with max remote calls: 10.
2026-01-25 10:46:58,476 INFO:httpx:HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-preview:generateContent "HTTP/1.1 200 OK"
2026-01-25 10:46:58,477 ERROR:ai_client.utils:Max retries (3) exceeded for _do_prompt
2026-01-25 10:46:58,479 CRITICAL:root:T0416: 'NoneType' object is not subscriptable

Context

• Benchmark: humanities_data_benchmark
• Test ID: T0416
• Date: 2026-01-24
• Model: gemini-3-pro-preview
• API Response: HTTP 200 OK (successful API call, but incomplete metadata)

Root Cause

Despite the API returning 200 OK, the response contains incomplete metadata. Two issues in ai_client/gemini_client.py:_create_response_from_raw():

1. Token count error (retries 1-3): usage_metadata fields (prompt_token_count, candidates_token_count, total_token_count) are None, causing TypeError when
   dividing by 1_000_000 in pricing calculation (ai_client/pricing.py:147-148)

2. Subscript error (final attempt): raw_response.candidates is None or empty, causing TypeError when accessing candidates[0] at gemini_client.py:248

Expected Behavior

The client should gracefully handle incomplete API response metadata by:

• Defaulting to 0 for missing token counts
• Checking candidates array exists and has length > 0 before subscript access
• Continuing execution without crashing when metadata is unavailable