Flaky tests in Haystack test suite

[anakin87]

In this issue, we aim to collect flaky (unit or integration) tests that are part of the Haystack test suite. Later, we'll investigate and fix them.

Example entry:
test/components/extractors/test_regex_text_extractor.py::TestRegexTextExtractor::test_multiple_matches_first_is_captured

Just an example, the test above is not flaky

[sjrl]

Another example is:

• test/components/generators/chat/test_openai_responses.py::TestOpenAIResponsesChatGenerator::test_live_run_with_agent_streaming_and_reasoning
• Link to failing tests
• Error
  Error: Error code: 400 - {'error': {'message': "Item 'fc_0cd18a3f182261f2006964ed1d6034819087b5747e92b1656d' of type 'function_call' was provided without its required 'reasoning' item: 'rs_0cd18a3f182261f2006964ed1adcbc8190bea68a78ecdec915'.", 'type': 'invalid_request_error', 'param': 'input', 'code': None}}

---

Fixed in #10390

[mpangrazzi]

@sjrl +1 https://github.com/deepset-ai/haystack/actions/runs/20952468527/job/60209361372?pr=10362

[anakin87]

FAILED test/components/extractors/test_llm_metadata_extractor.py::TestLLMMetadataExtractor::test_live_run - haystack.core.errors.PipelineRuntimeError: The following component failed to run:
Component name: 'extractor'
Component type: 'LLMMetadataExtractor'
Error: list indices must be integers or slices, not dict
https://github.com/deepset-ai/haystack/actions/runs/20989681541/job/60332212066

---

Fixed in #10382

[sjrl]

FAILED test/components/extractors/test_llm_metadata_extractor.py::TestLLMMetadataExtractor::test_live_run - haystack.core.errors.PipelineRuntimeError: The following component failed to run: Component name: 'extractor' Component type: 'LLMMetadataExtractor' Error: list indices must be integers or slices, not dict https://github.com/deepset-ai/haystack/actions/runs/20989681541/job/60332212066

Addressed in PR #10382

[anakin87]

FAILED test/core/pipeline/break_points/test_pipeline_breakpoints_agent_function_calling.py::TestPipelineBreakpoints::test_agent_components_pipeline_breakpoints[chat_generator] - assert None == 42

https://github.com/deepset-ai/haystack/actions/runs/21165987442/job/60871128024?pr=10404

---

fixed in #10422

[sjrl]

@anakin87 here is another flaky tests https://github.com/deepset-ai/haystack/actions/runs/21210672245/job/61018123808?pr=10403#step:5:5953

FAILED test/components/builders/test_prompt_builder.py::TestPromptBuilder::test_date_with_subtraction_offset - AssertionError: assert 'Time after 1...1-09 13:07:34' == 'Time after 1...1-09 13:07:33'

• Time after 12 days is: 2026-01-09 13:07:33
  ? ^

• Time after 12 days is: 2026-01-09 13:07:34
  ? ^

---

fixed with all similar tests in #10437

[anakin87]

  Traceback (most recent call last):
    File "/Users/stefano.fiorucci/dev/haystack/haystack/components/generators/chat/hugging_face_local.py", line 259, in __del__
      if self._owns_executor:
  AttributeError: 'HuggingFaceLocalChatGenerator' object has no attribute '_owns_executor'

This is not a failure per se, but we could improve this component, which sometimes also makes Windows tests hang in relation to executor deletion.

---

fixed in #10431

[anakin87]

FAILED test/components/generators/chat/test_openai_async.py::TestOpenAIChatGeneratorAsync::test_run_async_cancellation_integration - assert 0 > 0

from https://github.com/deepset-ai/haystack/actions/runs/21212784205/job/61025974132

---

fixed in #10427