CI: retry ollama pull

[olliewalsh]

ollama pull is unreliable, consistently failing on the first attempt. Add some retry logic.

Summary by Sourcery

Add retry logic around flaky ollama pull operations in tests to improve reliability.

Tests:

• Retry Ollama model pulls in Python e2e helper with a bounded retry loop.
• Wrap ollama pull commands in system tests with simple shell retry loops to reduce flakiness.

[sourcery-ai]

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Adds retry logic for flaky ollama pull operations in both Python E2E tests and shell-based system tests to improve CI robustness.

File-Level Changes

Change	Details	Files
Add retry logic around ollama model pulling in the Python E2E test harness.	Extend pull_model helper to accept a configurable retries parameter with a default of 5. Wrap the ollama pull subprocess call in a retry loop that decrements retries on CalledProcessError and ultimately re-raises when retries are exhausted. Remove capture_output from the subprocess call, allowing normal stdout/stderr behavior during retries.	test/e2e/conftest.py
Harden system tests by retrying flaky ollama pull commands in shell tests.	Replace single ollama pull invocations with a shell for-loop that attempts the pull up to four times before a final unwrapped attempt. Apply this retry wrapper to pulls of tinyllama, smollm:135m, and smollm:360m prior to running ramalama commands.	test/system/050-pull.bats

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[gemini-code-assist]

Summary of Changes

Hello @olliewalsh, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the reliability of CI pipelines by addressing the known flakiness of the ollama pull command. It integrates retry logic into both Python-based end-to-end tests and shell-based system tests, ensuring that transient failures during model pulling do not prematurely halt test execution. This change aims to improve the overall stability and consistency of automated testing.

Highlights

• Retry Logic for Ollama Pull in E2E Tests: Implemented a robust retry mechanism within the pull_model function in test/e2e/conftest.py to handle flaky ollama pull operations, with a default of 5 retries before failing.
• Retry Logic for Ollama Pull in System Tests: Introduced shell-based retry loops for ollama pull commands in test/system/050-pull.bats, allowing up to 4 attempts for model pulls like tinyllama and smollm.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• In pull_model, consider replacing the while True + manual decrement with a bounded loop (e.g. for _ in range(retries)) and adding a small sleep/backoff between attempts so failures don’t spin immediately and the max number of attempts is clearer.
• The retry pattern for ollama pull in 050-pull.bats is duplicated three times; consider extracting this into a small helper or function to keep the test logic DRY and easier to tweak if the retry behavior changes.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In `pull_model`, consider replacing the `while True` + manual decrement with a bounded loop (e.g. `for _ in range(retries)`) and adding a small sleep/backoff between attempts so failures don’t spin immediately and the max number of attempts is clearer.
- The retry pattern for `ollama pull` in `050-pull.bats` is duplicated three times; consider extracting this into a small helper or function to keep the test logic DRY and easier to tweak if the retry behavior changes.

## Individual Comments

### Comment 1
<location> `test/system/050-pull.bats:38-47` </location>
<code_context>
     ollama serve &
     sleep 3
-    ollama pull tinyllama
+    (for i in $(seq 0 3); do ollama pull tinyllama && exit 0; done; ollama pull tinyllama)
     run_ramalama pull ollama://tinyllama
     run_ramalama rm ollama://tinyllama
     ollama rm tinyllama

-    ollama pull smollm:135m
+    (for i in $(seq 0 3); do ollama pull smollm:135m && exit 0; done; ollama pull smollm:135m)
     run_ramalama pull https://ollama.com/library/smollm:135m
     run_ramalama list
     is "$output" ".*ollama://library/smollm:135m" "image was actually pulled locally from ollama cache"

-    ollama pull smollm:360m
+    (for i in $(seq 0 3); do ollama pull smollm:360m && exit 0; done; ollama pull smollm:360m)
     RAMALAMA_TRANSPORT=ollama run_ramalama pull smollm:360m
     run_ramalama pull ollama://smollm:360m
</code_context>

<issue_to_address>
**suggestion:** Factor out the repeated retry pattern into a helper to keep the system test DRY and more maintainable.

The retry snippet `(for i in $(seq 0 3); do ollama pull <model> && exit 0; done; ollama pull <model>)` is repeated for three models. Define a small Bats helper (e.g., `retry_ollama_pull <model>`) in `setup_suite` or a shared helper file and call that instead to avoid duplication and keep retry behavior consistent.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[gemini-code-assist]

Code Review

This pull request adds much-needed retry logic to address the flakiness of ollama pull in the test suite, which is a great step towards improving CI reliability. My review includes suggestions to make this new retry logic even more robust and maintainable. In the Python test helper, I've recommended using a more idiomatic for loop and adding a delay between retries. For the BATS tests, I've highlighted code duplication and the same missing delay, suggesting a refactoring into a helper function and providing an improved one-liner with a sleep command.

[telemaco]

LGTM

[mikebonnet]

PR #2289 marks the failing tests with xfail, any preference on approach?

[olliewalsh]

PR #2289 marks the failing tests with xfail, any preference on approach?

ollama seems to be stable again now. It's github project has a number of open issues related to pulling images though so I think this retry logic is still worth merging