Skip to content

Sync python dependencies with docker-base#117

Draft
victorlin wants to merge 3 commits intomainfrom
victorlin/sync-python-deps
Draft

Sync python dependencies with docker-base#117
victorlin wants to merge 3 commits intomainfrom
victorlin/sync-python-deps

Conversation

@victorlin
Copy link
Member

@victorlin victorlin commented Jul 7, 2025

Description of proposed changes

Add openpyxl and pyarrow. These were added to docker-base in:

Checklist

@victorlin victorlin self-assigned this Jul 7, 2025
@victorlin
Copy link
Member Author

victorlin commented Jul 7, 2025

Hmm, build is hanging on Getting pinned dependencies: ...working... on the macOS runners. I think it's related to the new packages because I successfully ran CI on main at the same time.

@victorlin victorlin force-pushed the victorlin/sync-python-deps branch from 753d1c0 to f3cc445 Compare July 8, 2025 16:03
@victorlin victorlin mentioned this pull request Jul 8, 2025
@victorlin
Copy link
Member Author

Can't merge this due to the failing build. I opened an issue to investigate further: #119

@victorlin victorlin closed this Jul 8, 2025
@victorlin victorlin deleted the victorlin/sync-python-deps branch July 8, 2025 18:53
@victorlin victorlin restored the victorlin/sync-python-deps branch July 10, 2025 18:24
@victorlin victorlin reopened this Jul 10, 2025
@victorlin victorlin force-pushed the victorlin/sync-python-deps branch 2 times, most recently from 06810d8 to ed47306 Compare July 10, 2025 22:33
- perl
# Pin pulp <2.8 for snakemake: https://github.com/snakemake/snakemake/issues/2607
- pulp <2.8
- pyarrow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, just realized this downgrades evofr in the CI summary. I'm guessing this is due to some dependency resolution for jax/jaxlib.

The evofr dependency specs were fixed in bioconda/bioconda-recipes#40779 which was released with v0.1.18 build_1. Looks like the build chose v0.1.18 build_0, so we might have to specify evofr >=0.1.19 for the dependencies to resolve correctly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 2f8c99f

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And of course the CI builds are hanging again...

Copy link
Member Author

@victorlin victorlin Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call on checking the CI summary, I'll try to do that more in the future.

In this case, it looks like evofr's tight pins around its dependencies from the linked bioconda PR are making it harder to resolve versions that work across all packages and their dependency pins, more notably in macos-13 (which took 40 minutes compared to 5 minutes on the latest stable run) and ubuntu-22 (which has been running for 47 minutes and counting).

I wonder if it's possible to relax evofr's tight pins by testing compatibility with newer versions. For example, the latest version of jax is 0.6.2. My speculation is that it's not allowed only because it wasn't tested at the time of pinning to 0.4.x. If it actually works fine with the latest version, the pin should be relaxed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ubuntu-22 eventually succeeded at 1hr 19min 😬

I believe the pins for jax are pretty tight because they use effort-based versioning, where any version bump can introduce breaking changes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get that, and if the intent of the evofr pins is to avoid any potential breaking changes, the pin should be even tighter capping at an exact 0.X.Y version rather than allowing any new Y as it currently does.

My point is that newer versions should be regularly tested, and for versions with no breaking changes, the pin updated to support it (e.g. nextstrain/augur@e6a57b9). I don't know anything about evofr's usage of its dependencies, but support for newer versions would avoid the significant slowdown in dependency resolution experienced here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would require changes in evofr, so I've opened blab/evofr#56

joverlee521 added a commit that referenced this pull request Jul 11, 2025
Pin minimum version for correct dependency resolution otherwise evofr will get 
downgraded to v0.1.18 build 0

<#117 (comment)>
victorlin and others added 3 commits July 22, 2025 14:38
This was added to docker-base in
nextstrain/docker-base@2904f40
This was added to docker-base in
nextstrain/docker-base@ca8a6fa
Pin minimum version for correct dependency resolution otherwise evofr will get 
downgraded to v0.1.18 build 0

<#117 (comment)>
@victorlin victorlin force-pushed the victorlin/sync-python-deps branch from 2f8c99f to 5ad043a Compare July 22, 2025 21:41
@victorlin victorlin changed the base branch from main to victorlin/update-c-ares July 22, 2025 21:41
@victorlin victorlin added the blocked Dependent on external development label Jul 22, 2025
@victorlin victorlin marked this pull request as draft July 22, 2025 21:42
@victorlin
Copy link
Member Author

I've split the c-ares update into a separate PR since that change is not blocked: #120

Base automatically changed from victorlin/update-c-ares to main July 24, 2025 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blocked Dependent on external development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to add pyarrow

3 participants

Comments