Skip to content

Comments

Create subqueries for geopackage reads (NGWPC-9010 & NGWPC-9018)#82

Draft
idtodd wants to merge 9 commits intodevelopmentfrom
idt-regionalization-workflow-eval-debug
Draft

Create subqueries for geopackage reads (NGWPC-9010 & NGWPC-9018)#82
idtodd wants to merge 9 commits intodevelopmentfrom
idt-regionalization-workflow-eval-debug

Conversation

@idtodd
Copy link

@idtodd idtodd commented Nov 19, 2025

sqlite can restrict the number of binds ("?" in the statement) that it can handle. Turning the single query into a set of subqueries gets around throwing an error when the number of catchments/nexuses gets too large. Prior to version 3.32.0, sqlite's max number of binds is 999, 900 was chosen to keep it well under the limit.

Additions

  • Debug logs to the geopackage read function.
  • Better error reporting when an error is encountered trying to read the geopackage.

Changes

  • When reading features from a geopackage, the query will be broken into subqueries with a maximum of 900 IDs per query.

Testing

Screenshots

Notes

Todos

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

Testing checklist (automated report can be put here)

Target Environment support

  • Linux

@idtodd idtodd self-assigned this Nov 19, 2025
@idtodd idtodd changed the title Create subqueries for geopackage reads (NGWPC-9010) Create subqueries for geopackage reads (NGWPC-9010 & NGWPC-9018) Nov 19, 2025
@idtodd idtodd removed the request for review from mdeshotel November 20, 2025 20:29
@mxkpp
Copy link

mxkpp commented Dec 3, 2025

@idtodd I'd like to use this PR for testing regionalization. Can you please do the following and incorporate them into one push to this branch (multiple commits, one push):

  1. Pull the latest updates from development
  2. Run git submodule update --recursive
  3. Update .gitmodules for ueb_bmi to point to idt-ueb-inclusive-end-step where you're currently working there, and then run git submodule update extern/ueb_bmi
  4. Push to this PR (once)
  5. Wait for this PR's GHCR image to rebuild and send me a note when it's ready

Then the GHCR image associated with this PR will be current for testing regionalization.

Then before we merge this PR, we just need to revert .gitmodules here to point to ueb_bmi's development branch (assuming your ueb_bmi branch has been merged to development branch of that repo by then).

@idtodd idtodd marked this pull request as draft December 11, 2025 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants