Skip to content

Comments

Local test data#23

Merged
dylanlee merged 13 commits intomainfrom
local-test-data
Jul 15, 2025
Merged

Local test data#23
dylanlee merged 13 commits intomainfrom
local-test-data

Conversation

@dylanlee
Copy link
Contributor

@dylanlee dylanlee commented Jul 8, 2025

Added test data and containers to docker compose that would allow for interacting with test data

dylanlee added 3 commits July 8, 2025 11:14
Edited docker compose to have everything necessary to spin up a local
stac api that can be queried. The api has only one collection and one
item and links to assets in the new test/ directory
@dylanlee dylanlee requested a review from ar-siddiqui July 8, 2025 17:08
@ar-siddiqui
Copy link
Member

Thanks for adding test data. How many MBs of data have we added with this PR?

@dylanlee
Copy link
Contributor Author

dylanlee commented Jul 9, 2025

82.7 mb. It is mostly from the HAND data. Specifically the REM tif

dylanlee added 2 commits July 9, 2025 17:14
- update local job definition to work with local containers
- fix broken data links in test hand index parquet
- update local nomad config to allow local containers to mount volumes
- updated code in a few places where paths only worked with s3
- tested that pipeline works with local data and all local containers
Can't create self hosted runner

fix docker compose in workflow

update STAC api health check

add additional debugging to workflow

more workflow debugging

fix postgres directory permissions in workflow

change permissions on pgstac-data directory

more permission checks on pgstac data dir

load bench db on the fly

update nomad job registration

add debugging to pipeline step

update workflow

workflow edits

fix hardcoded paths

workflow tweaks

modify job definitions to actually be able to access test data without
hard coded paths

workflow tweaks

add debug to job registration

workflow tweaks

workflow tweaks

Remove pgstac-data directory and add to .gitignore

tweaked resource usage for github runner

bump pipeline container alloc

at pipeline output stage

fix output pathing in workflow
remove debugging don't build uncessary containers etc
ar-siddiqui and others added 6 commits July 15, 2025 02:27
Edited readme to provide more context on .env conventions.

Added an additonal docker compose service to post test STAC
collection and item to STAC api

Updated hand index parquet to reflect new testdata directory structure

Updated local job definitions to reflect new testdata directory
structure
Modify refactor to sucessfully complete local test run
Refactor local deployment setup
@dylanlee dylanlee merged commit 9935464 into main Jul 15, 2025
@dylanlee dylanlee deleted the local-test-data branch July 15, 2025 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants