Skip to content

ETL Tests #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
vladd-bit wants to merge 33 commits into
base: main
Choose a base branch
from
Open

ETL Tests #47

vladd-bit wants to merge 33 commits into from

Conversation

@vladd-bit
Copy link
Member

@vladd-bit vladd-bit commented Nov 27, 2025 •
edited
Loading

This PR attempts to accomplish the following:

  • health checks for all core data-infra services: NiFi, NiFi-Registry, NGINX(Optional), Elastic, PgSQL, Kibana
  • tests for the provided default NiFi (v2.0+) ingestion workflows.
  • refactoring of older scripts and code-base for cleaner scripting
  • tests for some provided on-the-fly scripts in "nifi/user-scripts"
  • better environment variable validation & handling
  • additions to resource limits & env vars for each of the core containers (NiFi, NiFi Registry, Nginx, Metric/file(beats), Kibana, Gitea, Elastic, PgSQL

@vladd-bit
Deploy: added env var checkup script.
3980318
@vladd-bit
NiFi: updated docker image + packge list to install.
3318a4e
@vladd-bit
Merge branch 'main' of https://github.com/CogStack/CogStack-NiFi into…
a20cf8b 
… etl_tests
@vladd-bit
NiFi: dockerfile fix.
aa306b9
@vladd-bit
Deploy refactioring + pyproject.
b4ef495
@vladd-bit
.gitignore update.
496c612
@vladd-bit
Deploy: added resource limits to core services (NiFi/NiFiRegistry/Ngi…
3e0f848 
…nx/Kibana/ES/Metricbeat/FileBeat}
@vladd-bit
Deploy: env + services def updates (resource constraints per service).
c0f81b8
@vladd-bit
NiFi: dockerfile update.
5e0894c
@vladd-bit
Transioning NiFi scripts to python package.
9eec629
@vladd-bit
Re-added user scripts/templates/python_extensions folders.
66bfc7d
@vladd-bit
Re-added user_scripts folder. Added configs for pgsql/nifi/elastic se…
9f7e697 
…rvices + health tests.
@vladd-bit
NiFi: updated user_scripts.
05fc3b3
@vladd-bit
Security: fixed script typo.
ef26a79
@vladd-bit
Merge branch 'main' of https://github.com/CogStack/CogStack-NiFi into…
3e5cccb 
… etl_tests
@vladd-bit
Dev requirements + docker template gha update.
5c62a3c
@vladd-bit
Merge branch 'main' of https://github.com/CogStack/CogStack-NiFi into…
fe0b072 
… etl_tests
@vladd-bit
Update submodules to latest release tags (or main)
2af6cd8
@vladd-bit
Deploy: updated docker services restart flag on core services.
c730cf3
@vladd-bit
.gitignore update.
f3b805f
@vladd-bit
NiFi: updated processor user_scripts path ref.
b8df82e
@vladd-bit
Updated path refs.
f43467f
@vladd-bit
Updated path refs to NiFi scripts folders.
48a06ee
@vladd-bit
Updated path refs to NiFi scripts folders II.
25d5be0
@vladd-bit
Updated .gitattributes.
e45d5c4
@vladd-bit vladd-bit self-assigned this Dec 10, 2025
@vladd-bit vladd-bit added enhancement New feature or request help wanted Extra attention is needed github_actions Pull requests that update GitHub Actions code and removed help wanted Extra attention is needed labels Dec 10, 2025
@vladd-bit vladd-bit added the python Pull requests that update python code label Dec 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vladd-bit vladd-bit

Labels

enhancement New feature or request github_actions Pull requests that update GitHub Actions code python Pull requests that update python code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vladd-bit