Tutorial for Hop development to production workflow

[jrslhelcom]

Originally posted by @josersleal in #6151

I work for a non profit and am trying to create a demo that will help scientists use HOP.
The concept is to develop using the GUI locally, run and test on a remote Hop server, deployed with docker (long running/always available).
Github used extensively
the demo should read and write to a postgresql db, also deployed to docker
docker compose is orchestrating
The create workflow and pipelines should run using GUI and API calls for integration to other software being built
Once tests pass, the project is promoted to QA and later to Production.

Some problems, being the most evident the fact that running a pipeline directly from the GUI yields a good output to the remote database using Hop server remote
running a workflow that runs the same pipeline fails. This uses the same environment as the pipeline.

[jrslhelcom]

demo-hop-remote-docker.zip
here is the entire project.

[josersleal]

Again, thank you for your time.
"a)" is wobly. there are 2 areas to set a run configuration so...
"c)" is interesting.
Deploying a permanent, or long running server using docker, seems to be tricky. is there a practical example how to achieve this?

[bamaer]

a) It's not "wobly", really. You specify a run configuration for the platform you're executing from. In the case of a remote run configuration, you also specify a run configuration that will be used on that remote server. In most cases, that is local, which means local on that remote server.
c) Your workflows, pipelines and input files will still need to be available on the server, but it will probably be easier once PR6198 is merged and available in Apache Hop 2.17.0. You're not the only one struggling with this.

[jrslhelcom]

thank you for that.
as I see it, I might have missed it. this is the pipeline run configuration form:

so there is a engine type, a server and yet another (the wobbly part) run configuration.
The documentation is unclear to me. I think it needs a few examples explaining the combinations possible.
concerning the running on a remote dockerized or not server, the issue of making the files accessible is huge as it renders all null, unusable. (maybe not so dramatic)
the root folder, held by a variable used in the gui, only exists for the gui. yet all other depend on it. this value will NEVER be the same in 2 different systems, because its Hop installation available only. This is a major issue. I don't see in the docs how to make files available to different systems/environments.

So the question is: why is the excel input taking a variable relative URL to the used file(s) when that variable value is never passed to the execution engine? At least for a workflow.

The pipeline sems to work. How come it knows where the excel is and the workflow running the same pipeline does not know it?

[bamaer]

Each and every documentation page has this "edit this page" button that makes it as easy as possible to contribute and improve the documentation:

Regarding the files and availability: you're using a local Excel file. It would be perfectly fine to send these over the next work to the server. That wouldn't work if you have multiple GBs or even TBs of data to process. A mount in your container could be a quick and easy solution for your problem. You can map source and target folders in your remote run configuration, I point to that earlier.

The pipeline seems to work: is that local or remote? If remote: did you change the "required" flag to "Y" and/or did you verify if the data gets actually loaded to your table?

[jrslhelcom]

some success. I earlier could not understand the setting "named resources ref.."
its exactly the item needed.
a folder is mapped using a volume to the container
projetcs are subfolders of it.
use the setting "named resources ref.." to indicate for each run config at each project source folder at GUI OS and target folder at container as root for project. Did I get it right?

fact is that yes I can verify the rows are in the table inside the postgresql container.

some more steps and details to go through in this demo/tut but for now thank you very much for your patient and time