Performance HOP on Linux

[EricTonissen]

I have a HOP workflow running on windows with some performance.
Running the same workflow on Linux, it takes much more time to finish.
In the logging you see on several places that there is no logline for 1 sec.
Somebody any idea what I can do to have better performance on Linux?

Compare Pentaho vs HOP

PENTAHO Windows : Spoon - Processing ended after 5 seconds
PENTAHO Linux : Kitchen - Processing ended after 5 seconds
HOP Windows : Workflow duration : 6.628 seconds [ 6.628" ]
HOP Linux : Workflow duration : 12.382 seconds [ 12.382" ]

HOP Linux logfile

2025/12/03 20:28:56 - HopRun - Enabling project 'eridani'
2025/12/03 20:28:56 - HopRun - Starting workflow: /opt/hop/eridani-base/pdi-repo/jobscheduler/scheduler/util/jb_scheduler_report.hwf
2025/12/03 20:28:57 - jb_scheduler_report - Start of workflow execution
2025/12/03 20:28:57 - jb_scheduler_report - Starting action [START]
2025/12/03 20:28:58 - jb_scheduler_report - Finished action [START] (result=[true])
2025/12/03 20:28:58 - jb_scheduler_report - Starting action [Common environment]
2025/12/03 20:28:58 - jb_scheduler_report - Finished action [Common environment] (result=[true])
2025/12/03 20:28:58 - jb_scheduler_report - Starting action [Property file?]
2025/12/03 20:28:58 - jb_scheduler_report - Finished action [Property file?] (result=[false])
2025/12/03 20:28:58 - jb_scheduler_report - Starting action [Env Property file?]
2025/12/03 20:28:58 - jb_scheduler_report - Finished action [Env Property file?] (result=[false])
2025/12/03 20:28:58 - jb_scheduler_report - Starting action [Report queue]
2025/12/03 20:28:58 - Report queue - Using run configuration [local]
2025/12/03 20:28:58 - tr_scheduler_report - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2025/12/03 20:28:58 - tr_scheduler_report - Execution started for pipeline [tr_scheduler_report]
2025/12/03 20:29:02 - Seq Html per ehs.0 - init counter name: @@sequence:91e4786a-6d98-4426-9a09-7a89a14aa98c:seqProcess
2025/12/03 20:29:02 - Seq Not success.0 - init counter name: @@sequence:91e4786a-6d98-4426-9a09-7a89a14aa98c:seqNotSucces
2025/12/03 20:29:02 - Seq Waiting/running.0 - init counter name: @@sequence:91e4786a-6d98-4426-9a09-7a89a14aa98c:seqWaitingRunning
2025/12/03 20:29:02 - Examples.0 - Finished processing (I=0, O=0, R=0, W=12, U=0, E=0)
2025/12/03 20:29:02 - Add 18.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:29:02 - Get TS.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:29:02 - Get Variables.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:29:04 - Dummy (do nothing).0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:29:04 - Calc days.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:29:04 - Html html.0 - Finished processing (I=0, O=0, R=1, W=30, U=0, E=0)
2025/12/03 20:29:04 - Select Html html.0 - Finished processing (I=0, O=0, R=30, W=30, U=0, E=0)
2025/12/03 20:29:04 - Calc startTS.0 - Finished processing (I=0, O=0, R=1, W=5, U=0, E=0)
2025/12/03 20:29:04 - Get Queue.0 - Finished processing (I=32, O=0, R=1, W=32, U=0, E=0)
2025/12/03 20:29:04 - Get QueueLogArchive.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
2025/12/03 20:29:04 - Minimal fields from Queue.0 - Finished processing (I=0, O=0, R=32, W=32, U=0, E=0)
2025/12/03 20:29:05 - Get process.0 - Finished processing (I=32, O=0, R=32, W=32, U=0, E=0)
2025/12/03 20:29:05 - Join examples.0 - Finished processing (I=0, O=0, R=13, W=12, U=0, E=0)
2025/12/03 20:29:05 - Catch examples.0 - Finished processing (I=0, O=0, R=12, W=12, U=0, E=0)
2025/12/03 20:29:05 - Get QueueLog.0 - Finished processing (I=118, O=0, R=1, W=118, U=0, E=0)
2025/12/03 20:29:05 - Sort on processID.0 - Finished processing (I=0, O=0, R=118, W=118, U=0, E=0)
2025/12/03 20:29:05 - Not last based on processID.0 - Finished processing (I=0, O=0, R=52, W=52, U=0, E=0)
2025/12/03 20:29:05 - Unique on processID.0 - Finished processing (I=0, O=0, R=118, W=66, U=0, E=52)
2025/12/03 20:29:05 - Minimal fields from QueueLog.0 - Finished processing (I=0, O=0, R=66, W=66, U=0, E=0)
2025/12/03 20:29:05 - Rename config_name.0 - Finished processing (I=0, O=0, R=98, W=98, U=0, E=0)
2025/12/03 20:29:05 - Get config_name.0 - Finished processing (I=93, O=0, R=98, W=98, U=0, E=0)
2025/12/03 20:29:07 - Status to color.0 - Finished processing (I=0, O=0, R=98, W=98, U=0, E=0)
2025/12/03 20:29:07 - Filter on config_name.0 - Finished processing (I=0, O=0, R=98, W=98, U=0, E=0)
2025/12/03 20:29:07 - Filter on name.0 - Finished processing (I=0, O=0, R=98, W=98, U=0, E=0)
2025/12/03 20:29:07 - Seq Not success.0 - Finished processing (I=0, O=0, R=12, W=12, U=0, E=0)
2025/12/03 20:29:07 - Sort on sequence.0 - Finished processing (I=0, O=0, R=191, W=191, U=0, E=0)
2025/12/03 20:29:07 - Select not success.0 - Finished processing (I=0, O=0, R=12, W=12, U=0, E=0)
2025/12/03 20:29:07 - Switch / case.0 - Finished processing (I=0, O=0, R=43, W=43, U=0, E=0)
2025/12/03 20:29:07 - Select waiting/running.0 - Finished processing (I=0, O=0, R=39, W=39, U=0, E=0)

HOP Windows logfile

2025/12/03 20:56:53 - Hop - Starting workflow
2025/12/03 20:56:53 - jb_scheduler_report - Start of workflow execution
2025/12/03 20:56:53 - jb_scheduler_report - Starting action [START]
2025/12/03 20:56:53 - jb_scheduler_report - Finished action [START] (result=[true])
2025/12/03 20:56:53 - jb_scheduler_report - Starting action [Common environment]
2025/12/03 20:56:53 - jb_scheduler_report - Finished action [Common environment] (result=[true])
2025/12/03 20:56:53 - jb_scheduler_report - Starting action [Property file?]
2025/12/03 20:56:53 - jb_scheduler_report - Finished action [Property file?] (result=[false])
2025/12/03 20:56:53 - jb_scheduler_report - Starting action [Env Property file?]
2025/12/03 20:56:53 - jb_scheduler_report - Finished action [Env Property file?] (result=[false])
2025/12/03 20:56:53 - jb_scheduler_report - Starting action [Report queue]
2025/12/03 20:56:53 - Report queue - Using run configuration [local]
2025/12/03 20:56:53 - tr_scheduler_report - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2025/12/03 20:56:53 - tr_scheduler_report - Execution started for pipeline [tr_scheduler_report]
2025/12/03 20:56:54 - Seq Html per ehs.0 - init counter name: @@sequence:1f70e235-5be9-4bd0-93c9-cb25ec06c8b2:seqProcess
2025/12/03 20:56:54 - Seq Not success.0 - init counter name: @@sequence:1f70e235-5be9-4bd0-93c9-cb25ec06c8b2:seqNotSucces
2025/12/03 20:56:54 - Seq Waiting/running.0 - init counter name: @@sequence:1f70e235-5be9-4bd0-93c9-cb25ec06c8b2:seqWaitingRunning
2025/12/03 20:56:55 - Examples.0 - Finished processing (I=0, O=0, R=0, W=12, U=0, E=0)
2025/12/03 20:56:55 - Get Variables.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:56:55 - Get TS.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:56:55 - Add 18.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:56:55 - Calc days.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:56:55 - Calc startTS.0 - Finished processing (I=0, O=0, R=1, W=5, U=0, E=0)
2025/12/03 20:56:55 - Dummy (do nothing).0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2025/12/03 20:56:55 - Html html.0 - Finished processing (I=0, O=0, R=1, W=30, U=0, E=0)
2025/12/03 20:56:55 - Select Html html.0 - Finished processing (I=0, O=0, R=30, W=30, U=0, E=0)
2025/12/03 20:56:55 - Get QueueLogArchive.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
2025/12/03 20:56:56 - Get Queue.0 - Finished processing (I=6, O=0, R=1, W=6, U=0, E=0)
2025/12/03 20:56:56 - Join examples.0 - Finished processing (I=0, O=0, R=13, W=12, U=0, E=0)
2025/12/03 20:56:56 - Catch examples.0 - Finished processing (I=0, O=0, R=12, W=12, U=0, E=0)
2025/12/03 20:56:56 - Get process.0 - Finished processing (I=6, O=0, R=6, W=6, U=0, E=0)
2025/12/03 20:56:56 - Minimal fields from Queue.0 - Finished processing (I=0, O=0, R=6, W=6, U=0, E=0)
2025/12/03 20:56:56 - Get QueueLog.0 - Finished processing (I=148, O=0, R=1, W=148, U=0, E=0)
2025/12/03 20:56:56 - Sort on processID.0 - Finished processing (I=0, O=0, R=148, W=148, U=0, E=0)
2025/12/03 20:56:56 - Unique on processID.0 - Finished processing (I=0, O=0, R=148, W=66, U=0, E=82)
2025/12/03 20:56:56 - Not last based on processID.0 - Finished processing (I=0, O=0, R=82, W=82, U=0, E=0)
2025/12/03 20:56:56 - Minimal fields from QueueLog.0 - Finished processing (I=0, O=0, R=66, W=66, U=0, E=0)
2025/12/03 20:56:56 - Get config_name.0 - Finished processing (I=68, O=0, R=72, W=72, U=0, E=0)
2025/12/03 20:56:56 - Rename config_name.0 - Finished processing (I=0, O=0, R=72, W=72, U=0, E=0)
2025/12/03 20:56:58 - Not nulls.0 - Finished processing (I=0, O=0, R=72, W=72, U=0, E=0)
2025/12/03 20:56:58 - Filter on config_name.0 - Finished processing (I=0, O=0, R=72, W=72, U=0, E=0)
2025/12/03 20:56:58 - Filter on name.0 - Finished processing (I=0, O=0, R=72, W=72, U=0, E=0)
2025/12/03 20:56:58 - Status to color.0 - Finished processing (I=0, O=0, R=72, W=72, U=0, E=0)
2025/12/03 20:56:58 - Sort config_name.0 - Finished processing (I=0, O=0, R=72, W=144, U=0, E=0)
2025/12/03 20:56:58 - Html per ehs.0 - Finished processing (I=0, O=0, R=72, W=84, U=0, E=0)
2025/12/03 20:56:58 - Filter rows.0 - Finished processing (I=0, O=0, R=72, W=16, U=0, E=0)
2025/12/03 20:56:58 - Seq Html per ehs.0 - Finished processing (I=0, O=0, R=84, W=84, U=0, E=0)
2025/12/03 20:56:58 - Switch / case.0 - Finished processing (I=0, O=0, R=16, W=16, U=0, E=0)
2025/12/03 20:56:58 - Waiting/running.0 - Finished processing (I=0, O=0, R=9, W=13, U=0, E=0)
2025/12/03 20:56:58 - Not success.0 - Finished processing (I=0, O=0, R=7, W=11, U=0, E=0)
2025/12/03 20:56:58 - Select Html per ehs.0 - Finished processing (I=0, O=0, R=84, W=84, U=0, E=0)
2025/12/03 20:56:58 - Seq Waiting/running.0 - Finished processing (I=0, O=0, R=13, W=13, U=0, E=0)
2025/12/03 20:56:58 - Seq Not success.0 - Finished processing (I=0, O=0, R=11, W=11, U=0, E=0)
2025/12/03 20:56:58 - Select not success.0 - Finished processing (I=0, O=0, R=11, W=11, U=0, E=0)
2025/12/03 20:56:58 - Select waiting/running.0 - Finished processing (I=0, O=0, R=13, W=13, U=0, E=0)

[bamaer]

What are your configurations, system resources, Xmx values etc, and are those the same for Windows and Linux?

[dmainou]

ok... you can configure a lot of that as part of your config files and system variables outside hop (e.g. .bashrc)
could make your life easier.

[EricTonissen]

I know, but for me is now the most important part to check whether it will be possible to convert Pentaho (job/transformations) to HOP (workflow/pipelines). But if the performance is a lot worse, then it makes no sense for me to put more time in it. I am now busy with a more time consuming Pentaho job to convert to HOP, and I will see if this gives some better performance for HOP, than the small workflow did.

[dmainou]

I find execution times comparable on every job I have ported.

More, there are a million things that can be giving you this extra 6 seconds.

Please elaborate what your pipeline is doing., how you are triggering it, what mounts, type of mounts, number of steps, etc.

Which pentaho version, what drivers, etc.
I feel it takes 5 seconds just to start kitchen or hop, let alone processing anything.

So are you processing 5 rows or 5 million rows.

What is the use case.

[bamaer]

@dmainou @EricTonissen we're probably talking about startup time rather than "real" performance here. I guess performance would be very similar (and hopefully better in Hop) for longer running workflows/pipelines, but Hop has a penalty in startup time that PDI didn't have, especially in old versions (pre 8.x). We can probably shave off quite a bit of annotation scanning etc to significantly improve the startup times. I know @hansva has already been doing some work on this that will probably end up in 2.17.0.

[EricTonissen]

I am not sure whether I can agree.

I suppose that the startup from HOP is finished before the logging of 'Start of workflow execution'.
The same is the startup from Pentaho before the logging of 'Start of job execution'.

Below is the logging from both HOP and Pentaho.

It takes now more time than in previous compare, because there is more data to process.
But they both process same data, doing the same things, and both gives same results.
Both processes running on same machines, same memory parameters, and so on.
Only different java version, as mentioned before.
And running several times this processes, they all give the similar results.

2025/12/08 12:17:10 - HopRun - Enabling project 'eridani'
2025/12/08 12:17:10 - HopRun - Starting workflow: /opt/hop/eridani-base/pdi-repo/jobscheduler/scheduler/util/jb_scheduler_report.hwf
2025/12/08 12:17:10 - jb_scheduler_report - Start of workflow execution

2025/12/08 12:17:35 - jb_scheduler_report - Finished action [Report queue] (result=[true])
2025/12/08 12:17:35 - jb_scheduler_report - Workflow execution finished
2025/12/08 12:17:35 - jb_scheduler_report - Workflow duration : 24.867 seconds [ 24.867" ]

2025/12/08 12:15:58 - Kitchen - Logging is at level : Basic
2025/12/08 12:15:58 - Kitchen - Start of run.
2025/12/08 12:15:58 - RepositoriesMeta - Reading repositories XML file: /home/eridanivriend/.kettle/repositories.xml
2025/12/08 12:16:01 - jb_scheduler_report - Start of job execution

2025/12/08 12:16:16 - jb_scheduler_report - Job execution finished
2025/12/08 12:16:16 - Kitchen - Finished!
2025/12/08 12:16:16 - Kitchen - Start=2025/12/08 12:16:01.876, Stop=2025/12/08 12:16:16.916
2025/12/08 12:16:16 - Kitchen - Processing ended after 15 seconds.