Skip to content

Set compression level to analysis (LZ4).#117

Open
inkdot7 wants to merge 1 commit intoISOLDESolenoidalSpectrometer:mainfrom
inkdot7:compression
Open

Set compression level to analysis (LZ4).#117
inkdot7 wants to merge 1 commit intoISOLDESolenoidalSpectrometer:mainfrom
inkdot7:compression

Conversation

@inkdot7
Copy link
Contributor

@inkdot7 inkdot7 commented Jun 30, 2025

Default seems to be zlib. For smaller files, one might want to try

This was not as big a win as I would have hoped:

old:
real    1m4,279s
user    0m54,924s
sys     0m2,138s

new (kUseAnalysis):
real    1m1,991s
user    0m50,480s
sys     0m2,087s

other (kUseGeneralPurpose)
real    1m2,288s
user    0m53,618s
sys     0m2,055s

It trades speed for file size:

old:
-rw-r----- 1 f96hajo f96hajo 289259687 30 jun 04.28 R69_0.root
-rw-r----- 1 f96hajo f96hajo 175381347 30 jun 04.28 R69_0_events.root
-rw-r----- 1 f96hajo f96hajo      1445 30 jun 04.28 R69_0_events.log
-rw-r----- 1 f96hajo f96hajo  35727949 30 jun 04.28 R69_0_hists.root

new (kUseAnalysis):
-rw-r----- 1 f96hajo f96hajo 309561887 30 jun 04.22 R69_0.root
-rw-r----- 1 f96hajo f96hajo 330499708 30 jun 04.23 R69_0_events.root
-rw-r----- 1 f96hajo f96hajo      1445 30 jun 04.23 R69_0_events.log
-rw-r----- 1 f96hajo f96hajo  33897605 30 jun 04.23 R69_0_hists.root

other (kUseGeneralPurpose)
-rw-r----- 1 f96hajo f96hajo 240991275 30 jun 04.36 R69_0.root
-rw-r----- 1 f96hajo f96hajo 200734560 30 jun 04.36 R69_0_events.root
-rw-r----- 1 f96hajo f96hajo      2890 30 jun 04.36 R69_0_events.log
-rw-r----- 1 f96hajo f96hajo  11622583 30 jun 04.36 R69_0_hists.root

I'm running this on an NFS machine, so may be hampered by that. Perhaps you get different results.
I.e. putting this up more for testing than directly advocating it.

@inkdot7
Copy link
Contributor Author

inkdot7 commented Jun 30, 2025

Another thing might be: The output of the converter is what the event builder consumes?
The event builder could have an option to use the data packets directly from the converter vector, instead of the converter first writing them to a root file and reading them again?
If so, the converter data could be written to file as usual, but in a parallel thread while the event builder (and histogrammer) is running.

@lpgaff
Copy link
Contributor

lpgaff commented Jul 1, 2025

Another thing might be: The output of the converter is what the event builder consumes? The event builder could have an option to use the data packets directly from the converter vector, instead of the converter first writing them to a root file and reading them again? If so, the converter data could be written to file as usual, but in a parallel thread while the event builder (and histogrammer) is running.

This is not a quick thing to do, but it is of course possible. In the case where we sort from scratch, rather than just making the event building from file, we can of course keep the vector in memory and pass it from the Converter class to the EventBuilder class. The way it is structured right now though, the converter is destroyed once the execution is finished, taking all of the memory with it, and only then is the event builder created. Needs some thought... Especially as running them all simultaneously adds to the memory pressure (lot's of histograms as well as the trees).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants