Supporting persistent references

[knoepfel]

As discussed at the July 2025 developer retreat (FNAL SSO required), we believe we need to support persistent references (i.e. the equivalent of art::Ptr<T> objects). We considered several options for creating the references:

1. ❌ By index, framework independent (creation & input): framework independent. Data-product usage explicit (clear memory management semantics). Configuration is verbose, code is verbose. Inefficiency of reading in all hit collections, not just the ones dereferenced. Alter allowed signatures of algorithms to be registered. Will not pursue.

2. ❌ By index, framework dependent (creation & input): framework independent. Data-product usage explicit (clear memory management semantics). Configuration is verbose, code could depend on framework helpers to be less verbose. Inefficiency of reading in all hit collections, not just the ones dereferenced. Alter allowed signatures of algorithms to be registered. Will not pursue.

3. ❌ By reference, framework independent (creation & input): Retain framework independence, but writing data translating from framework-independent references to persistable references is untenable. Reading is possible. Will not pursue.

4. 👀 ✅ By reference, framework dependent (creation & input): Lose framework independence, but based on well-known approach that supports inter-product references within the same data cell. Will explore.

   1. Requires inter-data-cell (and inter-data-layer) referencing.

5. 👀 By reference, framework-dependent (creation) and -independent (input): Conceptual data products that have persistent references as data members must be created according to a layout that is framework dependent; such products could be used/read according to a layout that is framework-independent. Will explore.

   1. Requires 4.
   2. Should consider using IDL to do this.

6. ✅ By value: No inter-product reference necessary. Data product (element) is copied. Memory management could be an issue. Data duplication on file is an issue. Allowed.

   1. Give guidance re. the consequences (both good and bad) of doing this.

Of the six options, only options 4–6 are candidates for further consideration. Options 4 and 6 will be supported, with option 4 requiring further research to determine how to support inter-data-cell references. Option 5 is attractive but requires research.

[knoepfel]

Related to:

• Support inter-data-cell persistent references phlex#75
• Explore translating persistent references to framework-independent input-parameter types phlex#74