Reduce repository size by moving camera firmware blobs

[paolostivanin]

The cameras/ directory includes several firmware blobs that make the PHD2 repo quite large and add big binary deltas to the git history. Most users (and many contributors) don’t actually need these blobs in their checkout.

Would you consider moving the firmware into a small separate repository and pulling it in as a git submodule?

Proposed approach

1. Create a phd2-firmware repo with only the binary blobs.
2. Replace the current firmware files in cameras/* with a submodule pointing to it.
3. Keep paths the same so builds and runtime behavior don’t change.

Benefits

1. Smaller, faster-to-clone main repo
2. No binary bloat in git history
3. Cleaner separation of code vs. firmware
4. Easier packaging for distros

If this sounds good, I can prepare a PR with the full transition.

Thanks for considering it!

[agalasso]

@paolostivanin I'm open to the idea but the devil is in the details and we need to consider all the use cases.

Managing the external dependencies in PHD2 is a big rat's nest--numerous open and closed camera SDKs x 3 OS platforms (Windows, Mac (2-3 variants), Linux (multiple variants))

1. some key contributors work on Windows only and I want to keep the friction for them as low as possible. git clone --recursive may not be too much to ask, but it would need to be documented and communicated. The binary blobs are pretty much all required on Windows, so these folks won't see much benefit of the change (and would experience problems if they forget --recursive)
2. Some camera vendors submit PRs to phd2, they'll need to change to make prs to the new repo
3. Other camera vendors email me and ask me to download new camera SDKs from their website; there are a number of scripts in build/ that I use to unpack the SDKs and import them into PHD2. These would need to be updated and moved over to the new repo.
4. The release build process would need to be updated (maybe simply add --recursive) and tested.

An alternative to submodules that we could consider would be to move the blobs to a separate repo as you suggest and use CMake's FetchContent to pull them into the builds as needed. We already do this with OGMA cameras. We could repeat that pattern for all the closed source SDKs in cameras/ and have a nice separation between the open source cameras (kept in the phd2 repo) and closed ones (in the binary blobs repo). We'll still need to consider the items above, but at least item (1) would go away (become transparent -- part of running cmake). Hmm, I'm starting to think this could be a good approach...

[paolostivanin]

Thanks for the detailed context, that all makes sense.

You’re absolutely right that submodules would add some friction, especially for Windows-only contributors and occasional vendors. I like your idea of reusing the OGMA FetchContent pattern and keeping things transparent at the "run CMake and it just works" level.

To check I'm aligned, the rough shape I had in mind would be:

1. Create a separate phd2-firmware (or similar) repo containing only the closed camera SDK blobs.
2. In cameras/, keep just any needed code
3. For the closed SDKs, replace the committed blobs with CMake FetchContent logic:
   • On configure, CMake pulls the matching blobs from the firmware repo.
   • Paths on disk are kept compatible (same include/lib layout) so build logic and runtime behavior stay unchanged.
4. Update the existing scripts in build/ so that when you import new SDKs from vendors, they populate the firmware repo instead of the main tree (or at least have a clear “export to firmware repo” step).

That would give:

1. Transparent behavior for all platforms: no need to remember --recursive.
2. A smaller, faster-to-clone main repo with a clean history.
3. A clear split between "Core PHD2 + open SDKs" (main repo) and "Closed binary SDKs" (firmware repo, plugged in via CMake).

What do you think? I’m happy to help in any way - prototype a small example, draft a CMake module, or test the workflow (Linux only) if that would be useful.

[agalasso]

@paolostivanin thanks for your input & offer to prototype. Let's let this idea simmer for a while.

Bruce and I are both currently working on some other issues at the moment, and the timing is not good for making changes to our current workflows. We can come back to this in a few weeks when the other stuff clears up.

It probably makes more sense for me to work on the implementation since I have the Windows, Mac and Linux build/test environments in place, and the changes need to be tested on all platforms. Also, the switch-over would need to be coordinated between the new repo, any new SDKs released by the camera vendors, and the PHD2 release schedule.

[paolostivanin]

Thanks for considering the idea. No worries at all about the timing or the waiting, after all it’s just a minor non-user visible improvement :)