Sha 256 support

[Byron]

Even though the foundation is set, it needs another push to actually make it work with different kinds of hashes.

Tasks

• remove hash-type specific methods from git-hash and replace them with parametric usage of git_hash::Kind
• all code assuming hashes of len 20 should receive this value as parameter instead. This is what git does for the old index and pack file formats.
• a way to pass --object-hash information to the gix CLI
• Remove default sha1 feature from gix-hash crate and deal with the fallout
  • Add forwarding features to every plumbing crate that depends on it directly or indirectly
  • adjust tests to set sha1 by default
  • configure docs.rs to enable sha1
  • CI should execute script that runs cargo check -p gix-<name> to show it's failing with an error message
  • adjust all other jobs
  • let gix chose SHA1 as default
• remove SHA1 mention from git-features feature toggles
• parameterize hash len when decoding non-blob objects (see this for an example)
• understand and implement pack idx V3. - see if git actually implements this, and maybe decide that gitoxide won't handle the transition period, is either one has or another.
• add new Sha256 enum variant, consider putting it behind a feature flag, and add a hasher for it as well.
• general tests for reading refs and objects of different len
• tests for writing and reading objects of different len, maybe even write a conversion program which transforms an entire repo and double-checks with git-fsck
• when cloning, check the uninmplemented!() invocation to configure the repo for expecting a different hash

Implementation ideas

• make sure once Sha256 is added as ObjectId variant, that it's behind a feature toggle to allow builds that opt-out of SHA256 support to not unnecessarily use more memory than needed. Maybe there are alternatives to this, too.

• One way to do that with approximately zero overhead would be to such functions generic on the object ID, using a trait that has a method to get the type. Then object IDs with a known type return a constant from that method, and object IDs with a runtime dispatched type return the value of that enum.

  • @joshtriplett - taken verbatim as I'd barely be able to improve on it when paraphrasing. In short, have a trait for oid or allow efficient conversions to oid (it's just a slice, so that should work for specifically sized types as well especially if these were provided by git-hash.

Notes

• find ways to use the existing highly-parallel pack traversal (along with integration of loose-objects) to build an inverse-ref table to quickly traverse objects bottom-up to change the hash used along with all references, while being fast. This ties into being able to build new packs quickly, ideally even with delta-compression (the latter then has to be re-created as most objects actually change) - re-using deltas for blobs is the only way.
  • The existing traversal can mutate data in the tree, which is enough to decode the object and keep direct references for later.

Related

• Make sha1 optional in gix-hash #2332
• feat: add sha256 feature to gix-object #2359
• feat: add sha256 feature to gix-commitgraph #2377

[shea256]

+1 to this feature. Excited to see this implemented.

[msrd0]

+1. Given that git no longer considers sha256 support experimental and forges such as forgejo added support, this has became more relevant for new git repositories.

[Sturdy7435]

+1. Without sha256 support Starship prompt cannot detect repositories using that algorithm.

[jokeyrhyme]

@Byron

make sure once Sha256 is added as ObjectId variant, that it's behind a feature toggle to allow builds that opt-out of SHA256 support to not unnecessarily use more memory than needed. Maybe there are alternatives to this, too.

should the old SHA1 support also be put behind a feature toggle (enabled by default), so that builds can choose to only support SHA256 ?

[Byron]

+1. Without sha256 support Starship prompt cannot detect repositories using that algorithm.

Starship would definitely be a great motivation to finally go through with this.

gix-hash has been created with that in mind:

gitoxide/gix-hash/src/object_id.rs

Lines 13 to 14 in ec69c88

	/// A SHA 1 hash digest
	Sha1([u8; SIZE_OF_SHA1_DIGEST]),

Right now it's an enum and one could add another variant to it, behind a feature toggle, as the size of these hash digests definitely affects the resource requirements of operations like clone. As the size of these hashes would increase, this would affect the memory consumption even when cloning SHA1 repositories (unfortunately).
Maybe there are better ways to represent this - I think Git simply allocates for it which would be one solution here as well.

In any case, if this were to move forward, I think it would be good to review the hash representation first, as it's very relevant for performance.

@Byron

make sure once Sha256 is added as ObjectId variant, that it's behind a feature toggle to allow builds that opt-out of SHA256 support to not unnecessarily use more memory than needed. Maybe there are alternatives to this, too.

should the old SHA1 support also be put behind a feature toggle (enabled by default), so that builds can choose to only support SHA256 ?

I see the point and gitoxide should allow to be tailored to what application developers need. However, it by default it will try to be compatible to Git which means SHA1 and SHA256 must be supported, which is certainly what this would start out as.
Once there is explicit demand, I think a feature toggle for turning off SHA1 support could be contributed. This will be some work though, as the test-suite directly refers to it in plenty of places.