Allow optional commit and tag metadata in Manifests and registries

[simonbyrne]

Currently we only identify versions by their git-tree-sha1. However this is sub-optimal when looking up git histories: GitHub doesn't provide a convenient way to find commits of a given tree, which means that e.g. TagBot has to jump through all sorts of funny hoops to try to link the tag back to a given commit.

I propose the following:

• Registry Versions.toml and Manifest.toml allow optional git-commit-sha1, git-tag-sha1 (for Annotated Tags only) and git-tag-name (for the tag reference) that link to the corresponding objects
• In the case of subpackages, we can have an optional git-tree-path giving the path in the commit/tag to the corresponding tree.
• Unlike git-tree-sha1, these should be considered mutable (i.e. registries may add/update/remove these as required)

cc: @IanButterworth

Current PRs:

• add commit hash information to registry JuliaRegistries/RegistryTools.jl#97
• add commit hash information to registry JuliaRegistries/Registrator.jl#429 (requires above PR)
• add support for commit hash information JuliaRegistries/RegistryCI.jl#537 (independent, but need to agree on names)

[simonbyrne]

Another use case would be rewriting file paths in CI stacktraces so that we can provide a HTML link to the GitHub URL

[DilumAluthge]

Can we split this into two separate issues, one for the General registry (Versions.toml), and a separate one for local Manifest.toml files? It seems to me that those two can be implemented independently of each other.

[simonbyrne]

I think you would need the registry one first, no?

[simonbyrne]

What would actually need to be done for the registry changes?

From what I can tell, it seems that:
https://github.com/JuliaRegistries/RegistryTools.jl/blob/77e2a02e62185ce865653bdae95203a3a40510f0/src/register.jl#L326
would need to be updated, along with Registrator.jl?

[DilumAluthge]

I think you would need the registry one first, no?

Oh, I was thinking that the manifest would be getting its info from a local Git clone. But yeah, if the plan is for the manifest to get the info from the registry, then we first need to implement this in the registry.

[KristofferC]

Makes sense to me to have optional metadata tied to versions that can be used to improve various tooling. You would have to verify that the commit metadata resolves to the correct tree, right?

[simonbyrne]

The commit info we can probably get from registrator. But I was thinking we could have a cron job that periodically queries the repos and updates the registry as required.

[ericphanson]

In the case of subpackages, we can have an optional git-tree-path giving the path in the commit/tag to the corresponding tree.

We do have subdir that is similar but that's one subdir per package, rather than per version

[simonbyrne]

We do have subdir that is similar but that's one subdir per package, rather than per version

Perhaps I should rename it for consistency? Should I just call it subdir as well?

[ericphanson]

Yeah, maybe subdir should just move to be per version? Or have both for awhile until all supported Pkg’s know the new location

[simonbyrne]

Yeah, maybe subdir should just move to be per version? Or have both for awhile until all supported Pkg’s know the new location

We probably need to keep a global one for non-released versions (e.g. a specific git commit)

[GunnarFarneback]

I'd like to better understand the use cases for the different pieces of information.

• git-commit-sha1: Ok, I understand this one. This is readily available at the time of registration and not uniquely determined by the tree hash. Moreover it can only change if the repository history is rewritten.
• git-tree-path: Presumably needed to create source code links and taking height for the possibility that packages are moved around in the repository so the global subdir is insufficient?
• git-tag-sha1: Not needed according to add commit hash information to registry JuliaRegistries/RegistryTools.jl#97 (comment).
• git-tag-name: For what purpose is it useful to have this information in the registry?