A Bazel extension to download archives, files and packages for use within Bazel targets
Add the following to MODULE.bazel:
bazel_dep(name="download_utils", version="0.0.0")Use the repository rules in MODULE.bazel to download artifacts:
Download an archive and unpack it:
download_archive = use_repo_rule("@download_utils//download/archive:defs.bzl", "download_archive")
download_archive(
name = "archive",
urls = ["https://some.thing/archive.tar"],
)Download a single file and optionally make it executable:
download_file = use_repo_rule("@download_utils//download/file:defs.bzl", "download_file")
download_file(
name = "file",
output = "executable",
executable = True,
urls = ["https://some.thing/executable-amd64-linux"],
)Download a Debian package, unpack it then unpack the data.tar.{xz,zst}:
download_deb = use_repo_rule("@download_utils//download/deb:defs.bzl", "download_deb")
download_deb(
name = "deb",
integrity = "sha256-vMiq8kFBwoSrVEE+Tcs08RvaiNp6MsboWlXS7p1clO0=",
urls = ["https://some.thing/test_1.0-1_all.deb"],
commands = {
"chmod": [
"$(location @coreutils)",
"chmod",
"u+x",
"some-script.sh",
],
},
links = {
"etc/test/fixture.txt": "fixture.txt",
},
tools = [
"@coreutils",
],
)Use an extension to template the attributes based on provided substitutions:
download = use_extension("@download_utils//download/template:defs.bzl", "download_template")
download.archive(
name = "coreutils-{version}-{triplet}",
srcs = ["entrypoint"],
links = {
"coreutils{executable.extension}": "entrypoint",
},
# Run `bazel run @download_utils//download/template/lock coreutils/lock.json` to lock integrities
lock = "//coreutils:lock.json",
strip_prefix = "coreutils-{version}-{rust.triplet}",
substitutions = {
"version": [
"0.0.28",
"0.1.0",
],
"triplets": [
"arm64-linux-gnu",
"amd64-linux-gnu",
"arm64-linux-musl",
"amd64-linux-musl",
"amd64-windows-msvc",
"arm64-macos-darwin",
"amd64-macos-darwin",
],
},
# Run `bazel run @download_utils//download/template/lock:upload coreutils/lock.json` to mirror binaries and lock integrities
uploads = [
"https://gitlab.arm.com/api/v4/projects/bazel%2Fdownload_utils/packages/generic/coreutils/{version}/{rust.archive.basename}",
],
urls = [
"https://gitlab.arm.com/api/v4/projects/bazel%2Fdownload_utils/packages/generic/coreutils/{version}/{rust.archive.basename}",
"https://github.com/uutils/coreutils/releases/download/{version}/coreutils-{version}-{rust.archive.basename}",
],
)
# Run `bazel mod tidy` to update these
use_repo(
download,
"coreutils-0.0.28-amd64-linux-gnu",
"coreutils-0.0.28-amd64-linux-musl",
"coreutils-0.0.28-amd64-macos-darwin",
"coreutils-0.0.28-amd64-windows-msvc",
"coreutils-0.0.28-arm64-linux-gnu",
"coreutils-0.0.28-arm64-linux-musl",
"coreutils-0.0.28-arm64-macos-darwin",
"coreutils-0.1.0-amd64-linux-gnu",
"coreutils-0.1.0-amd64-linux-musl",
"coreutils-0.1.0-amd64-macos-darwin",
"coreutils-0.1.0-amd64-windows-msvc",
"coreutils-0.1.0-arm64-linux-gnu",
"coreutils-0.1.0-arm64-linux-musl",
"coreutils-0.1.0-arm64-macos-darwin",
)Extra substitutions, such as {rust.archive.basename} are created using the download_template.substitution and download_template.substitutions APIs:
download = use_extension("@download_utils//download/template:defs.bzl", "download_template")
# Create derived `{cpu}` substitution based on an incomming `{triplet}` substitution.
download.substitution(
name = "cpu",
match = "{triplet}",
select = {
"//conditions:default": "{triplet.split('-')[0]}",
},
)
# Read substitutions from a JSON file
download.substitutions(
srcs = [
"substitutions.json",
],
)The layout of the JSON is as so:
{
"<name>": "<value>",
"<name>": {
"<match>": {
"<key>": "<value>",
},
},
}Concretly, as an example, two substitutions:
{
"os": "{triplet.split('-')[1]}",
"rust.vendor": {
"{os}": {
"macos": "apple",
"windows": "pc",
"//conditions:default": "unknown"
}
}
}@download_utils has a set of subtitutions already registered. See download/template/*.json
The sub-resource integrity (SRI) is not required for secure URLs. For non-secure (http, ftp) it is. It is
recommended to always add the integrity to allow reproducible builds and sharing of downloads. When the integrity is
omitted and the rule is resolved, the correct SRI is output to the terminal. The easiest way to download the artifact is
to query the targets within the repository: bazelisk query @archive//....
The rules accept patches to modify the content after download. Examples are provided in the end to end tests.
Patches are applied after the BUILD.bazel file is written so can be used to customise the targets exposed to the
Bazel build. Usually, however, a custom BUILD.bazel file is provided to the build argument.
Hermetic commands can be ran against the unpacked repository. Hermetic binaries can be provided via the tools argument and be used with $(location <label>) in the
commands. The end to end tests provided examples of this.
By default, the BUILD.bazel file exports all files downloaded. This can be controlled with the srcs attribute to
customise the files that are exposed to the Bazel build.
Symlinks/hardlinks can be created in the download repositories by providing the links map of target files to link names.
This ruleset is entirely hermetic and does not require anything from the system.
The commands supports $(location <label>) for running hermetic commands on the downloaded data.
The project publishes the relevant files to GitLab releases for use when a version has not been added to the upstream BCR.
This is often the case for pre-release versions.
Add the following to .bazelrc:
# `bzlmod` pre-release registries
common --registry https://bcr.bazel.build
common --registry=https://gitlab.arm.com/bazel/download_utils/-/releases/v1.0.0-alpha.1/downloads
Then a GitLab release version can be used in bazel_dep.