Skip to content

GitHub actions #2190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 7, 2020
Merged

GitHub actions #2190

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9672db3
CI: Un-travis-ify our scripting for linux/mac
kinnison Jan 6, 2020
5e797b5
CI: Github Actions templates
kinnison Jan 6, 2020
20a24e8
CI: Script to generate github actions workflows
kinnison Jan 6, 2020
5803ad6
CI: Remove vestiges of travis and appveyor
kinnison Jan 7, 2020
b34e892
CI: Generate the workflows for everything
kinnison Jan 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
CI: Github Actions templates 
These templates are for Github Actions Workflows and will be processed
by a script for use.  Do not edit workflows in .github/workflows but
rather edit these templates and then run the script.

Signed-off-by: Daniel Silverstone <[email protected]>
  • Loading branch information
@kinnison
kinnison committed Jan 7, 2020
commit 5e797b55cf6c9f022b2586889ef03f326450cbde
96 changes: 96 additions & 0 deletions ci/actions-templates/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# Rustup Github Actions Workflows

This directory contains all the workflows we use in Rustup for Github Actions.

## Triggers for CI builds

Rustup has four situations in which we perform CI builds:

1. On PR changes
2. On merge to master
3. Time-based rebuilds of master
4. Pushes to the stable branch

The goals for each of those situations are subtly different. For PR changes,
we want to know as quickly as possible if the change is likely to be an issue.

Once a change hits master, we want to know that all our targets build.

Time based rebuilds of master are about determining if updates to the toolchain
have caused us problems, and also to try and hilight if we have flaky tests.

The stable branch is about making releases. Builds from that branch are uploaded
to S3 so that we can then make a release of rustup.

## Targets we need to build

We build for all the Tier one targets and a non-trivial number of the tier two
targets of Rust. We do not even attempt tier three builds.

We don't run the tests on all the targets because many are cross-built. If we
cross-build we don't run the tests. All the builds which aren't mac or windows
are built on an x86_64 system because that's the easiest way to get a performant
system.

| Target | Cross | Tier | On PR? | On master? |
| ----------------------------- | ---------- | ----- | ------ | ---------- |
| x86_64-unknown-linux-gnu | No | One | Yes | Yes |
| armv7-unknown-linux-gnueabihf | Yes | Two | Yes | Yes |
| aarch64-linux-android | Yes | Two | Yes | Yes |
| aarch64-unknown-linux-gnu | Yes | Two | No | Yes |
| powerpc64-unknown-linux-gnu | Yes | Two | No | Yes |
| x86_64-unknown-linux-musl | Yes | Two | No | Yes |
| i686-unknown-linux-gnu | Yes | One | No | No |
| arm-unknown-linux-gnueabi | Yes | Two | No | No |
| arm-unknown-linux-gnueabihf | Yes | Two | No | No |
| x86_64-unknown-freebsd | Yes | Two | No | No |
| x86_64-unknown-netbsd | Yes | Two | No | No |
| powerpc-unknown-linux-gnu | Yes | Two | No | No |
| powerpc64le-unknown-linux-gnu | Yes | Two | No | No |
| mips-unknown-linux-gnu | Yes | Two | No | No |
| mips64-unknown-linux-gnu | Yes | Two | No | No |
| mipsel-unknown-linux-gnu | Yes | Two | No | No |
| mips64el-unknown-linux-gnu | Yes | Two | No | No |
| s390x-unknown-linux-gnu | Yes | Two | No | No |
| arm-linux-androideabi | Yes | Two | No | No |
| armv7-linux-androideabi | Yes | Two | No | No |
| i686-linux-android | Yes | Two | No | No |
| x86_64-linux-android | Yes | Two | No | No |
| ----------------------------- | ---------- | ----- | ------ | ---------- |
| x86_64-apple-darwin | No | One | Yes | Yes |
| ----------------------------- | ---------- | ----- | ------ | ---------- |
| x86_64-pc-windows-msvc | No | One | Yes | Yes |
| x86_64-pc-windows-gnu | No | One | No | Yes |
| i686-pc-windows-msvc | No | One | No | No |
| i686-pc-windows-gnu | No | One | No | No |

We also have a clippy/shellcheck target which runs on x86_64 linux and is
run in all cases. It does a `cargo fmt` check, a `cargo clippy` check on the
beta toolchain, and also runs `rustup-init.sh` through to completion inside
a centos 6 docker to ensure that we continue to work on there OK.

## Useful notes about how we run builds

For the builds which run on x86_64 linux, we deliberately run inside a docker
image which comes from `rust-lang/rust`'s CI so that we know we're linking against
the same libc etc as the release of Rust.

For the builds which run on Windows, we retrieve mingw from Rust's CI as well
so that we're clearly using the right version of that.

In all cases, we attempt to use the `rustup-init.sh` from the branch under test
where at all possible, so that we spot errors in that ASAP.

Given that, we prefer to not use a preinstalled rust/rustup at all if we can,
so we start from as bare a VM as makes sense.

For Windows builds, we use a Visual Studio 2017 image if we can.

## The workflows

Due to limitations in how github workflows work, we have to create our workflows
from template files and then commit them.

The templates are in this directory, and the built workflows end up in the
`.github/workflows` directory. `-all` always runs, `-on-pr` `-on-master` and
`-on-stable` do the obvious.
93 changes: 93 additions & 0 deletions ci/actions-templates/centos-fmt-clippy-template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# This is ci/actions-templates/centos-fmt-clippy.yaml
# Do not edit this file in .github/workflows

name: Centos checks, formatting, clippy, and shellcheck, of Rustup

on:
pull_request:
branches:
- "*"
push:
branches:
- master
- stable
schedule:
- cron: "30 0 * * 1" # Every Monday at half past midnight

jobs:
check:
name: Checks
runs-on: ubuntu-latest
strategy:
fail-fast: false
steps:
- uses: actions/checkout@v2
with:
# v2 defaults to a shallow checkout, but we need at least to the previous tag
fetch-depth: 0
- name: Acquire tags for the repo
run: |
git fetch --no-tags --prune --depth=1 origin +refs/tags/*:refs/tags/*
- name: Display the current git status
run: |
git status
git describe
- name: Prep cargo dirs
run: |
mkdir -p ~/.cargo/{registry,git}
- name: Set environment variables appropriately for the build
run: |
echo "::add-path::$HOME/.cargo/bin"
- name: Cache cargo registry
uses: actions/cache@v1
with:
path: ~/.cargo/registry
key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }}
- name: Cache cargo index
uses: actions/cache@v1
with:
path: ~/.cargo/git
key: ${{ runner.os }}-cargo-index-${{ hashFiles('**/Cargo.lock') }}
- name: Cache cargo build
uses: actions/cache@v1
with:
path: target
key: ${{ runner.os }}-cargo-build-clippy-beta-${{ hashFiles('**/Cargo.lock') }}
- name: Install Rustup using ./rustup-init.sh
run: |
sh ./rustup-init.sh --default-toolchain=none --profile=minimal -y
- name: Ensure Beta is up to date
run: |
if rustc +beta -vV >/dev/null 2>/dev/null; then
rustup toolchain uninstall beta
fi
rustup toolchain install --profile=minimal beta
rustup default beta
- name: Ensure we have the components we need
run: |
rustup component add rustfmt
rustup component add clippy
- name: Run the centos check within the docker image
run: |
HERE=$(pwd)
docker run \
--volume "$HERE":/checkout:ro \
--workdir /checkout \
--tty \
--init \
--rm \
centos:6 \
sh ./ci/raw_init.sh
- name: Run shell checks
run: |
shellcheck -s dash -- rustup-init.sh
git ls-files -- '*.sh' | xargs shellcheck -s dash -e SC1090
git ls-files -- '*.bash' | xargs shellcheck -s bash -e SC1090
- name: Run formatting checks
run: |
cargo fmt --all -- --check
- name: Run cargo check and clippy
run: |
cargo check --all --all-targets
git ls-files -- '*.rs' | xargs touch
cargo clippy --all --all-targets
154 changes: 154 additions & 0 deletions ci/actions-templates/linux-builds-template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
# This is ci/actions-templates/linux-builds-template.yaml
# Do not edit this file in .github/workflows

name: Linux-hosted builds of Rustup (PR) # skip-master skip-stable
name: Linux-hosted builds of Rustup (master) # skip-pr skip-stable
name: Linux-hosted builds of Rustup (stable) # skip-master skip-pr

on:
pull_request: # skip-master skip-stable
branches: # skip-master skip-stable
- "*" # skip-master skip-stable
push: # skip-pr
branches: # skip-pr
- master # skip-pr skip-stable
- stable # skip-pr skip-master
schedule: # skip-pr skip-stable
- cron: "30 0 * * 1" # Every Monday at half past midnight UTC skip-pr skip-stable

jobs:
build:
name: Build
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
target:
- x86_64-unknown-linux-gnu
- armv7-unknown-linux-gnueabihf
- aarch64-linux-android
- aarch64-unknown-linux-gnu # skip-pr
- powerpc64-unknown-linux-gnu # skip-pr
- x86_64-unknown-linux-musl # skip-pr
- i686-unknown-linux-gnu # skip-pr skip-master
- arm-unknown-linux-gnueabi # skip-pr skip-master
- arm-unknown-linux-gnueabihf # skip-pr skip-master
- x86_64-unknown-freebsd # skip-pr skip-master
- x86_64-unknown-netbsd # skip-pr skip-master
- powerpc-unknown-linux-gnu # skip-pr skip-master
- powerpc64le-unknown-linux-gnu # skip-pr skip-master
- mips-unknown-linux-gnu # skip-pr skip-master
- mips64-unknown-linux-gnuabi64 # skip-pr skip-master
- mipsel-unknown-linux-gnu # skip-pr skip-master
- mips64el-unknown-linux-gnuabi64 # skip-pr skip-master
- s390x-unknown-linux-gnu # skip-pr skip-master
- arm-linux-androideabi # skip-pr skip-master
- armv7-linux-androideabi # skip-pr skip-master
- i686-linux-android # skip-pr skip-master
- x86_64-linux-android # skip-pr skip-master
include:
- target: x86_64-unknown-linux-gnu
run_tests: YES
steps:
- uses: actions/checkout@v2
with:
# v2 defaults to a shallow checkout, but we need at least to the previous tag
fetch-depth: 0
- name: Acquire tags for the repo
run: |
git fetch --no-tags --prune --depth=1 origin +refs/tags/*:refs/tags/*
- name: Display the current git status
run: |
git status
git describe
- name: Prep cargo dirs
run: |
mkdir -p ~/.cargo/{registry,git}
- name: Set environment variables appropriately for the build
run: |
echo "::add-path::$HOME/.cargo/bin"
echo "::set-env name=TARGET::${{matrix.target}}"
- name: Skip tests
run: |
echo "::set-env name=SKIP_TESTS::yes"
if: matrix.run_tests == ''
- name: Cache cargo registry
uses: actions/cache@v1
with:
path: ~/.cargo/registry
key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }}
- name: Cache cargo index
uses: actions/cache@v1
with:
path: ~/.cargo/git
key: ${{ runner.os }}-cargo-index-${{ hashFiles('**/Cargo.lock') }}
- name: Cache cargo build
uses: actions/cache@v1
with:
path: target
key: ${{ runner.os }}-cargo-build-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }}
- name: Install Rustup using ./rustup-init.sh
run: |
sh ./rustup-init.sh --default-toolchain=none --profile=minimal -y
- name: Ensure Stable is up to date
run: |
if rustc +stable -vV >/dev/null 2>/dev/null; then
rustup toolchain uninstall stable
fi
rustup toolchain install --profile=minimal stable
- name: Ensure we have our goal target installed
run: |
rustup target install "$TARGET"
- name: Determine which docker we need to run in
run: |
case "$TARGET" in
*-linux-android*) DOCKER=android ;; # Android uses a local docker image
*) DOCKER="$TARGET" ;;
esac
echo "::set-env name=DOCKER::$DOCKER"
- name: Fetch the docker
run: bash ci/fetch-rust-docker.bash "${TARGET}"
- name: Maybe build a docker from there
run: |
if [ -f "ci/docker/$DOCKER/Dockerfile" ]; then
docker build -t "$DOCKER" -f "ci/docker/${DOCKER}/Dockerfile" .
fi
- name: Run the build within the docker image
run: |
HERE=$(pwd)
mkdir -p "${HERE}/target"
chown -R "$(id -u)":"$(id -g)" "${HERE}/target"
docker run \
--entrypoint sh \
--user "$(id -u)":"$(id -g)" \
--volume "$(rustc --print sysroot)":/rustc-sysroot:ro \
--volume "${HERE}":/checkout:ro \
--volume "${HERE}"/target:/checkout/target \
--workdir /checkout \
--env TARGET="${TARGET}" \
--env SKIP_TESTS="${SKIP_TESTS}" \
--volume "${HOME}/.cargo:/cargo" \
--env CARGO_HOME=/cargo \
--env CARGO_TARGET_DIR=/checkout/target \
--env LIBZ_SYS_STATIC=1 \
--tty \
--init \
--rm \
"${DOCKER}" \
-c 'PATH="${PATH}":/rustc-sysroot/bin bash ci/run.bash'
- name: Acquire the AWS tooling
run: |
pip3 install -U setuptools
pip3 install awscli
if: github.event_name == 'push' && github.ref == 'refs/heads/stable'
- name: Prepare the dist
run: |
bash ci/prepare-deploy.bash
if: github.event_name == 'push' && github.ref == 'refs/heads/stable'
- name: Deploy build to dev-static dist tree for release team
run: |
aws s3 cp --recursive deploy/ s3://dev-static-rust-lang-org/rustup/
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
if: github.event_name == 'push' && github.ref == 'refs/heads/stable'
Loading