From 539121832774020f612adee3371573050131ab02 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Sun, 6 Mar 2022 21:52:41 -0300 Subject: [PATCH 01/22] fix editorconfig --- .editorconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.editorconfig b/.editorconfig index 6ef8cb3..09a667b 100644 --- a/.editorconfig +++ b/.editorconfig @@ -6,6 +6,6 @@ insert_final_newline = true indent_style = tab trim_trailing_whitespace = true -[*.{yml,yaml}] +[*.{yml,yaml,md}] indent_style = space indent_size = 2 From f01e316f112d5eb26f722ef504047614dd5cd7ce Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Sun, 6 Mar 2022 21:55:20 -0300 Subject: [PATCH 02/22] adjust .env.example --- .env.example | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/.env.example b/.env.example index 435c17f..5fb9ac6 100644 --- a/.env.example +++ b/.env.example @@ -1,14 +1,12 @@ -# Move this file to .env in order to have it take effect. -# All values are only examples. You should figure out if they are proper by the -# application's code. +# Set the appropriate values and copy this file to .env # --- REQUIRED VARIABLES --- # The port where the server will be started at WEBHOOK_PORT=8080 -# The organization name or repository owner username of where the Github App is -# installed -INSTALLATION_LOGIN=org +# The organization name or the repository owner's username of where the Github +# App is installed +INSTALLATION_LOGIN=placeholder # The path of the database directory. If it's not an absolute path, it will be # relative to this repository's root. @@ -23,18 +21,18 @@ REPOSITORIES_PATH=repositories PRIVATE_KEY_PATH=githubPrivateKey.pem # The webhook secret according to the Github App's settings. -WEBHOOK_SECRET=hunter2 +WEBHOOK_SECRET=placeholder # The Github App ID according to the Github App's settings. GITHUB_APP_ID=123 # The GitHub team which whose members will be used as team leads -# Must be in the same organization where this bot is installed -TEAM_LEADS_TEAM=team +# Must be from the organization where the GitHub App is installed +TEAM_LEADS_TEAM=placeholder # The GitHub team which whose members will be used as core developers -# Must be in the same organization where this bot is installed -CORE_DEVS_TEAM=team +# Must be from the organization where the GitHub App is installed +CORE_DEVS_TEAM=placeholder # --- OPTIONAL VARIABLES --- From f99a4b624e2f839cd674641b76b80570cafa5312 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Sun, 6 Mar 2022 21:55:41 -0300 Subject: [PATCH 03/22] update docs --- README.md | 302 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 223 insertions(+), 79 deletions(-) diff --git a/README.md b/README.md index 19c62b9..345c30a 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,69 @@ -# 👾 processbot - -# Commands - -To be posted as a comment in the Pull Request. **Your whole comment should only -have the command**. +# Introduction + +parity-processbot is a +[GitHub App](https://docs.github.com/en/developers/apps/getting-started-with-apps/about-apps) +which drives the +[Companion Build System](https://github.com/paritytech/parity-processbot/issues/327)'s +merge process. It receives [commands](#commands) and from subsequent GitHub +events it will resume processing all pending pull requests until they can be +merged; a more detailed breakdown is available in the +[Implementation section](#implementation). + +Note that parity-processbot does not implement the Companion Build System's +cross-repository integration check, which is done on CI (see +[check_dependent_project](https://github.com/paritytech/pipeline-scripts#check_dependent_project) +for that). + +Before starting to work on this project we recommend reading the +[Implementation section](#implementation). + +# TOC + +- [Commands](#commands) + - [Relation to CI](#commands-relation-to-ci) +- [Criteria for merge](#criteria-for-merge) + - [Approvals](#criteria-for-merge-approvals) + - [Checks and statuses](#criteria-for-merge-checks-and-statuses) +- [GitHub App](#github-app) + - [Configuration](#github-app-configuration) + - [Installation](#github-app-installation) +- [Setup](#setup) + - [Requirements](#setup-requirements) + - [Environment variables](#setup-environment-variables) +- [Development](#development) + - [Run the application](#development-run) + - [Integration tests](#development-integration-tests) + - [Example workflows](#development-example-workflows) + - [Test repositories](#development-test-repositories) +- [Deployment](#deployment) + - [Logs](#deployment-logs) + +# Commands + +The following commands should be posted as pull request comments. **Your whole +comment should only have the command**. - `bot merge`: [if approved](#criteria-for-merge), merge once checks pass. - `bot merge force`: [if approved](#criteria-for-merge), merge immediately - while disregarding checks. + while disregarding checks + ([not all of them can be disregarded](#criteria-for-merge-checks-and-statuses)). - `bot merge cancel`: cancel a pending `bot merge`; does not affect anything outside of processbot, only stops the bot from following through with the merge. - `bot rebase`: create a merge commit from origin/master into the PR. Note: The commands will only work if you are a member of the organization where -this bot is installed. Organization membership is always gotten fresh from the -Github API at the time a comment arrives. +the GitHub App is installed. Organization membership is fetched from the GitHub +API at the time a comment arrives. -## Relation to CI +## Relation to CI processbot categorizes CI statuses as following, ranked in descending order of importance: ### 1. Required -Required through Github branch protection rules +Required through GitHub branch protection rules They are meant to be blockers so can't be skipped anyhow. @@ -33,7 +72,7 @@ They are meant to be blockers so can't be skipped anyhow. Derived from Gitlab Jobs which **do not** have `allow_failure: true` They are relevant but not blockers, thus can be skipped with `bot merge force` -but will not pass `bot merge`. Note that the companion build system follows the +but will not pass `bot merge`. Note that the merge of companions follows the logic of `bot merge`, thus a brittle job in this category might get in the way of a companion merge. @@ -45,9 +84,9 @@ Unstable statuses will have `allow_failure: true` encoded in their descriptions ([delivered from vanity-service](https://gitlab.parity.io/parity/websites/vanity-service/-/blob/ddc0af0ec8520a99a35b9e33de57d28d37678686/service.js#L77)) which will allow processbot to detect and disregard them. -# Criteria for merge +# Criteria for merge -## Approvals +## Approvals A Pull Request needs either (meaning, only **one of** the following requirements needs to be fulfilled) @@ -56,12 +95,12 @@ requirements needs to be fulfilled) - One [substrateteamleads](https://github.com/orgs/paritytech/teams/substrateteamleads) member approval This criterion strictly matters only for the bot's internal logic irrespective -of Github Repository Settings and will not trump the latter in any case. For +of GitHub Repository Settings and will not trump the latter in any case. For instance, the rule: > One substrateteamleads member approval -does not imply that the pull request will be mergeable if the Github Settings +does not imply that the pull request will be mergeable if the GitHub Settings require more approvals than that. The bot's rules work *in addition* to the repository's settings while still respecting them. Specifically when it comes to the approvals' count, however, the bot might able to help if a @@ -75,96 +114,150 @@ a [team lead](https://github.com/orgs/paritytech/teams/substrateteamleads). The reasoning for this feature is as follows: 1. PR authors cannot approve their own merge requests, although - [team leads](https://github.com/orgs/paritytech/teams/substrateteamleads) - should have the means to bypass that requirement e.g. for trivial or urgent - changes. + [team leads](https://github.com/orgs/paritytech/teams/substrateteamleads) + should have the means to bypass that requirement e.g. for trivial or urgent + changes. 2. If the - [team lead](https://github.com/orgs/paritytech/teams/substrateteamleads) - has already approved and it's still - short of one, they cannot "approve twice" in order to meet the quota. In that - case, the bot should contribute one approval in order to help them meet that - requirement. + [team lead](https://github.com/orgs/paritytech/teams/substrateteamleads) + has already approved and it's still + short of one, they cannot "approve twice" in order to meet the quota. In that + case the bot should contribute one approval in order to help them meet the + approval count. + +## Checks and statuses + +All [Important and above](#commands-relation-to-ci) checks should be green when +using `bot merge`. + +Non-Required statuses can bypassed by using `bot merge force`. + +# GitHub App -## Checks and statuses +You can create a new app at . More context: +- +- -All [Important and above](#relation-to-ci) checks should be green when using -`bot merge` (can be bypassed by using `bot merge force`). +After creating the app, you should [configure](#github-app-configuration) and +[install it](#github-app-installation) (make sure the +[environment](#setup-environment-variables) is properly set up before using it). -# Github App Configuration +## Configuration -Repository permissions +### Repository permissions - Contents: Read & write + - Enables pushing commits for updating companions after their dependencies + have been merged - Issues: Read & write -- Metadata: Read-only + - Enables comment on pull requests - Pull requests: Read & write -- Projects: Read-only + - Enables merging pull requests - Commit statuses: Read-only + - Enables fetching the CI statuses before merge - Checks: Read-only + - Enables fetching the checks's statuses before merge -Organization permissions +### Organization permissions - Members: Read-only + - Enables fetching the command's requester organization membership even if + their membership is private -Events: +### Events -- Check run - Issue comment -- Status -- Workflow job + - Enables reacting to [commands](#commands) from GitHub comments +- Check run, Status, Workflow job + - Used to trigger the processing of pending pull requests ---- +## Installation + +Having [created](#github-app) and [configured](#github-app-configuration) the +GitHub App, install it in a repository through +`https://github.com/settings/apps/${APP}/installations`. If processbot has to merge PRs into protected branches which have the "Restrict who can push to matching branches" rule enabled, it should be added to the allowlist for that rule, otherwise merging will not work -(example: https://github.com/paritytech/polkadot/pull/4122#issuecomment-948680155). - +([example](https://github.com/paritytech/polkadot/pull/4122#issuecomment-948680155)). In such cases it's necessary to add the app to the allowlist, as demonstrated below: ![image](https://user-images.githubusercontent.com/77391175/138313741-b33b86a5-ee58-4031-a7da-12703ea9958e.png) -# Local development +# Setup -This project is a standard Rust project with some notable requirements. +## Requirements -## Requirements +- Rust + - [rustup](https://rustup.rs/) is the recommended way of setting up a Rust + toolchain +- libssl +- libclang +- git -Before you can generate a debug or release binary, we have some library -requirements to install. +## Environment variables -```sh -$ sudo apt install \ - libssl-dev \ - libclang-dev -``` +All relevant environment variables are documented in the +[.env.example](./.env.example) file. For development you're welcome to copy that +file to `.env` so that all values will be loaded automatically once the +application starts. -## Environment variables +# Development -The bot requires some environment variables listed in -[config.rs](./src/config.rs). +## Run the application -They can, optionally, be set through an `.env` -file which should be placed at the root of this repository. We provide an -[example .env file](./.env.example) which can be used as a starting point. +1. [Set up the GitHub App](#github-app) +2. [Set up the application](#setup) + - During development it's handy to use a [smee.io](https://smee.io/) proxy, + through the `WEBHOOK_PROXY_URL` environment variable, for receiving GitHub + Webhook Events in your local server instance. +3. Run the project with `cargo run` +4. Optionally [try out the example workflows](#development-example-workflows) in + the repositories where you have installed the app or the + [test repositories](#development-test-repositories) after a deployment -``` -$ cp .env.example .env -``` +## Test repositories -During **development**, it's handy to use a [smee.io](https://smee.io/) proxy, -through the `WEBHOOK_PROXY_URL` environment variable, for receiving Github -Webhook Events in your local instance of processbot. +The staging instance is installed in the following repositories: -## Running the bot +- https://github.com/paritytech/main-for-processbot-staging +- https://github.com/paritytech/companion-for-processbot-staging -```sh -$ cargo run -``` +The GitHub App for staging is managed by +[paritytech](http://github.com/paritytech)'s Organizational GitHub Admins. + +## Example workflows + +### Single merge use-case + +Example: https://github.com/paritytech/main-for-processbot-staging/pull/55 + +Steps: + +1. Create a pull request in the repositories where the app is installed +2. Comment `bot merge` + +### Companion use-case + +Example: + - Repository A: https://github.com/paritytech/main-for-processbot-staging/pull/53 + - Repository B: https://github.com/paritytech/companion-for-processbot-staging/pull/31 + +Steps: + +1. Install the app in Repository A +2. Install the app in Repository B + - Repository B needs to be a dependency of Repository A + ([example](https://github.com/paritytech/companion-for-processbot-staging/blob/8ff68ae8287342f2a4581b1950913b4e9e88a0e0/Cargo.toml#L8)) +3. Create a pull request on Repository B and copy its link +4. Create a pull request on Repository A and say `companion: [link from step 3]` +5. Comment `bot merge` on the pull request in Repository A +6. Observe that the the pull request in Repository A will be merged first and + the pull request on Repository B will be merged after -## Integration tests +## Integration tests The integration tests are executed as follows: @@ -176,25 +269,76 @@ We use [insta](https://github.com/mitsuhiko/insta#introduction) for integration tests' snapshots. After creating or modifying a snapshot, use `cargo insta review` to manage the results. -# Deployment +# Deployment -The bot is automatically deployed by pushing a tag with one of the following formats +When you push a deployment tag to GitHub, it will be mirrored to GitLab and then +the [CI pipeline](./.gitlab-ci.yml) will be ran on the GitLab Runners for +deploying the app. Deployment tags should conform to one of the following +patterns: -- `/^v[0-9]+\.[0-9]+.*$/`, e.g. `v1.1`, will deploy it to production (cluster - `parity-prod`). +- `/^v[0-9]+\.[0-9]+.*$/`, e.g. `v1.1`, will be deployed to production + - The production instance is installed in + [Substrate](https://github.com/paritytech/substrate), + [Polkadot](https://github.com/paritytech/polkadot) and + [Cumulus](https://github.com/paritytech/cumulus) -- `/^pre-v[0-9]+\.[0-9]+.*$/`, e.g. `pre-v0.6` will deploy to staging (cluster - `parity-stg`). - - The staging package is deployed to a dedicated separate Github App. There's - [a (private) repository for staging](https://github.com/paritytech/polkadot-for-processbot-staging) - which has it installed, already set up with a mirrored - [a (private) project on Gitlab](https://gitlab.parity.io/parity/polkadot-for-processbot-staging) - for CI. +- `/^pre-v[0-9]+\.[0-9]+.*$/`, e.g. `pre-v0.6` will be deployed to staging + - The staging instance is installed in the [test repositories](#development-test-repositories) The deployment's status can be followed through [Gitlab Pipelines on the parity-processbot mirror](https://gitlab.parity.io/parity/parity-processbot/-/pipelines) ([example](https://gitlab.parity.io/parity/parity-processbot/-/jobs/867102)). All of the relevant configuration for deployment lives in the [./helm](./helm) -folder. The values for each specific environment are in `values-*.yml`. If you -add a value, it needs to be used in `templates/processbot.yaml`. +folder. The values for each specific environment are in +`helm/values-${ENVIRONMENT}.yml`. If you add a value, it needs to be used in +[helm/templates/processbot.yaml](helm/templates/processbot.yaml). + +## Logs + +See https://gitlab.parity.io/groups/parity/opstooling/-/wikis/Home + +# Implementation + +To have a better understanding on the Companion Build System we strongly +recommend to +[read its explanation](https://github.com/paritytech/parity-processbot/issues/327) +and +[watch the presentation](https://drive.google.com/file/d/1E4Fd3aO2QRJuoUBI4j0Zp4027yGeHeer/view). + +A +[web server](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/server.rs#L88) +(set up from +[main](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/main.rs#L107)) +receives +[GitHub Webhook events](https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads) +as HTTP `POST` requests. + +When someone comments in a pull request, the +[issue comment event is parsed](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L220) +and from it a +[command is extracted](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L906) +and +[handled](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L752). + +The merge chain is +[started](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L761) +from a +[merge command](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L785). If the pull request at the root of the chain is +[ready to be merged, it will be merged immediately](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L787), +otherwise it will +[be saved to the database](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L813) +and +[merged later once its requirements are ready](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L580); +by "requirements" we mean its statuses, checks and dependencies +([the root of the chain is started without dependencies](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L777-L778), +hence why it can be merged first). + +After a pull request is merged, +[its dependents are checked](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L831) +and possibly merged if all of their requirements are ready (note that a pull +request my might depend on more than one pull request, as +[explained in the presentation at 4:48](https://drive.google.com/file/d/1E4Fd3aO2QRJuoUBI4j0Zp4027yGeHeer/view?t=4m48s)). +This process is repeated for each item that is merged throughout the merge +chain (referred as "Phase 1 and Phase 2" +[in the presentation at 25:48](https://drive.google.com/file/d/1E4Fd3aO2QRJuoUBI4j0Zp4027yGeHeer/view?t=25m48s)). From d32ffd88621e861f9d9f44cf4d7c974912d12ac1 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Sun, 6 Mar 2022 23:43:42 -0300 Subject: [PATCH 04/22] update links --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 345c30a..e15c41d 100644 --- a/README.md +++ b/README.md @@ -296,7 +296,7 @@ folder. The values for each specific environment are in ## Logs -See https://gitlab.parity.io/groups/parity/opstooling/-/wikis/Home +See https://gitlab.parity.io/groups/parity/opstooling/-/wikis/.Home # Implementation From c8b22d1aaa89e92d275245abb5f6ac5c12c31336 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Mon, 7 Mar 2022 05:49:38 -0300 Subject: [PATCH 05/22] update link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e15c41d..cdfffcc 100644 --- a/README.md +++ b/README.md @@ -296,7 +296,7 @@ folder. The values for each specific environment are in ## Logs -See https://gitlab.parity.io/groups/parity/opstooling/-/wikis/.Home +See https://gitlab.parity.io/groups/parity/opstooling/-/wikis # Implementation From 6db59c26f6447ffecdd67c1d9b86af0bdda5631b Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Mon, 7 Mar 2022 08:46:30 -0300 Subject: [PATCH 06/22] fix nested blocks --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index cdfffcc..9326bc3 100644 --- a/README.md +++ b/README.md @@ -210,9 +210,11 @@ application starts. 1. [Set up the GitHub App](#github-app) 2. [Set up the application](#setup) - - During development it's handy to use a [smee.io](https://smee.io/) proxy, + + During development it's handy to use a [smee.io](https://smee.io/) proxy, through the `WEBHOOK_PROXY_URL` environment variable, for receiving GitHub Webhook Events in your local server instance. + 3. Run the project with `cargo run` 4. Optionally [try out the example workflows](#development-example-workflows) in the repositories where you have installed the app or the From c3364c1a13341ad4b58ed77a1a0ac65f1a338d75 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Tue, 8 Mar 2022 06:48:43 -0300 Subject: [PATCH 07/22] add links to the slides --- README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 9326bc3..d197a74 100644 --- a/README.md +++ b/README.md @@ -304,9 +304,7 @@ See https://gitlab.parity.io/groups/parity/opstooling/-/wikis To have a better understanding on the Companion Build System we strongly recommend to -[read its explanation](https://github.com/paritytech/parity-processbot/issues/327) -and -[watch the presentation](https://drive.google.com/file/d/1E4Fd3aO2QRJuoUBI4j0Zp4027yGeHeer/view). +[consult its explanation](https://github.com/paritytech/parity-processbot/issues/327). A [web server](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/server.rs#L88) @@ -340,7 +338,11 @@ After a pull request is merged, [its dependents are checked](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/webhook.rs#L831) and possibly merged if all of their requirements are ready (note that a pull request my might depend on more than one pull request, as -[explained in the presentation at 4:48](https://drive.google.com/file/d/1E4Fd3aO2QRJuoUBI4j0Zp4027yGeHeer/view?t=4m48s)). +[explained in the presentation at 4:48](https://drive.google.com/file/d/1E4Fd3aO2QRJuoUBI4j0Zp4027yGeHeer/view?t=4m48s) +or +[slide number 6](https://docs.google.com/presentation/d/12ksmejR_UXC1tIHD2f4pQQZ1uw5NK3n8enmwkTCPOpw/edit?usp=sharing)). This process is repeated for each item that is merged throughout the merge chain (referred as "Phase 1 and Phase 2" -[in the presentation at 25:48](https://drive.google.com/file/d/1E4Fd3aO2QRJuoUBI4j0Zp4027yGeHeer/view?t=25m48s)). +[in the presentation at 25:48](https://drive.google.com/file/d/1E4Fd3aO2QRJuoUBI4j0Zp4027yGeHeer/view?t=25m48s) +or +[slide number 21](https://docs.google.com/presentation/d/12ksmejR_UXC1tIHD2f4pQQZ1uw5NK3n8enmwkTCPOpw/edit?usp=sharing)). From cc75c6e8c56f264589157216905a31e7c392c9d4 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Wed, 9 Mar 2022 05:23:57 -0300 Subject: [PATCH 08/22] clarify companion description procedure --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index d197a74..db18b19 100644 --- a/README.md +++ b/README.md @@ -254,7 +254,8 @@ Steps: - Repository B needs to be a dependency of Repository A ([example](https://github.com/paritytech/companion-for-processbot-staging/blob/8ff68ae8287342f2a4581b1950913b4e9e88a0e0/Cargo.toml#L8)) 3. Create a pull request on Repository B and copy its link -4. Create a pull request on Repository A and say `companion: [link from step 3]` +4. Create a pull request on Repository A and put `companion: [link from step 3]` + in its description 5. Comment `bot merge` on the pull request in Repository A 6. Observe that the the pull request in Repository A will be merged first and the pull request on Repository B will be merged after From 4a661becce3b1bcde76ac96032949a966b515644 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Wed, 9 Mar 2022 05:24:03 -0300 Subject: [PATCH 09/22] remove link to stale job --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index db18b19..92ad79c 100644 --- a/README.md +++ b/README.md @@ -289,8 +289,7 @@ patterns: - The staging instance is installed in the [test repositories](#development-test-repositories) The deployment's status can be followed through -[Gitlab Pipelines on the parity-processbot mirror](https://gitlab.parity.io/parity/parity-processbot/-/pipelines) -([example](https://gitlab.parity.io/parity/parity-processbot/-/jobs/867102)). +[Gitlab Pipelines on the parity-processbot mirror](https://gitlab.parity.io/parity/parity-processbot/-/pipelines). All of the relevant configuration for deployment lives in the [./helm](./helm) folder. The values for each specific environment are in From 13627fb2db2a5e81fb5f660845033d6b372c1cf0 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Wed, 9 Mar 2022 05:35:55 -0300 Subject: [PATCH 10/22] clarify comment --- src/webhook.rs | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/src/webhook.rs b/src/webhook.rs index f92451d..539c553 100644 --- a/src/webhook.rs +++ b/src/webhook.rs @@ -635,10 +635,9 @@ pub async fn handle_dependents_after_merge( state, &dependent, &WaitToMergeMessage::Default, - // This is the start of the dependency chain, therefore the dependents - // should always be registered to the database - // TODO: perhaps merge the dependencies if this dependent already exists in - // the database + // The dependent should always be registered to the database as a pending + // item since its dependency just got merged, therefore it becomes eligible + // for merge in the future true, !depends_on_another_pr, ) From 11edcc8868a53b306230a8bfa33c24c1b8685c4c Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Thu, 10 Mar 2022 13:26:15 -0300 Subject: [PATCH 11/22] typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 92ad79c..aa00477 100644 --- a/README.md +++ b/README.md @@ -156,7 +156,7 @@ After creating the app, you should [configure](#github-app-configuration) and - Commit statuses: Read-only - Enables fetching the CI statuses before merge - Checks: Read-only - - Enables fetching the checks's statuses before merge + - Enables fetching the checks' statuses before merge ### Organization permissions From 1ce0f08864af92f9800a8498d0eb5655358ad6a4 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Thu, 10 Mar 2022 13:27:09 -0300 Subject: [PATCH 12/22] typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index aa00477..6e9fcc4 100644 --- a/README.md +++ b/README.md @@ -161,7 +161,7 @@ After creating the app, you should [configure](#github-app-configuration) and ### Organization permissions - Members: Read-only - - Enables fetching the command's requester organization membership even if + - Enables fetching the command requester's organization membership even if their membership is private ### Events From ce7632e873ca350c64adba98f76c5de2e0cbf7f3 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Thu, 10 Mar 2022 13:28:28 -0300 Subject: [PATCH 13/22] update github app section --- README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 6e9fcc4..12b216c 100644 --- a/README.md +++ b/README.md @@ -134,9 +134,13 @@ Non-Required statuses can bypassed by using `bot merge force`. # GitHub App -You can create a new app at . More context: -- -- +The GitHub App is necessary for the application to receive +[webhook events](https://docs.github.com/en/developers/webhooks-and-events/webhooks/about-webhooks) +and access the GitHub API properly. + +Follow the instructions of + +for creating a new GitHub App. After creating the app, you should [configure](#github-app-configuration) and [install it](#github-app-installation) (make sure the From e08b5e51fba02c2fb22b1a264a86193fd70b6a56 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Thu, 10 Mar 2022 13:31:54 -0300 Subject: [PATCH 14/22] explain the requirements --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 12b216c..4e39412 100644 --- a/README.md +++ b/README.md @@ -194,12 +194,12 @@ demonstrated below: ## Requirements -- Rust +- Rust for running the application - [rustup](https://rustup.rs/) is the recommended way of setting up a Rust toolchain -- libssl -- libclang -- git +- libssl for the HTTPS requests library +- libclang for building the database (RocksDB) +- git for cloning companions and updating them ## Environment variables From 294bf71f4d9766752a00ccc43e4e99731bd911b5 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Thu, 10 Mar 2022 13:35:22 -0300 Subject: [PATCH 15/22] improve gitlab mention --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 4e39412..286b080 100644 --- a/README.md +++ b/README.md @@ -278,10 +278,10 @@ review` to manage the results. # Deployment -When you push a deployment tag to GitHub, it will be mirrored to GitLab and then -the [CI pipeline](./.gitlab-ci.yml) will be ran on the GitLab Runners for -deploying the app. Deployment tags should conform to one of the following -patterns: +When you push a deployment tag to GitHub, it will be [mirrored to +GitLab](https://gitlab.parity.io/parity/parity-processbot) and then its [CI +pipeline](./.gitlab-ci.yml) will be ran for deploying the app. Deployment tags +should conform to one of the following patterns: - `/^v[0-9]+\.[0-9]+.*$/`, e.g. `v1.1`, will be deployed to production - The production instance is installed in From db70224a892b94fe1f34870d8fea6965e7a7fbfd Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Thu, 10 Mar 2022 13:35:59 -0300 Subject: [PATCH 16/22] reformat --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 286b080..c8e3b28 100644 --- a/README.md +++ b/README.md @@ -278,10 +278,10 @@ review` to manage the results. # Deployment -When you push a deployment tag to GitHub, it will be [mirrored to -GitLab](https://gitlab.parity.io/parity/parity-processbot) and then its [CI -pipeline](./.gitlab-ci.yml) will be ran for deploying the app. Deployment tags -should conform to one of the following patterns: +When you push a deployment tag to GitHub, it will be +[mirrored to GitLab](https://gitlab.parity.io/parity/parity-processbot) and then +its [CI pipeline](./.gitlab-ci.yml) will be run for deploying the app. +Deployment tags should conform to one of the following patterns: - `/^v[0-9]+\.[0-9]+.*$/`, e.g. `v1.1`, will be deployed to production - The production instance is installed in From 6d0944b775855d0d4a34ff8177349b8da78e0996 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Thu, 10 Mar 2022 13:37:52 -0300 Subject: [PATCH 17/22] wording --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index c8e3b28..cb9425a 100644 --- a/README.md +++ b/README.md @@ -306,9 +306,9 @@ See https://gitlab.parity.io/groups/parity/opstooling/-/wikis # Implementation -To have a better understanding on the Companion Build System we strongly -recommend to -[consult its explanation](https://github.com/paritytech/parity-processbot/issues/327). +Before reading any of this, we strongly recommend to have a good understanding +of the Companion Build System by +[consulting its explanation](https://github.com/paritytech/parity-processbot/issues/327). A [web server](https://github.com/paritytech/parity-processbot/blob/4b36d6dcb8dd6d2ba9063c28c1c61bff503c364d/src/server.rs#L88) From 2d9a36e9a51743d57850fa8b1f498bd049ee3a03 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Fri, 11 Mar 2022 13:20:03 -0300 Subject: [PATCH 18/22] clarify --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cb9425a..02c71a9 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ events it will resume processing all pending pull requests until they can be merged; a more detailed breakdown is available in the [Implementation section](#implementation). -Note that parity-processbot does not implement the Companion Build System's +Note that parity-processbot works independently of Companion Build System's cross-repository integration check, which is done on CI (see [check_dependent_project](https://github.com/paritytech/pipeline-scripts#check_dependent_project) for that). From 8efd7abb50d3c5ee30f89ff5d78a181574b88f08 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Fri, 11 Mar 2022 13:21:29 -0300 Subject: [PATCH 19/22] typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 02c71a9..4433441 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ events it will resume processing all pending pull requests until they can be merged; a more detailed breakdown is available in the [Implementation section](#implementation). -Note that parity-processbot works independently of Companion Build System's +Note that parity-processbot works independently of the Companion Build System's cross-repository integration check, which is done on CI (see [check_dependent_project](https://github.com/paritytech/pipeline-scripts#check_dependent_project) for that). From 9562021ea33ad49e09b5f4b9f666a07a0e38225d Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Fri, 11 Mar 2022 13:30:59 -0300 Subject: [PATCH 20/22] expand on deployment environments --- README.md | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 4433441..333aa7a 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,7 @@ Before starting to work on this project we recommend reading the - [Test repositories](#development-test-repositories) - [Deployment](#deployment) - [Logs](#deployment-logs) + - [Environments](#deployment-environments) # Commands @@ -278,31 +279,35 @@ review` to manage the results. # Deployment +All of the relevant configuration for deployment lives in the [./helm](./helm) +folder. The values for each specific environment are in +`helm/values-${ENVIRONMENT}.yml`. If you add a value, it needs to be used in +[helm/templates/processbot.yaml](helm/templates/processbot.yaml). + +## Logs + +See + +## Environments + When you push a deployment tag to GitHub, it will be [mirrored to GitLab](https://gitlab.parity.io/parity/parity-processbot) and then its [CI pipeline](./.gitlab-ci.yml) will be run for deploying the app. -Deployment tags should conform to one of the following patterns: -- `/^v[0-9]+\.[0-9]+.*$/`, e.g. `v1.1`, will be deployed to production - - The production instance is installed in - [Substrate](https://github.com/paritytech/substrate), - [Polkadot](https://github.com/paritytech/polkadot) and - [Cumulus](https://github.com/paritytech/cumulus) +The application can be deployed to the following environments: -- `/^pre-v[0-9]+\.[0-9]+.*$/`, e.g. `pre-v0.6` will be deployed to staging - - The staging instance is installed in the [test repositories](#development-test-repositories) +- Production: push a tag with the pattern `/^pre-v[0-9]+\.[0-9]+.*$/`, e.g. + `pre-v0.6` -The deployment's status can be followed through -[Gitlab Pipelines on the parity-processbot mirror](https://gitlab.parity.io/parity/parity-processbot/-/pipelines). + The production instance is installed in + [Substrate](https://github.com/paritytech/substrate), + [Polkadot](https://github.com/paritytech/polkadot) and + [Cumulus](https://github.com/paritytech/cumulus). -All of the relevant configuration for deployment lives in the [./helm](./helm) -folder. The values for each specific environment are in -`helm/values-${ENVIRONMENT}.yml`. If you add a value, it needs to be used in -[helm/templates/processbot.yaml](helm/templates/processbot.yaml). - -## Logs +- Staging: push a tag with the pattern `/^v[0-9]+\.[0-9]+.*$/`, e.g. `v1.1` -See https://gitlab.parity.io/groups/parity/opstooling/-/wikis + The staging instance is installed in the + [test repositories](#development-test-repositories). # Implementation From c43fd353b6d5c4a511e7e6d5987b86cbcf776bfe Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Fri, 11 Mar 2022 13:34:36 -0300 Subject: [PATCH 21/22] fix switch between production and staging environment --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 333aa7a..63dafc8 100644 --- a/README.md +++ b/README.md @@ -296,15 +296,15 @@ its [CI pipeline](./.gitlab-ci.yml) will be run for deploying the app. The application can be deployed to the following environments: -- Production: push a tag with the pattern `/^pre-v[0-9]+\.[0-9]+.*$/`, e.g. - `pre-v0.6` +- Production: push a tag with the pattern `/^v[0-9]+\.[0-9]+.*$/`, e.g. `v1.1` The production instance is installed in [Substrate](https://github.com/paritytech/substrate), [Polkadot](https://github.com/paritytech/polkadot) and [Cumulus](https://github.com/paritytech/cumulus). -- Staging: push a tag with the pattern `/^v[0-9]+\.[0-9]+.*$/`, e.g. `v1.1` +- Staging: push a tag with the pattern `/^pre-v[0-9]+\.[0-9]+.*$/`, e.g. + `pre-v0.6` The staging instance is installed in the [test repositories](#development-test-repositories). From 07b87755d3b695ed1131a85d0a59486ccee313b2 Mon Sep 17 00:00:00 2001 From: joao-paulo-parity Date: Thu, 17 Mar 2022 05:41:52 -0300 Subject: [PATCH 22/22] reorganize --- README.md | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 63dafc8..66322ef 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,7 @@ parity-processbot is a [GitHub App](https://docs.github.com/en/developers/apps/getting-started-with-apps/about-apps) which drives the [Companion Build System](https://github.com/paritytech/parity-processbot/issues/327)'s -merge process. It receives [commands](#commands) and from subsequent GitHub -events it will resume processing all pending pull requests until they can be -merged; a more detailed breakdown is available in the -[Implementation section](#implementation). +merge process. Note that parity-processbot works independently of the Companion Build System's cross-repository integration check, which is done on CI (see @@ -19,6 +16,7 @@ Before starting to work on this project we recommend reading the # TOC +- [How it works](#how-it-works) - [Commands](#commands) - [Relation to CI](#commands-relation-to-ci) - [Criteria for merge](#criteria-for-merge) @@ -32,13 +30,22 @@ Before starting to work on this project we recommend reading the - [Environment variables](#setup-environment-variables) - [Development](#development) - [Run the application](#development-run) - - [Integration tests](#development-integration-tests) - [Example workflows](#development-example-workflows) - [Test repositories](#development-test-repositories) + - [Integration tests](#development-integration-tests) - [Deployment](#deployment) - [Logs](#deployment-logs) - [Environments](#deployment-environments) +# How it works + +processbot receives [commands](#commands) from pull request comments. + +The merge commands will either merge the pull request right away, if possible, +or merge it automatically once all of its requirements are passing; a guided +description of how that works internally is provided in the +[Implementation section](#implementation). + # Commands The following commands should be posted as pull request comments. **Your whole @@ -225,16 +232,6 @@ application starts. the repositories where you have installed the app or the [test repositories](#development-test-repositories) after a deployment -## Test repositories - -The staging instance is installed in the following repositories: - -- https://github.com/paritytech/main-for-processbot-staging -- https://github.com/paritytech/companion-for-processbot-staging - -The GitHub App for staging is managed by -[paritytech](http://github.com/paritytech)'s Organizational GitHub Admins. - ## Example workflows ### Single merge use-case @@ -265,6 +262,16 @@ Steps: 6. Observe that the the pull request in Repository A will be merged first and the pull request on Repository B will be merged after +## Test repositories + +The staging instance is installed in the following repositories: + +- https://github.com/paritytech/main-for-processbot-staging +- https://github.com/paritytech/companion-for-processbot-staging + +The GitHub App for staging is managed by +[paritytech](http://github.com/paritytech)'s Organizational GitHub Admins. + ## Integration tests The integration tests are executed as follows: