WIP: Enable continuous package release to crates.io #9417

Xanewok · 2021-07-22T14:16:18Z

...rather than doing it only on select tags. This should publish only versions that are not published yet.

Related #7634

...rather than using an explicit filter in CI jobs.

TriplEight · 2021-07-22T14:47:50Z

.gitlab-ci.yml

  <<:                              *docker-env
  <<:                              *test-refs-no-trigger
  script:
    - cargo install cargo-unleash ${CARGO_UNLEASH_INSTALL_PARAMS}


Is it mature enough to be installed into CI image?

Probably - until it's not v1 I'd prefer not to; the time savings is miniscule I'd say comparing to the actual checking done by the tool later on

not on every commit tho

TriplEight · 2021-07-22T14:48:45Z

.gitlab-ci.yml

 unleash-check:
  stage:                           test
  <<:                              *docker-env
  <<:                              *test-refs-no-trigger


do you really want to publish before the tests are green?

That's the unleash-check - I think it should be run early/at the same time as tests; only publishing should be done after every test is green

In my view (I actually do not know what does it check, Check whether crates can be packaged in the docs is probably not enough) it's an overkill to run these checks on every commit. We should run them before the release target, so maybe on master, or even nightly?

If I understand this correctly, what you propose may lead to the PR checks being green at first but only discovered to be red once merged. If that's the case, then won't be much incentive to fix this once a PR is merged and it'll just hang around being red until someone disables it again - I'd rather we catch that early.

If this is deemed to expensive to run on every commit, we might want to run this as an extra step for bot merge to not merge a given PR until we know for sure that this check is green.

It is expensive, the job takes 30 minutes. Any possible way to combine it with something to reuse the cache?

Ideally, it will run on the pre-merge pipeline and a merge-train, but they are not ready yet.

Maybe it's better to either check only if certain files are changed or versions got bumped?

TriplEight · 2021-07-22T14:52:50Z

.gitlab-ci.yml

 publish-to-crates-io:
  stage:                           publish
+  needs:
+    - job:                         unleash-check


Either you should list the jobs necessary to be green in order to start this job, because this way it starts right after unleash-check without waiting anything else. Or remove this needs and the stage won't start if the previous jobs are not successful. I prefer the latter.

Can we somehow make it so that it runs at publish but also make sure that unleash-check is never allow_failure: true?

By publish I mean only after everything else completes from the test stage and so on

We can run unleash check on the build stage, so after all the code tests pass, and unleash em-dragons on the publish phase after everything's proven to be buildable.

I can move this to the build stage if you want me to, running parallel to the tests is also good (at least wrt developer feedback latency, maybe not so much wrt CI builder overhead 😀 )

it's worth it, the test stage already brings a lot of concurrent jobs. The longest job in the test stage is test-linux-stable , which takes 36 minutes. This one takes ~32. The longest on the build stage is check-polkadot-companion-build , so unless we make use of caching (or simplify it somehow) it will severely contribute to the pipelines concurrency.

TriplEight · 2021-07-22T15:02:46Z

@s3krit what's your take on releasing on every commit to master? That's something even beyond nightly releases in my view.

TriplEight · 2021-07-22T15:15:24Z

.gitlab-ci.yml

 publish-to-crates-io:
  stage:                           publish
+  needs:
+    - job:                         unleash-check


We can run unleash check on the build stage, so after all the code tests pass, and unleash em-dragons on the publish phase after everything's proven to be buildable.

TriplEight · 2021-07-22T15:16:35Z

.gitlab-ci.yml

+    # cycle and cannot proceed with publishing/checking
+    - cargo unleash de-dev-deps
+    - cargo unleash em-dragons --no-check --owner github:paritytech:core-devs --dry-run
  allow_failure:                   true


this may be removed as soon as it will publish stably.

bkchr · 2021-07-23T10:01:59Z

This means we can finally increase a crate version and it will be published?

s3krit · 2021-07-23T12:39:30Z

@s3krit what's your take on releasing on every commit to master? That's something even beyond nightly releases in my view.

My general view would be that it might be better for developers to have the power themselves to bump versions and only then would it get pushed to crates.io. I.e., rather than us using CI to detect whether or not we need to push to crates.io or whatever, we simply let the much smarter developers decide whether they want to release a new version of a crate (and subsequent dependencies).

That being said, if we're not really doing much in the way of additional testing/QA once something's been committed to master, I'm not really sure what the drawback is. If we're happy for something to be merged with master, due to the way we currently use substrate and builders use substrate, we're basically saying 'this is OK to use' (as I understand it). And pushing to crates.io just makes it easier for developers. Of course, I could be entirely wrong here.

TriplEight

Right, let's publish just on version bumps.

.gitlab-ci.yml

gnunicorn · 2021-07-29T07:55:04Z

This means we can finally increase a crate version and it will be published?

Yes.

We should run cargo unleash em-dragons only on the version bumps, please return it -- @TriplEight

That is inconsistent with the requirement from @bkchr just stated above. Why should this not be run unless a specific tag is set?

@s3krit what's your take on releasing on every commit to master? That's something even beyond nightly releases in my view.

This discussing is irrelevant, as this is not what is happening here.

To clarify: cargo unleash em-dragons first syncs the latest crates.io versions, then checks the local tree against them. Only the crates that locally have a version not found on crates.io (and don't have a publish = false flag in their Cargo.toml) will be attempted to be released. If the PR didn't change any version (for the majority of PRs) – no matter what others changes have occurred – this does exactly nothing, states that all is fine and quits.

TriplEight · 2021-07-29T08:00:04Z

Ok, this makes sense then. Thanks @gnunicorn . Then it should run on master.

Xanewok · 2021-08-03T11:13:15Z

@ascjones do we need to wait until #8615 is merged?

ascjones · 2021-08-03T11:17:05Z

@ascjones do we need to wait until #8615 is merged?

If this PR will release frame-metadata from this repo, then yes.

If it won't release that crate then should be good to go.

Xanewok · 2021-08-03T11:24:49Z

AIUI an alternative would be to proceed with publishing the 14.0.0-dev here, and yank it once #8615 is merged (which also would properly migrate to an "actual" v14, so to speak); I'm not sure how much time #8615 needs.

ascjones · 2021-08-03T11:29:31Z

I'm not sure how much time #8615 needs.

It's waiting for another review, but not sure how long that will take tbh.

an alternative would be to proceed with publishing the 14.0.0-dev here

That might work, though I would need to change the deps on my PRs to use the git dependency which would break the unleash CI. But I think that has changed now to be non-mandatory.

TriplEight · 2021-08-03T11:27:56Z

.gitlab-ci.yml

 unleash-check:
  stage:                           test
  <<:                              *docker-env
  <<:                              *test-refs-no-trigger


Maybe it's better to either check only if certain files are changed or versions got bumped?

.gitlab-ci.yml

Co-authored-by: Denis Pisarev <[email protected]>

stale · 2021-09-02T12:29:28Z

Hey, is anyone still working on this? Due to the inactivity this issue has been automatically marked as stale. It will be closed if no further activity occurs. Thank you for your contributions.

TriplEight · 2021-09-13T15:10:35Z

.gitlab-ci.yml

  stage:                           publish
+  needs:
+    - job:                         unleash-check
  <<:                              *docker-env


Suggested change

<<: *docker-env

<<: *docker-env

<<: *build-refs

TriplEight · 2021-09-13T15:14:31Z

.gitlab-ci.yml

 unleash-check:
  stage:                           test
  <<:                              *docker-env
  <<:                              *test-refs-no-trigger


Suggested change

<<: *test-refs-no-trigger

<<: *build-refs

TriplEight · 2021-09-13T15:20:11Z

cc #9705

stale · 2021-10-13T18:42:07Z

Hey, is anyone still working on this? Due to the inactivity this issue has been automatically marked as stale. It will be closed if no further activity occurs. Thank you for your contributions.

Xanewok added 3 commits July 22, 2021 15:40

Run de-dev deps manually in publish-to-crates step

71e7e22

Mark packages we don't want to publish as such in the manifest

9ef1b1f

...rather than using an explicit filter in CI jobs.

WIP: See if we can unleash packages

e05a582

Xanewok marked this pull request as ready for review July 22, 2021 14:19

Xanewok requested a review from a team as a code owner July 22, 2021 14:19

oops

26ba28c

TriplEight approved these changes Jul 22, 2021

View reviewed changes

TriplEight self-requested a review July 22, 2021 14:53

TriplEight reviewed Jul 22, 2021

View reviewed changes

TriplEight suggested changes Jul 28, 2021

View reviewed changes

.gitlab-ci.yml Show resolved Hide resolved

ascjones mentioned this pull request Jul 30, 2021

Release v14.0.0-rc.1 paritytech/frame-metadata#22

Merged

TriplEight suggested changes Aug 3, 2021

View reviewed changes

Update .gitlab-ci.yml

48c847d

Co-authored-by: Denis Pisarev <[email protected]>

stale bot added the A5-stale Pull request did not receive any updates in a long time. No review needed at this stage. Close it. label Sep 2, 2021

TriplEight suggested changes Sep 13, 2021

View reviewed changes

stale bot removed the A5-stale Pull request did not receive any updates in a long time. No review needed at this stage. Close it. label Sep 13, 2021

stale bot added the A5-stale Pull request did not receive any updates in a long time. No review needed at this stage. Close it. label Oct 13, 2021

bkchr closed this Oct 14, 2021

WIP: Enable continuous package release to crates.io #9417

WIP: Enable continuous package release to crates.io #9417

Uh oh!

Conversation

Xanewok commented Jul 22, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TriplEight commented Jul 22, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bkchr commented Jul 23, 2021

Uh oh!

s3krit commented Jul 23, 2021

Uh oh!

TriplEight left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gnunicorn commented Jul 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TriplEight commented Jul 29, 2021

Uh oh!

Xanewok commented Aug 3, 2021

Uh oh!

ascjones commented Aug 3, 2021

Uh oh!

Xanewok commented Aug 3, 2021

Uh oh!

ascjones commented Aug 3, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stale bot commented Sep 2, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TriplEight commented Sep 13, 2021

Uh oh!

stale bot commented Oct 13, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

gnunicorn commented Jul 29, 2021 •

edited

Loading