-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Preserve artifact cache unless stale #1918
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit
Hold shift + click to select a range
a563e25
make pruning explicit
eagr f31d354
preserve cache unless stale
eagr 195bbce
barely working
eagr 99aa012
use ArtifactId::from_file_name()
eagr 17973ef
ignore non-unicode file names
eagr e81b456
generalize concat_const!()
eagr 851a77a
per advices
eagr a3b0fcf
break on IO error
eagr b323c28
make pruning sound
eagr a2808f8
log more events
eagr 0657d41
refactor
eagr 0384ae7
doc
eagr 4b1bb0a
Refactor indentation
mrcnski a8bcce4
refactor
eagr 50b7ccc
checksum poc
eagr b6c1a07
Revert "checksum poc"
eagr 2ad5262
redo checksum p1
eagr 3723806
p2
eagr e44c451
remove corrupted cache
eagr 6c19164
diversify results
eagr dad5285
fix tests
eagr 1ede201
fix pruning
eagr 2d51b52
fix message serialization
eagr 36a33e8
clean up
eagr 69f6a44
retire path_prefix()
eagr d3254f5
improve test
eagr f4d22fa
Merge branch 'master' into preserve-art
mrcnski 39448ff
Fix test
mrcnski 3a2c1cd
cargo fmt
mrcnski a02cb06
as per advices
eagr 53e4557
tag artifact with runtime version
eagr d4f3083
fix tests
eagr 9d2142e
Merge branch 'master' into preserve-art
mrcnski 931aae1
upstream build fn to substrate
eagr 5de4e8e
glitch
eagr a0b71c5
wrong attribution
eagr c85adfe
as per suggestions
eagr b897f1c
glitch
eagr 89ada31
prevent `cargo tree` from accessing network
eagr 868426a
glitch
eagr 652bec7
Merge branch 'master' into preserve-art
eagr File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
refactor
- Loading branch information
commit a8bcce4eba245eac451ee476c3811ebcf701cfeb
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -64,7 +64,6 @@ use polkadot_parachain_primitives::primitives::ValidationCodeHash; | |||||
| use polkadot_primitives::ExecutorParamsHash; | ||||||
| use std::{ | ||||||
| collections::HashMap, | ||||||
| future::Future, | ||||||
| path::{Path, PathBuf}, | ||||||
| str::FromStr as _, | ||||||
| time::{Duration, SystemTime}, | ||||||
|
|
@@ -231,6 +230,73 @@ impl Artifacts { | |||||
| /// valid, e.g., matching the current node version. The ones deemed invalid will be pruned. | ||||||
| pub async fn new_and_prune(cache_path: &Path) -> Self { | ||||||
| let mut artifacts = Self { inner: HashMap::new() }; | ||||||
| artifacts.insert_and_prune(cache_path).await; | ||||||
| artifacts | ||||||
| } | ||||||
|
|
||||||
| async fn insert_and_prune(&mut self, cache_path: &Path) { | ||||||
| fn is_stale(file_name: &str) -> bool { | ||||||
| !file_name.starts_with(ARTIFACT_PREFIX) | ||||||
| } | ||||||
|
|
||||||
| // Inserts the entry into the artifacts table if it is a valid artifact file, or prune it | ||||||
| // otherwise. | ||||||
| async fn insert_or_prune( | ||||||
| artifacts: &mut Artifacts, | ||||||
| entry: &tokio::fs::DirEntry, | ||||||
| cache_path: &Path, | ||||||
| ) { | ||||||
| let file_type = entry.file_type().await; | ||||||
| let file_name = entry.file_name(); | ||||||
|
|
||||||
| match file_type { | ||||||
| Ok(file_type) => | ||||||
| if !file_type.is_file() { | ||||||
| return | ||||||
| }, | ||||||
| Err(err) => { | ||||||
| gum::warn!( | ||||||
| target: LOG_TARGET, | ||||||
| ?err, | ||||||
| "unable to get file type for {:?}", | ||||||
| file_name, | ||||||
| ); | ||||||
| return | ||||||
| }, | ||||||
| } | ||||||
|
|
||||||
| if let Some(file_name) = file_name.to_str() { | ||||||
| let id = ArtifactId::from_file_name(file_name); | ||||||
| let file_path = cache_path.join(file_name); | ||||||
|
|
||||||
| if is_stale(file_name) || id.is_none() { | ||||||
| gum::debug!( | ||||||
| target: LOG_TARGET, | ||||||
| "discarding invalid artifact {:?}", | ||||||
| &file_path, | ||||||
| ); | ||||||
| let _ = tokio::fs::remove_file(&file_path).await; | ||||||
| return | ||||||
| } | ||||||
|
|
||||||
| if let Some(id) = id { | ||||||
| artifacts.insert_prepared(id, SystemTime::now(), Default::default()); | ||||||
| gum::debug!( | ||||||
| target: LOG_TARGET, | ||||||
| "reusing {:?} for node version v{}", | ||||||
|
||||||
| "reusing {:?} for node version v{}", | |
| "reusing existing artifact {:?} for node version v{}", |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: There are three warn logs in this fn ("discarding invalid artifact" should be a warn). We could reduce some LoC by returning an error and logging in a single place, at the call site.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gonna punt on this, coz only one of them is
Err