Skip to content

Shrink Unicode tables (even more) #70486

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bors merged 14 commits into from
Mar 28, 2020
Merged

Shrink Unicode tables (even more) #70486

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
903f67d
Avoid re-fetching Unicode data
Mark-Simulacrum Mar 19, 2020
7c4baed
Dynamically choose best chunk size
Mark-Simulacrum Mar 19, 2020
580a634
Generate tests for Unicode property data
Mark-Simulacrum Mar 19, 2020
6c7691a
Pre-pop zero chunks before mapping LAST_CHUNK_MAP
Mark-Simulacrum Mar 20, 2020
b0e121d
Shrink bitset words through functional mapping
Mark-Simulacrum Mar 21, 2020
7b29b70
Add a right shift mapping
Mark-Simulacrum Mar 21, 2020
5f71d98
Deduplicate test and primary range_search definitions
Mark-Simulacrum Mar 21, 2020
233ab2f
Push the byte of LAST_CHUNK_MAP into the array
Mark-Simulacrum Mar 21, 2020
a7ec6f8
Arrange for zero to be canonical
Mark-Simulacrum Mar 21, 2020
af243d4
Avoid relying on const parameters to function
Mark-Simulacrum Mar 21, 2020
33b9e6f
Add richer printing
Mark-Simulacrum Mar 24, 2020
9c1ceec
Add skip list based implementation for smaller encoding
Mark-Simulacrum Mar 26, 2020
b6bc906
Remove separate encoding for a single nonzero-mapping byte
Mark-Simulacrum Mar 27, 2020
ad679a7
Update the documentation comment
Mark-Simulacrum Mar 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Avoid re-fetching Unicode data 
If the unicode-downloads folder already exists, we likely just fetched the data,
so don't make any further network requests. Unicode versions are released rarely
enough that this doesn't matter much in practice.
  • Loading branch information
@Mark-Simulacrum
Mark-Simulacrum committed Mar 20, 2020
commit 903f67d599cf12d2d202d5177bb3edecb6cdab00
11 changes: 8 additions & 3 deletions src/tools/unicode-table-generator/src/unicode_download.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,15 @@ static RESOURCES: &[&str] =

pub fn fetch_latest() {
let directory = Path::new(UNICODE_DIRECTORY);
if directory.exists() {
eprintln!(
"Not refetching unicode data, already exists, please delete {:?} to regenerate",
directory
);
return;
}
if let Err(e) = std::fs::create_dir_all(directory) {
if e.kind() != std::io::ErrorKind::AlreadyExists {
panic!("Failed to create {:?}: {}", UNICODE_DIRECTORY, e);
}
panic!("Failed to create {:?}: {}", UNICODE_DIRECTORY, e);
}
let output = Command::new("curl").arg(URL_PREFIX.to_owned() + README).output().unwrap();
if !output.status.success() {
Expand Down