Skip to content

feat(npm): support Yarn 2 offline cache and zero-installs #7220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JamieMagee merged 11 commits into from
Oct 19, 2020
Merged

feat(npm): support Yarn 2 offline cache and zero-installs #7220

Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
11f1562
refactor(npm): extract updateYarnOffline function
ylemkimon Sep 8, 2020
06ac7aa
feat(npm): support Yarn 2 offline cache and zero-installs
ylemkimon Sep 8, 2020
de99046
refactor: fix typo UpdatedArtifcats
ylemkimon Sep 9, 2020
6830063
Merge branch 'master' into yarn-2-zero-installs
rarkins Sep 9, 2020
b69f8eb
Merge branch 'master' into yarn-2-zero-installs
rarkins Sep 9, 2020
15f6f02
Merge branch 'master' into yarn-2-zero-installs
rarkins Sep 9, 2020
06342b6
fix: check for .yarnrc.yml first before .yarnrc
ylemkimon Sep 9, 2020
65b1c88
Merge branch 'master' into yarn-2-zero-installs
ylemkimon Sep 9, 2020
7e9a093
Merge branch 'master' into yarn-2-zero-installs
rarkins Sep 9, 2020
44ca5f4
Merge branch 'master' into yarn-2-zero-installs
JamieMagee Oct 19, 2020
1c3c988
Merge branch 'master' into yarn-2-zero-installs
JamieMagee Oct 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 67 additions & 40 deletions lib/manager/npm/post-update/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import path from 'path';
import is from '@sindresorhus/is';
import { parseSyml } from '@yarnpkg/parsers';
import upath from 'upath';
import { SYSTEM_INSUFFICIENT_DISK_SPACE } from '../../../constants/error-messages';
import { id as npmId } from '../../../datasource/npm';
Expand Down Expand Up @@ -274,7 +275,7 @@ interface ArtifactError {
stderr: string;
}

interface UpdatedArtifcats {
interface UpdatedArtifacts {
name: string;
contents: string | Buffer;
}
Expand Down Expand Up @@ -336,9 +337,71 @@ async function resetNpmrcContent(
}
}

// istanbul ignore next
async function updateYarnOffline(
lockFileDir: string,
localDir: string,
updatedArtifacts: UpdatedArtifacts[]
): Promise<void> {
try {
const resolvedPaths: string[] = [];
const yarnrc = await getFile(upath.join(lockFileDir, '.yarnrc'));
const yarnrcYml = await getFile(upath.join(lockFileDir, '.yarnrc.yml'));

if (yarnrc) {
Copy link

@arcanis arcanis Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd recommend to switch those two blocks (first check for .yarnrc.yml, then for .yarnrc). In my work repo for instance, we have both a .yarnrc and a .yarnrc.yml, with the first one pointing to a binary explaining that the global Yarn isn't recent enough because it doesn't read .yarnrc.yml, and needs to be upgraded.

Copy link
Contributor

@rarkins rarkins Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a minimum version of Yarn v1 that supports "bootstrapping" Yarn v2?

Copy link

@arcanis arcanis Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yarn 1 reading paths from .yarnrc.yml started with 1.18.0. Before that it was still possible, but you had to put the right path in both files.

Copy link
Contributor Author

@ylemkimon ylemkimon Sep 9, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the PR 😄

// Yarn 1 (offline mirror)
const mirrorLine = yarnrc
.split('\n')
.find((line) => line.startsWith('yarn-offline-mirror '));
if (mirrorLine) {
const mirrorPath = mirrorLine
.split(' ')[1]
.replace(/"/g, '')
.replace(/\/?$/, '/');
resolvedPaths.push(upath.join(lockFileDir, mirrorPath));
}
} else if (yarnrcYml) {
// Yarn 2 (offline cache and zero-installs)
const config = parseSyml(yarnrcYml);
Copy link

@arcanis arcanis Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fwiw, the name is a bit misleading but it's regular YAML, so you can use any parser you want, not necessarily the Yarn one. Or you can use the Configuration class from @yarnpkg/core (which would take care of the default values, etc), but it's perhaps a bit more complicated than necessary.

Copy link
Contributor Author

@ylemkimon ylemkimon Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arcanis Thank you for your valuable comments!

parseSyml is already used to parse the lockfile:

renovate/lib/manager/npm/extract/yarn.ts

Line 15 in c65d400

const parsed = parseSyml(yarnLockRaw);
and @yarnpkg/parsers seems to use js-yaml so I think it's OK and would give more consistent behavior with Yarn, e.g., empty file handling.

resolvedPaths.push(
upath.join(lockFileDir, config.cacheFolder || './.yarn/cache')
);

resolvedPaths.push(upath.join(lockFileDir, '.pnp'));
if (config.pnpDataPath) {
resolvedPaths.push(upath.join(lockFileDir, config.pnpDataPath));
}
}
logger.debug({ resolvedPaths }, 'updateYarnOffline resolvedPaths');

if (resolvedPaths.length) {
const status = await getRepoStatus();
for (const f of status.modified.concat(status.not_added)) {
if (resolvedPaths.some((p) => f.startsWith(p))) {
const localModified = upath.join(localDir, f);
updatedArtifacts.push({
name: f,
contents: await readFile(localModified),
});
}
}
for (const f of status.deleted || []) {
if (resolvedPaths.some((p) => f.startsWith(p))) {
updatedArtifacts.push({
name: '|delete|',
contents: f,
});
}
}
}
} catch (err) {
logger.error({ err }, 'Error updating yarn offline packages');
}
}

export interface WriteExistingFilesResult {
artifactErrors: ArtifactError[];
updatedArtifacts: UpdatedArtifcats[];
updatedArtifacts: UpdatedArtifacts[];
}
// istanbul ignore next
export async function getAdditionalFiles(
Expand All @@ -347,7 +410,7 @@ export async function getAdditionalFiles(
): Promise<WriteExistingFilesResult> {
logger.trace({ config }, 'getAdditionalFiles');
const artifactErrors: ArtifactError[] = [];
const updatedArtifacts: UpdatedArtifcats[] = [];
const updatedArtifacts: UpdatedArtifacts[] = [];
if (!packageFiles.npm?.length) {
return { artifactErrors, updatedArtifacts };
}
Expand Down Expand Up @@ -538,43 +601,7 @@ export async function getAdditionalFiles(
name: lockFileName,
contents: res.lockFile,
});
// istanbul ignore next
try {
const yarnrc = await getFile(upath.join(lockFileDir, '.yarnrc'));
if (yarnrc) {
const mirrorLine = yarnrc
.split('\n')
.find((line) => line.startsWith('yarn-offline-mirror '));
if (mirrorLine) {
const mirrorPath = mirrorLine
.split(' ')[1]
.replace(/"/g, '')
.replace(/\/?$/, '/');
const resolvedPath = upath.join(lockFileDir, mirrorPath);
logger.debug('Found yarn offline mirror: ' + resolvedPath);
const status = await getRepoStatus();
for (const f of status.modified.concat(status.not_added)) {
if (f.startsWith(resolvedPath)) {
const localModified = upath.join(config.localDir, f);
updatedArtifacts.push({
name: f,
contents: await readFile(localModified),
});
}
}
for (const f of status.deleted || []) {
if (f.startsWith(resolvedPath)) {
updatedArtifacts.push({
name: '|delete|',
contents: f,
});
}
}
}
}
} catch (err) {
logger.error({ err }, 'Error updating yarn offline packages');
}
await updateYarnOffline(lockFileDir, config.localDir, updatedArtifacts);
} else {
logger.debug("yarn.lock hasn't changed");
}
Expand Down