refactor: Change monorepo tooling to Turborepo from NX for easier caching #419
Conversation
ENG-544 Switch to TurboRepo from NX
NX is increasingly locking their features behind a paywall which leads to uncertainty around the future of its feature sets that will be usable without cloud lock-in. TurboRepo (at least for now) is more open and allows easier remote caching. Therefore, we will aim to switch to TurboRepo for now since we use NX minimally in any case. Blog: https://medium.com/daveahern/a-cautionary-tale-of-migrating-to-nx-211f18ce8c4f |
🦋 Changeset detectedLatest commit: da28982 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
WalkthroughThe pull request introduces a comprehensive transition from NX to Turborepo as the monorepo tooling across the project. This migration involves updating configuration files, modifying GitHub Actions workflows, and adjusting package scripts to use Turborepo's commands and caching mechanisms. Multiple packages under the Changes
Sequence DiagramsequenceDiagram
participant Dev as Developer
participant Turbo as Turborepo
participant CI as GitHub Actions
Dev->>Turbo: Run build/test/lint
Turbo-->>Turbo: Check cache
Turbo->>Turbo: Execute affected tasks
Turbo-->>Dev: Return results
Dev->>CI: Push changes
CI->>Turbo: Run workflows
Turbo-->>Turbo: Cached task execution
Turbo-->>CI: Workflow results
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (5)
packages/project-builder-test/turbo.json (1)
1-11: LGTM! Well-structured Turbo configurationThe configuration correctly:
- Extends the root config
- Sets up build dependencies
- Configures appropriate input patterns
- Allows TURBO_* environment variables to pass through
Consider adding outputs configuration if the e2e tests generate any artifacts that could be cached.
packages/ui-components/turbo.json (1)
1-12: LGTM! Appropriate configuration for UI componentsThe configuration correctly handles:
- Build task with proper input/output patterns
- Storybook build outputs
- Sensible exclusions for README and story files
Consider adding cache configuration for the build:storybook task to improve build times.
tests/simple/turbo.json (1)
1-26: LGTM! Comprehensive task configurationThe configuration properly handles:
- Task dependencies and build order
- Cache settings for different tasks
- Output logging configuration
- Persistent tasks
Two suggestions for consideration:
- Add
cacheconfiguration forprettier:checkto improve performance- Consider adding
outputMode: "hash-only"for thelinttask to reduce log noiseturbo.json (1)
1-36: Consider additional Turborepo optimizationsThe task configuration looks good, but consider these Turborepo-specific optimizations:
- Add
globalDependenciesfor files that affect all tasks (e.g.,tsconfig.json,.env)- Configure
pipelineto enable parallel execution where possibleHere's an enhanced configuration:
{ "$schema": "https://turbo.build/schema.json", + "globalDependencies": ["tsconfig.json", ".env"], + "pipeline": { "tasks": { "build": { "dependsOn": ["^build"], "inputs": ["$TURBO_DEFAULT$", "!README.md"], - "outputs": ["build/**", "dist/**", ".next/**", "!.next/cache/**"] + "outputs": ["build/**", "dist/**", ".next/**", "!.next/cache/**"], + "dependsOn": ["^build"] }, // ... other tasks ... } }.github/workflows/test.yml (1)
Line range hint
13-14: Update outdated commentThe comment still references Nx affected commands, but we've migrated to Turborepo.
- # We need to fetch all branches and commits so that Nx affected has a base to compare against. + # We need to fetch all branches and commits so that Turborepo can determine affected packages.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Lite
⛔ Files ignored due to path filters (2)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yamltests/simple/pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (19)
.changeset/silent-laws-teach.md(1 hunks).github/workflows/lint.yml(1 hunks).github/workflows/test-e2e.yml(2 hunks).github/workflows/test.yml(1 hunks).gitignore(1 hunks)nx.json(0 hunks)package.json(1 hunks)packages/create-project/templates/.gitignore(1 hunks)packages/project-builder-test/src/environment/index.ts(1 hunks)packages/project-builder-test/turbo.json(1 hunks)packages/tools/package.json(0 hunks)packages/ui-components/package.json(0 hunks)packages/ui-components/turbo.json(1 hunks)plugins/baseplate-plugin-storage/package.json(1 hunks)tests/simple/baseplate/project-definition.json(2 hunks)tests/simple/nx.json(0 hunks)tests/simple/package.json(1 hunks)tests/simple/turbo.json(1 hunks)turbo.json(1 hunks)
💤 Files with no reviewable changes (4)
- packages/tools/package.json
- tests/simple/nx.json
- nx.json
- packages/ui-components/package.json
✅ Files skipped from review due to trivial changes (1)
- .gitignore
🔇 Additional comments (12)
packages/project-builder-test/src/environment/index.ts (1)
72-72: Verify environment variable inheritance with TurborepoThe addition of
extendEnv: trueensures environment variables are inherited from the parent process. However, settingCI: 'true'might interfere with Turborepo's own CI detection.Run this script to check how Turborepo handles CI detection:
Also applies to: 73-78
✅ Verification successful
Environment configuration changes are safe to proceed ✅
The
extendEnv: trueandCI: 'true'settings won't interfere with Turborepo's functionality. The CI flag is only used for test configurations and display optimizations, while environment inheritance is a consistent pattern throughout the codebase.🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Check if Turborepo has its own CI detection logic rg -A 5 "process.env.CI" .Length of output: 1173
Script:
#!/bin/bash # Check for CI-related patterns in configuration files and scripts rg -g "package.json" -g "*.yaml" -g "*.yml" '"CI":' # Look for other CI environment checks rg -i "continuous integration|ci mode|ci environment" . # Check for CI-related configuration files fd "^\.ci" --type fLength of output: 339
Script:
#!/bin/bash # Check for environment variable handling patterns rg -g "*.ts" -g "*.js" "process\.env\." -A 3 # Check hidden CI config files fd -H "^\.ci" --type f # Look for CI environment setup rg -g "*.ts" -g "*.js" "env: \{" -A 3Length of output: 8867
tests/simple/package.json (1)
9-15: LGTM! Scripts correctly migrated to TurborepoThe migration from NX to Turborepo looks good. All scripts have been properly updated to use
turbo runcommands.Let's verify the script commands are consistent across all workspace packages:
Also applies to: 18-18, 20-20
✅ Verification successful
✅ Migration to Turborepo verified successfully
All workspace packages have been properly migrated:
- No NX dependencies or commands found in any package.json
- Workspace root correctly uses turbo for orchestration
- Package-specific scripts maintain their direct tool commands (tsc, vite, vitest, etc.)
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check for consistency in turbo script commands across packages # Look for any package.json files that might still be using nx commands rg -g "package.json" "nx run|@nrwl" .Length of output: 37
Script:
#!/bin/bash # Find all package.json files and extract their scripts section echo "=== Checking scripts in all package.json files ===" fd package.json --exec sh -c 'echo "\nFile: {}"; jq -r ".scripts // empty" {}' echo -e "\n=== Checking for any remaining NX dependencies ===" fd package.json --exec sh -c 'echo "\nFile: {}"; jq -r ".dependencies,.devDependencies | select(. != null) | to_entries[] | select(.key | contains(\"nx\") or contains(\"nrwl\")) | .key" {}'Length of output: 10585
.changeset/silent-laws-teach.md (1)
1-20: LGTM! Comprehensive package updates for Turborepo migrationThe changeset appropriately marks all affected packages for a patch version bump, which is the correct semantic versioning choice for this tooling change.
.github/workflows/test.yml (1)
21-22: Avoid pinning to specific commit hashUsing a pinned commit hash for the caching action could be risky. Consider using a version tag instead.
.github/workflows/lint.yml (2)
Line range hint
13-14: Update outdated commentThe comment still references Nx affected commands, but we've migrated to Turborepo.
22-23: Avoid pinning to specific commit hashUsing a pinned commit hash for the caching action could be risky. Consider using a version tag instead.
package.json (2)
8-10: Verify Turborepo command structure and filtersThe migration to Turborepo looks good, but let's verify the command structure:
Run this script to check if the Turborepo pipeline configuration matches these commands:
Also applies to: 13-15, 17-23, 26-26
✅ Verification successful
Turborepo configuration and commands are correctly structured
The migration to Turborepo is properly configured with:
- Valid pipeline configuration in turbo.json matching package.json scripts
- Correct task dependencies and cache settings
- Proper workspace setup supporting --affected filters
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify Turborepo pipeline configuration matches package.json scripts # Check if turbo.json exists and contains pipeline configuration if [ -f "turbo.json" ]; then echo "Checking turbo.json pipeline configuration..." cat turbo.json else echo "Error: turbo.json not found. Please ensure Turborepo is properly configured." exit 1 fi # List all workspace packages to verify filter syntax echo "Verifying workspace package names for filter syntax..." find . -name "package.json" -not -path "*/node_modules/*" -exec jq -r '.name' {} \;Length of output: 1642
34-34: Verify Turborepo version compatibilityThe Turborepo version (2.3.3) should be verified for compatibility with the Node.js version specified in the engines field (^20.0.0).
Run this script to check for any known issues or compatibility problems:
✅ Verification successful
Turborepo version 2.3.3 is compatible with Node.js ^20.0.0
The version is verified to be the latest release and has no documented Node.js version restrictions or compatibility issues.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check Turborepo version compatibility and latest releases # Get latest Turborepo version from npm echo "Checking latest Turborepo version..." curl -s https://registry.npmjs.org/turbo/latest | jq -r '.version' # Check for known issues in GitHub releases echo "Checking for known issues in version 2.3.3..." gh api repos/vercel/turbo/releases | jq -r '.[] | select(.tag_name=="v2.3.3") | .body'Length of output: 660
Script:
#!/bin/bash # Check Turborepo's Node.js version requirements echo "Checking Turborepo's package.json for Node.js requirements..." curl -s https://registry.npmjs.org/turbo/2.3.3 | jq -r '.engines' # Search for Node.js version mentions in README echo -e "\nChecking documentation for Node.js requirements..." curl -s https://raw.githubusercontent.com/vercel/turbo/v2.3.3/README.md | grep -i "node" || trueLength of output: 419
.github/workflows/test-e2e.yml (1)
34-44: Playwright setup looks goodThe Playwright browser caching and version detection have been properly reorganized in the workflow.
packages/create-project/templates/.gitignore (1)
138-138: Gitignore updated correctly for TurborepoThe addition of
.turboto .gitignore is correct and aligns with Turborepo's requirements.tests/simple/baseplate/project-definition.json (1)
24-24: Verify schema version compatibilityThe schema version has been bumped to 9 and the feature reference field has been renamed. Please ensure all dependent packages support these changes.
Let's check for potential compatibility issues:
Also applies to: 79-79
✅ Verification successful
Schema changes are safely handled by migration system
The schema version bump and field renaming are automatically handled by the project's migration system. The codebase includes proper validation and auto-migration capabilities, ensuring compatibility across different versions.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check for schema version references and potential incompatibilities # Search for schema version references echo "Checking for schema version references..." rg -l "schemaVersion.*[89]" --type json # Search for old 'feature' field usage echo "Checking for old feature field usage..." rg -l '"feature":' --type jsonLength of output: 362
Script:
#!/bin/bash # Check the context of 'feature' field usage in preview-app.json echo "Content of preview-app.json:" cat packages/project-builder-web/src/services/preview-app.json # Also check if there are any direct dependencies on project-definition.json schema echo -e "\nChecking for project-definition.json references:" rg -l "project-definition.json" --type ts --type jsLength of output: 41048
Script:
#!/bin/bash # Check how project-definition.json schema is handled echo "Checking schema handling in referenced files:" rg -A 5 -B 5 "project-definition.json" --type ts --type js # Also check for any schema version validation echo -e "\nChecking for schema version validation:" rg -A 3 -B 3 "schemaVersion" --type ts --type jsLength of output: 13276
plugins/baseplate-plugin-storage/package.json (1)
47-48: Verify workspace dependency configurationThe addition of workspace dependencies looks correct, but let's verify the configuration:
Run this script to check workspace package availability:
✅ Verification successful
Workspace dependencies are correctly configured
Both
@halfdomelabs/project-builder-liband@halfdomelabs/syncpackages exist in the monorepo and are properly configured for workspace usage.🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify workspace package configuration # Check if the required workspace packages exist echo "Checking for required workspace packages..." find . -name "package.json" -not -path "*/node_modules/*" -exec jq -r 'select(.name | contains("@halfdomelabs/project-builder-lib") or contains("@halfdomelabs/sync"))' {} \; # Verify package.json workspace configuration echo "Checking workspace configuration..." jq -r '.pnpm.peerDependencyRules' package.json 2>/dev/null || echo "No peer dependency rules found"Length of output: 4489
| - name: Set environment variable for Turborepo | ||
| run: echo "TURBO_SCM_BASE=$NX_BASE" >> $GITHUB_ENV | ||
| - name: Cache turbo build setup | ||
| uses: rharkor/caching-for-turbo@7220c7f366bf362025ec742bd5bcf2bac6e1770e |
There was a problem hiding this comment.
💡 Codebase verification
Remove unnecessary NX dependency
The workflows are using nrwl/nx-set-shas action solely to derive base SHAs for Turborepo, but Turborepo has built-in support for this through environment variables like TURBO_REMOTE_ONLY=true and direct Git references. Consider removing the NX dependency as it's not needed for Turborepo operation.
.github/workflows/test-e2e.yml: Removenrwl/nx-set-shasaction and setTURBO_SCM_BASEdirectly.github/workflows/test.yml: Same change needed.github/workflows/lint.yml: Same change needed
🔗 Analysis chain
Review NX dependency and caching setup
The workflow still uses nrwl/nx-set-shas action while setting up Turborepo environment. Consider these points:
- The NX SHA setup might be unnecessary after migration
- The caching action uses a specific commit hash for security
Let's verify the NX dependencies and caching action:
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for remaining NX dependencies and verify caching action
# Check for remaining NX dependencies
echo "Checking for remaining NX dependencies..."
rg -l 'nx|nrwl' .github/workflows/
# Verify the caching action's commit
echo "Verifying caching action commit..."
gh api repos/rharkor/caching-for-turbo/commits/7220c7f366bf362025ec742bd5bcf2bac6e1770e
Length of output: 11326
Script:
#!/bin/bash
# Check actual NX usage in workflow files
echo "Checking NX usage in workflows..."
rg -A 5 -B 5 'nx|nrwl' .github/workflows/
# Check NX_BASE usage
echo "Checking NX_BASE usage..."
rg 'NX_BASE' .github/workflows/
Length of output: 3132
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
package.json (2)
13-17: Consider adding a prettier:write:affected script for consistency.The lint and formatting scripts have been correctly migrated. For consistency with other affected-only variants, consider adding a
prettier:write:affectedscript."prettier:check": "turbo run prettier:check", "prettier:check:affected": "turbo run prettier:check --affected", "prettier:write": "turbo run prettier:write", + "prettier:write:affected": "turbo run prettier:write --affected",
19-19: Consider enhancing the watch script with package filtering.The development scripts have been correctly migrated. Consider enhancing the watch script to support package-specific watching.
- "watch": "turbo run watch" + "watch": "turbo run watch", + "watch:pkg": "turbo run watch --filter="Also applies to: 24-24, 27-27
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Lite
📒 Files selected for processing (1)
package.json(1 hunks)
🔇 Additional comments (3)
package.json (3)
20-23: LGTM! Verify test configurations across packages.The test scripts have been correctly migrated to use Turborepo's syntax, including package-specific test commands.
Let's verify the test configurations across packages:
✅ Verification successful
✅ Test configurations are properly set up across all packages
- Root
turbo.jsonprovides the base test configuration with proper build dependencies- Package-specific configurations correctly extend or override the base setup
- Project-builder-test package properly configures e2e tests with necessary dependencies and environment variables
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify test configurations in all package turbo.json files # Expect: Test tasks should be properly configured in affected packages # Find all turbo.json files and check their test configurations fd turbo.json$ --exec sh -c 'echo "=== {} ==="; jq ".pipeline | select(.test != null or .\"test:e2e\" != null)" "{}"'Length of output: 272
Script:
#!/bin/bash # Description: Check complete pipeline configurations in turbo.json files echo "=== Checking complete pipeline configurations ===" fd turbo.json$ --exec sh -c 'echo "\n=== {} ==="; cat "{}" | jq .'Length of output: 2489
35-35: Verify Turborepo version compatibility.Please verify that Turborepo v2.3.3 is the latest stable version compatible with Node.js v20.
✅ Verification successful
Turborepo version 2.3.3 is compatible with Node.js v20
You're using the latest version of Turborepo, and it works well with the Node.js v20 requirement in your project. No changes needed.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check latest Turborepo version and its Node.js compatibility # Expect: Version should be latest stable and compatible with Node.js v20 # Check latest version on npm echo "Latest version on npm:" npm view turbo version # Check if there are any known issues with Node.js v20 echo -e "\nIssues mentioning Node.js v20 compatibility:" gh search issues "repo:vercel/turbo node 20" --json title,url,state --jq '.[] | select(.state=="OPEN") | {title,url,state}'Length of output: 520
Script:
#!/bin/bash # Description: Check Turborepo's Node.js compatibility requirements # Search for Node.js version mentions in documentation and package files # Search for Node.js version requirements in markdown files echo "Searching documentation for Node.js requirements:" rg -i "node.*(?:version|require|support|compat)" --type md -A 2 -B 2 # Check if there's any engine requirements in package.json echo -e "\nChecking engine requirements in package.json:" fd package.json --type f --exec cat {} \; | rg -i "\"engines\"|\"node\""Length of output: 3867
8-10: LGTM! Verify the build pipeline configuration.The build scripts have been correctly migrated to use Turborepo's syntax.
Let's verify the build pipeline configuration:
✅ Verification successful
✅ Build pipeline is properly configured
The turbo.json configuration shows:
- Correct task dependencies with
^build- Appropriate caching for build artifacts
- Well-defined inputs and outputs
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the build pipeline configuration in turbo.json # Expect: Build dependencies and cache configuration should be present # Check if turbo.json exists and contains build configuration if [ -f "turbo.json" ]; then jq '.pipeline.build' turbo.json else echo "turbo.json not found at repository root" fiLength of output: 60
Script:
#!/bin/bash # Check the complete turbo.json structure echo "=== Turbo Configuration ===" cat turbo.json echo -e "\n=== Workspace Packages ===" # Find package.json files in workspaces and check for build scripts fd package.json --type f --exclude node_modules --exec sh -c 'echo "=== {} ==="; jq -c "select(.scripts.build != null) | {name: .name, buildScript: .scripts.build}" {}'Length of output: 3245
Summary by CodeRabbit
Tooling Migration
Package Updates
Configuration Changes
.gitignoreto exclude.turbodirectory.turbo.jsonconfiguration files in various packages.