Merge develop into master

.github/workflows/examples.yaml

@@ -2,9 +2,9 @@ name: Examples
 
 on:
   pull_request:
-    paths: [examples/**, scripts/**]
+    paths: [examples/**, scripts/**, .github/workflows/examples.yaml]
   push:
-    paths: [examples/**, scripts/**]
+    paths: [examples/**, scripts/**, .github/workflows/examples.yaml]
 
 jobs:
   examples:
@@ -14,6 +14,12 @@ jobs:
         with:
           submodules: recursive
 
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          toolchain: '1.94.0'
+          targets: riscv64imac-unknown-none-elf
+          components: rustfmt,clippy
+
       - uses: actions/cache@v4
         with:
           path: |

README.md

@@ -34,6 +34,23 @@ If you run into an issue on our documentation website you can contact us on [Ner
 - `website`: The doc site is built with [docusaurus](https://docusaurus.io/) and under the `website` folder.
 - `examples`: The `examples` folder contains full tutorial codes you can clone.
 
+### Release
+
+Production release should be on the master branch. 
+
+Please follow the steps below:
+
+1. Create a new PR that bumps the version in the `package.json` under `/website`
+2. Merge the PR on `develop` branch
+3. Create a new PR from `develop` that targeting on the `master` branch
+4. Merge the PR into `master` branch
+5. Create a new tag and release on github targeting on the master branch
+6. The release content should include description of changes following 3 sections:
+   - New Content
+   - Fixes
+   - Others
+7. Example release content: https://github.com/nervosnetwork/docs.nervos.org/releases/tag/v2.35.0 
+
 ### Develop
 
 #### Clone the Repo
@@ -89,18 +106,72 @@ yarn gen-terms
 After running the command, you can navigate to `src/components/Tooltip` to verify that the `key-terms.json` file has been generated successfully.
 
 ###  Broken Link Checker
-Use `./scripts/check-urls.sh` to scan for dead links.
 
-**How to Use**
+The link checker scans for dead links and maintains a collaborative dead link registry.
+
+**Quick Start**
 
 1. Install [lychee](https://github.com/lycheeverse/lychee).
 2. Build the website: `cd website && yarn build`.
-3. Run the script: `./scripts/check-urls.sh`.
+3. Run the check: `cd website && yarn link:check`.
+4. Review results: `cd website && yarn link:report`.
+
+**Environment Variables**
+
+Customize the check behavior with these optional environment variables:
+
+- `GITHUB_TOKEN`: GitHub API token to increase rate limits
+- `LINK_CHECK_CONCURRENCY`: Number of concurrent requests (default: 1)
+- `LINK_CHECK_TIMEOUT`: Request timeout in seconds (default: 30)
+- `LINK_CHECK_RETRIES`: Number of retries per URL (default: 2)
+- `LINK_CHECK_ACCEPT`: Acceptable HTTP status codes (default: "200..=299,403")
+
+Example: `GITHUB_TOKEN=your_token LINK_CHECK_CONCURRENCY=2 yarn link:check`
+
+**Report Files**
+
+All reports are stored in `website/reports/link-check/`:
+
+- `summary.json`: Run statistics and metrics
+- `dead-links.json`: All currently failing links with metadata
+- `new-failures.json`: Links that failed but aren't in the baseline
+- `recovered.json`: Previously failing links that now work
+- `unresolved-known.json`: Known issues that still fail
+- `known-failures.json`: Baseline of accepted failures (manual maintenance)
+- `history.jsonl`: Historical run data
+- `report.md`: Human-readable summary report
+
+**Workflow**
+
+1. **Run Check**: Execute `yarn link:check` to scan all links
+2. **Review Report**: Check `report.md` for new failures and recoveries
+3. **Take Action**:
+   - Fix broken links in documentation
+   - For unfixable external links, add to `known-failures.json` with reason
+   - Remove recovered links from `known-failures.json`
+4. **Commit Changes**: Include report updates and documentation fixes
+
+**Baseline Management**
+
+The `known-failures.json` file contains links that are expected to fail. Each entry should have:
+
+```json
+{
+  "url": "https://example.com/broken",
+  "expectedStatus": 404,
+  "reason": "External service discontinued",
+  "owner": "team-member",
+  "addedAt": "2024-01-01T00:00:00.000Z"
+}
+```
 
-The script prints any links that fail the check.
+**Only add links to baseline if:**
+- The link points to an external service you don't control
+- The failure is permanent (not temporary network issues)
+- You've confirmed the link is truly broken
 
 **Notes**
 
 * A cache file `./website/.lycheecache` will be created. For local development, the cache TTL is 30 days; you can change `max_cache_age` in `website/.lychee.toml`.
 * Because the docs contain many GitHub links, requests are routed via `api.github.com`. Providing a GitHub token increases the rate limit.
-* For this reason, the check is not integrated into CI at the moment.
+* The check is designed for local/manual execution; CI integration is not currently implemented.

scripts/check-urls.sh

@@ -1,19 +1,91 @@
 #!/bin/bash
 
-cd website
-# yarn build
-
-# Used by Lychee to check if URLs are valid.
-# Because there are a large number of GitHub URLs, a direct check would fail (returning a 429 error due to too many requests in a short period of time).
-# This function converts GitHub URLs to api.github.com. GitHub's API can support more requests.
-if [ ! -f build/github-api-urls.txt ]; then
-  grep -rho "https://github.com[^\"'<> ]*" build | sort -u > build/github-urls.txt
-  node ../scripts/convert-github-urls.js build/github-urls.txt > build/github-api-urls.txt
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+WEBSITE_DIR="$REPO_ROOT/website"
+REPORTS_DIR="reports/link-check"
+
+# Configuration via environment variables
+LINK_CHECK_CONCURRENCY=${LINK_CHECK_CONCURRENCY:-1}
+LINK_CHECK_TIMEOUT=${LINK_CHECK_TIMEOUT:-30}
+LINK_CHECK_RETRIES=${LINK_CHECK_RETRIES:-2}
+LINK_CHECK_ACCEPT=${LINK_CHECK_ACCEPT:-"200..=299,403"}
+GITHUB_TOKEN=${GITHUB_TOKEN:-""}
+
+cd "$WEBSITE_DIR"
+
+# Pre-flight checks
+echo "🔍 Pre-flight checks..."
+if ! command -v lychee &> /dev/null; then
+    echo "❌ Error: lychee is not installed. Please install it first: https://github.com/lycheeverse/lychee"
+    exit 1
+fi
+
+if ! command -v node &> /dev/null; then
+    echo "❌ Error: node is not installed."
+    exit 1
+fi
+
+if [ ! -d "build" ]; then
+    echo "❌ Error: build directory not found. Please run 'yarn build' first."
+    exit 1
+fi
+
+echo "✅ Pre-flight checks passed"
+
+# Always regenerate URL lists to ensure freshness
+echo "🔄 Extracting GitHub URLs from build directory..."
+grep -rho "https://github.com[^\"'<> ]*" build | sort -u > build/github-urls.txt
+echo "📝 Found $(wc -l < build/github-urls.txt) unique GitHub URLs"
+
+echo "🔄 Converting GitHub URLs to API endpoints..."
+node ../scripts/convert-github-urls.js build/github-urls.txt > build/github-api-urls.txt
+echo "📝 Generated $(wc -l < build/github-api-urls.txt) API URLs"
+
+# Prepare lychee command with optional token
+lychee_args=(
+    --config .lychee.toml
+    --cache-exclude-status "400..=699"
+    --format json
+    --output "$REPORTS_DIR/dead-links-raw.json"
+    --max-concurrency "$LINK_CHECK_CONCURRENCY"
+    --timeout "$LINK_CHECK_TIMEOUT"
+    --max-retries "$LINK_CHECK_RETRIES"
+    --accept "$LINK_CHECK_ACCEPT"
+)
+
+if [ -n "$GITHUB_TOKEN" ]; then
+    lychee_args+=(--header "Authorization: Bearer $GITHUB_TOKEN")
 fi
 
-# lychee current version: 0.20.1
-lychee \
-    --config .lychee.toml \
-    --cache-exclude-status="400..=699" \
-    build 2>&1 \
-  | grep --color=never -v 'InvalidPathToUri'
+echo "🚀 Running link check with concurrency=$LINK_CHECK_CONCURRENCY, timeout=${LINK_CHECK_TIMEOUT}s, retries=$LINK_CHECK_RETRIES..."
+DISPLAY_CMD="lychee --config .lychee.toml --cache-exclude-status \"400..=699\" --format json --output $REPORTS_DIR/dead-links-raw.json --max-concurrency $LINK_CHECK_CONCURRENCY --timeout $LINK_CHECK_TIMEOUT --max-retries $LINK_CHECK_RETRIES --accept \"$LINK_CHECK_ACCEPT\""
+if [ -n "$GITHUB_TOKEN" ]; then
+    DISPLAY_CMD="$DISPLAY_CMD --header \"Authorization: Bearer ***REDACTED***\""
+fi
+echo "Command: $DISPLAY_CMD build"
+
+# Run lychee and capture both output and exit code
+set +e
+lychee "${lychee_args[@]}" build 2>&1 | grep --color=never -v 'InvalidPathToUri'
+LYCHEE_EXIT_CODE=${PIPESTATUS[0]}
+set -e
+
+echo "📊 Processing results..."
+
+# Process the raw JSON output into our structured format
+node "$REPO_ROOT/scripts/process-link-results.js"
+
+echo "✅ Link check completed. Results saved to website/reports/link-check/"
+FAILED_TOTAL=$(node -e "const fs=require('fs');const s=JSON.parse(fs.readFileSync('reports/link-check/summary.json','utf8'));process.stdout.write(String(s.failedTotal ?? 0));")
+IGNORED_TOTAL=$(node -e "const fs=require('fs');const s=JSON.parse(fs.readFileSync('reports/link-check/summary.json','utf8'));process.stdout.write(String(s.ignoredTransientTotal ?? 0));")
+echo "📋 Summary: $FAILED_TOTAL failed links found"
+if [ "$IGNORED_TOTAL" -gt 0 ]; then
+    echo "ℹ️ Ignored transient errors (e.g. 429): $IGNORED_TOTAL"
+fi
+
+if [ "$LYCHEE_EXIT_CODE" -ne 0 ]; then
+    echo "⚠️ lychee exited with code $LYCHEE_EXIT_CODE. Processed report artifacts were still generated."
+fi

scripts/compare-link-report.js

@@ -0,0 +1,90 @@
+#!/usr/bin/env node
+
+const fs = require('fs');
+const path = require('path');
+
+const REPORTS_DIR = 'reports/link-check';
+const DEAD_LINKS_FILE = path.join(REPORTS_DIR, 'dead-links.json');
+const KNOWN_FAILURES_FILE = path.join(REPORTS_DIR, 'known-failures.json');
+const NEW_FAILURES_FILE = path.join(REPORTS_DIR, 'new-failures.json');
+const RECOVERED_FILE = path.join(REPORTS_DIR, 'recovered.json');
+const UNRESOLVED_KNOWN_FILE = path.join(REPORTS_DIR, 'unresolved-known.json');
+const REPORT_MD_FILE = path.join(REPORTS_DIR, 'report.md');
+
+// Read current dead links
+let deadLinks = [];
+if (fs.existsSync(DEAD_LINKS_FILE)) {
+  try {
+    deadLinks = JSON.parse(fs.readFileSync(DEAD_LINKS_FILE, 'utf-8'));
+  } catch (error) {
+    console.error('Error reading dead links:', error.message);
+    process.exit(1);
+  }
+}
+
+// Read known failures baseline
+let knownFailures = [];
+if (fs.existsSync(KNOWN_FAILURES_FILE)) {
+  try {
+    knownFailures = JSON.parse(fs.readFileSync(KNOWN_FAILURES_FILE, 'utf-8'));
+  } catch (error) {
+    console.error('Error reading known failures:', error.message);
+    process.exit(1);
+  }
+}
+
+// Create lookup sets
+const knownUrls = new Set(knownFailures.map(kf => kf.url));
+const currentFailingUrls = new Set(deadLinks.map(link => link.url));
+
+// Calculate categories
+const newFailures = deadLinks.filter(link => !knownUrls.has(link.url));
+const recovered = knownFailures.filter(kf => !currentFailingUrls.has(kf.url));
+const unresolvedKnown = knownFailures.filter(kf => currentFailingUrls.has(kf.url));
+
+// Write categorized results
+fs.writeFileSync(NEW_FAILURES_FILE, JSON.stringify(newFailures, null, 2));
+fs.writeFileSync(RECOVERED_FILE, JSON.stringify(recovered, null, 2));
+fs.writeFileSync(UNRESOLVED_KNOWN_FILE, JSON.stringify(unresolvedKnown, null, 2));
+
+// Generate markdown report
+const reportContent = `# Link Check Report
+
+Generated at: ${new Date().toISOString()}
+
+## Summary
+- **New Failures**: ${newFailures.length}
+- **Recovered**: ${recovered.length}
+- **Unresolved Known Issues**: ${unresolvedKnown.length}
+- **Total Current Failures**: ${deadLinks.length}
+
+## New Failures
+${newFailures.length === 0 ? 'None! 🎉' : newFailures.map(link => `- ${link.url} (${link.status})`).join('\n')}
+
+## Recovered Links
+${recovered.length === 0 ? 'None' : recovered.map(link => `- ${link.url}`).join('\n')}
+
+## Unresolved Known Issues
+${unresolvedKnown.length === 0 ? 'None' : unresolvedKnown.map(link => `- ${link.url} (${link.reason || 'No reason specified'})`).join('\n')}
+
+## Actions Required
+${newFailures.length > 0 ? `⚠️ **${newFailures.length} new dead links found!** Please review and either fix the links or add them to known-failures.json with appropriate reasons.` : '✅ No new failures detected.'}
+
+${recovered.length > 0 ? `ℹ️ **${recovered.length} links have recovered.** Consider removing them from known-failures.json.` : ''}
+`;
+
+fs.writeFileSync(REPORT_MD_FILE, reportContent);
+
+console.log('📋 Comparison complete. Results:');
+console.log(`   - New failures: ${newFailures.length}`);
+console.log(`   - Recovered: ${recovered.length}`);
+console.log(`   - Unresolved known: ${unresolvedKnown.length}`);
+console.log(`   - Report saved to: ${REPORT_MD_FILE}`);
+
+// Exit with error if there are new failures (configurable behavior)
+if (newFailures.length > 0) {
+  console.log('❌ New dead links detected. Review the report above.');
+  process.exit(1);
+} else {
+  console.log('✅ No new dead links found.');
+}