Build: Update Gutenberg => Core integration

[youknowriad]

Trac ticket: https://core.trac.wordpress.org/ticket/64393

Summary

This PR changes WordPress Core's Gutenberg integration from npm packages to a checkout-and-build approach. Instead of syncing individual npm packages, Core now checks out the Gutenberg repository, builds it, and copies the build artifacts. This enables Core to use Gutenberg's advanced features like route-based navigation, full-page rendering, and the new Font Library. But also offers us the possibility to streamline more backports later (an example in this PR is the automatic copy of theme.json file). Check the issue for more details on the problem at hand.

New Build Pipeline

1. Checkout: npm run gutenberg:checkout clones Gutenberg at a specified ref
2. Build: npm run gutenberg:build runs Gutenberg's build process
3. Copy: npm run gutenberg:copy copies and transforms build output to Core
4. Integrate: npm run gutenberg:integrate runs all three steps

What Gets Copied

From Gutenberg build to Core:

• /build/routes/ → /src/wp-includes/build/routes/
• /build/pages/ → /src/wp-includes/build/pages/
• /build/modules/ → /src/wp-includes/js/dist/script-modules/
• /build/scripts/ → /src/wp-includes/js/dist/
• /build/styles/ → /src/wp-includes/css/dist/
• /build/blocks/ → /src/wp-includes/blocks/

Path Transformations

The copy script transforms Gutenberg plugin paths to Core paths:

• plugins_url() → includes_url()
• plugin_dir_path() → ABSPATH . WPINC . '/build/'
• Boot module paths adjusted for Core's directory structure
• Enqueue conditions work with both menu pages and direct file access

Ideally these transformations shouldn't be needed, maybe we can make our "build" tool more generic to streamline the integration better and allow "building for core" directly. I'll look into that separately.

Webpack Changes

Removed webpack configs that are now replaced by Gutenberg's build:

• ❌ tools/webpack/blocks.js - Using Gutenberg's block builds
• ❌ tools/webpack/packages.js - Using Gutenberg's script builds
• ❌ tools/webpack/script-modules.js - Using Gutenberg's module builds
• ❌ tools/webpack/development.js - Using Gutenberg's dev builds
• ❌ tools/webpack/vendors.js - Using Gutenberg's vendors
• ✅ tools/webpack/media.js - Kept (Core-specific files). to be honest, I'm not really sure if Webpack is needed for these files but I kept it unchanged for now, we could decide to remove the webpack dependency later entirely.

Font Library Integration

Added /wp-admin/font-library.php as the first proof-of-concept using this new architecture.

Some other changes.

src/wp-includes/script-modules.php

• Removed outdated debug version handling for @wordpress/interactivity (this has been removed from Gutenberg)

Gruntfile.js

• Added a vendor copy step that was previously done in webpack/packages.js

package.json

• The "hash" or Gutenberg version used is defined here. Right now I picked a recent Gutenberg commit but this is likely to refer commits in wp/* release branches in Gutenberg.

Testing

• There should be no big disruptions in the core development workflow. I tried to keep the flow the same, the output files in the same places...
• The initial build might be slow because of typescript building, the next ones should be ok.

---

I think there are a lot of simplifications we could do later (specially to the copy script) by aligning the folder structures more closely together (the build folder of Gutenberg, and the file structure for built scripts, styles, modules, pages... on Core). It does require changing some paths on Core. I do think it would be for the better but I kept it out of this PR.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @1178653+wordpress-develop-pr-bot[bot]@users.noreply.github.com.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

Core Committers: Use this line as a base for the props when committing in SVN:

Props youknowriad, jonsurrell, ellatrix, westonruter, desrosj, isabel_brison.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[youknowriad]

Looks like there's an infinite loop because wp-build use time based hashes which triggers the "commit unsaved changes" on each commit. I'm going to disable that workflow temporarily.

[ellatrix]

This is exciting! Also paves the way to run Gutenberg e2e tests :)

[github-actions]

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

• The Plugin and Theme Directories cannot be accessed within Playground.
• All changes will be lost when closing a tab with a Playground instance.
• All changes will be lost when refreshing the page.
• A fresh instance is created each time the link below is clicked.
• Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
  it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

[youknowriad]

Ok I think this is ready to land personally. I'd love reviews though :)

[ellatrix]

Not sure if we need to manually add this to svnignore too

[ellatrix]

What's this for?

[youknowriad]

Typescript building on MacOS consumes more memory than the default allocated memory to node processes. I recall we had to do something similar elsewhere. (Without it the build fails on MacOS on CI)

[ellatrix]

This is no longer an old file?

[youknowriad]

We do have an undo-manager script, so no, it's no longer an old file. That said, I didn't understand why it was here before, I think that script existed in Gutenberg for some time.

[ellatrix]

Works great and makes a lot of sense. Let's try it. 🙂

[desrosj]

I've started to take a look at this, but it's a pretty big PR and I need a bit more time.

Is this a blocker for anything specific? Or is there room for a few more days to review? I'd love to do finish a proper review and testing.

[youknowriad]

Can wait for a few more days. Maybe we should aim to land it early next week.

[tellthemachines]

I gave this a test run on my local env and it's working really well so far! I did have to point the env to the build directory (the default in the .env file is src) and then of course npm run build.

Without having looked extensively at the code yet, I love the direction! It will save us so much manual work in porting code from Gutenberg to Core ❤️

[youknowriad]

I thought npm run build was meant to build the "build" folder and npm run dev was for the "src" folder. Now, it's possible that this assumption is wrong and I'm happy to correct my scripts if needed. It's really unclear to me what script is supposed to target which output folder.

[tellthemachines]

I thought npm run build was meant to build the "build" folder and npm run dev was for the "src" folder

Yes that's correct! The WORKING_DIR is set to src whenever the dev flag is passed. So:

• npm run dev watches the src folder
• npm run build:dev builds the src folder
• npm run watch watches the build folder
• npm run build builds the build folder

With this PR, I wasn't able to get my env running or build it from the src directory, it only worked from the build one. I thought I'd mention that in the comment so other folks who test it don't run into the same issue.

[youknowriad]

I just tried and confirmed that both npm run dev and npm run build:dev work for me with the default "src" environment.

[youknowriad]

@desrosj Hi Jonathan, did you have time to look at the PR. I'd love to avoid letting it sit for too long. Regardless, I'll be available for any follow-ups...

[desrosj]

Unfortunately, I did not get to look much further before heading out on my holiday break.

One suggestion I was going to make was around the new files that are replacing old ones that did the same thing. For example, managing packages. I was going to suggest that those are added to version control using an svn copy, and then all of the changes be applied.

While this is a new approach, this maintains the history of those files so that a visual comparison is available instead of just a sea of red and green in separate files.

[youknowriad]

One suggestion I was going to make was around the new files that are replacing old ones that did the same thing. For example, managing packages. I was going to suggest that those are added to version control using an svn copy, and then all of the changes be applied.

While this is a new approach, this maintains the history of those files so that a visual comparison is available instead of just a sea of red and green in separate files.

Thanks for quickly chiming in, I guess what you're saying is that we shouldn't git ignore the files that come from Gutenberg.

I found that the previous approach of gitignoring some files (JS, CSS) and no others (PHP, JSON) was not very consistent and also created a lot of confusion of where the source of truth of these files is. I think with the changes in this PR, it's a lot more clear. These are external dependencies maintained in Gutenberg.

[desrosj]

Sorry, I'm not at my computer currently, so that was poorly worded without an example. That comment was specifically targeted at any files within the tools directory and build configuration related files only.

[youknowriad]

Ah I see :) Thanks for clarifying. It makes more sense now. I don't really think there's a 1 to 1 relationship with what was being done before though, so it's a bit hard to think of this as "file renames", it's a completely different pipeline.