feat(parser,estree,coverage): collect tokens in parser and convert to ESTree format by camc314 · Pull Request #19497 · oxc-project/oxc

camc314 · 2026-02-17T13:53:07Z

Implement a collect_tokens option for parser. When enabled, the parser stores all tokens produced during parsing in a Vec. Implement serializing the tokens to ESTree format.

Currently this is not hooked up to NAPI parser or linter. That happens in a later PR in this stack (#19498).

Serialization of tokens implementation is in new oxc_estree_tokens crate. Unfortunately it can't go in existing oxc_estree crate because oxc_ast depends on oxc_estree, and the tokens serialization code requires AST types - which would be a circular dependency. We may be able to move it into another crate (parser?) later on.

camc314 · 2026-02-17T13:53:21Z

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

0-merge - adds this PR to the back of the merge queue
hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

codspeed-hq · 2026-02-17T13:59:37Z

Merging this PR will degrade performance by 15.38%

❌ 8 regressed benchmarks
✅ 39 untouched benchmarks
⏩ 3 skipped benchmarks¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`parser[RadixUIAdoptionSection.jsx]`	81.3 µs	87.8 µs	-7.37%
❌	Simulation	`parser[binder.ts]`	3.3 ms	3.5 ms	-7.03%
❌	Simulation	`parser[react.development.js]`	1.3 ms	1.4 ms	-6.94%
❌	Simulation	`parser[cal.com.tsx]`	26.1 ms	28.1 ms	-7.34%
❌	Simulation	`lexer[react.development.js]`	358.3 µs	410.7 µs	-12.76%
❌	Simulation	`lexer[RadixUIAdoptionSection.jsx]`	21.1 µs	23.9 µs	-11.81%
❌	Simulation	`lexer[cal.com.tsx]`	5.5 ms	6.5 ms	-15.38%
❌	Simulation	`lexer[binder.ts]`	885.3 µs	1,013.4 µs	-12.64%

_{Comparing c/02-17-feat_parser_estree_coverage_add_parser_tokens_and_shared_estree_conversion (7f91f57) with main (a4d5b34)}

3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

overlookmotel

I know this is in a rough state but I couldn't resist having a look.

What would be really useful is if you could ask Codex to add some comments to the code explaining why things are as they are.

The biggest mystery to me is why the extra AST visitation. I'm unclear if this is purely to work around the few edge case differences between ESLint and TS-ESLint (escaped identifiers etc), or whether it'd still be required if we ignored those differences.

There are other parts which have me puzzled too (e.g. comments below).

Some of this code is wildly inefficient, but we can fix that in follow-ups. We'd also need to find a way to avoid the perf penalty on compiler pipeline (not trivial, see #16785 for one attempt but even that had a negative perf impact) - again, could be a follow-up PR (and we don't merge any of stack until we have a solution).

Passing all the tests is a huge feat in itself.

crates/oxc_parser/src/lexer/mod.rs

crates/oxc_parser/src/lexer/punctuation.rs

crates/oxc_estree_tokens/src/lib.rs

crates/oxc_parser/src/lexer/mod.rs

crates/oxc_estree_tokens/src/lib.rs

### 🚀 Features - 733d6dc parser: Report error on `infer` outside conditional type (#19879) (camc314) - c2a42f6 allocator: Add `Vec::into_bump_slice_mut` (#19895) (overlookmotel) - ee4982b parser: Add `VARIANTS` const to `Kind` via `fieldless_enum!` macro (#19877) (overlookmotel) - b3dceae data_structures: Add `fieldless_enum!` macro (#19876) (overlookmotel) - 12b841e parser: Make all `Kind::is_*` methods `const` (#19874) (overlookmotel) - 25c2e25 estree/tokens: Add function to update tokens in place (#19856) (overlookmotel) - f78e6df parser: Add `mutate_tokens` Cargo feature (#19853) (overlookmotel) - 5036bb6 parser: Report error on `for await` in static blocks (#19844) (camc314) - 42bd431 parser: Report error for missing initializer in using decl (#19824) (camc314) - a2f58e5 parser: Report error for `implements` clause in non-ts files (#19820) (Cameron) - b25228a estree: Add `IS_COMPACT` const to `Formatter` trait (#19787) (overlookmotel) - e2a1b79 estree: Expose buffer and formatter of serializers (#19773) (overlookmotel) - 4699498 data_structures: Add `CodeBuffer::print_strs_array` (#19760) (overlookmotel) - 233f947 estree: `oxc_estree` crate export config and formatter types (#19724) (overlookmotel) - 5937a32 semantic: Introduce `symbol_declarations` method (#19609) (camc314) - ea6b796 parser: Add `LexerConfig::TOKENS_METHOD_IS_STATIC` const (#19683) (overlookmotel) - 655c38f semantic: Add "did you mean?" suggestions to undefined name errors (#19102) (copilot-swe-agent) - 9e11dc6 parser,estree,coverage: Collect tokens in parser and convert to ESTree format (#19497) (camc314) - c4a3677 parser: Report error for initializer in ambient context (#19187) (camc314) ### 🐛 Bug Fixes - abc7e19 codegen: Improve parenthesised checks when printing types (#19880) (camc314) - 017de5d parser: Update error code for type annotation in `for...in` statement (#19882) (camc314) - 7682e5a linter/plugins: Decode escapes in identifier tokens (#19838) (overlookmotel) - 06767ed estree/tokens: Convert `this` tokens in `TSTypeName` (#19815) (overlookmotel) - ef798af parser: Use TS8037 for satisfies expression in JS files diagnostic (#19819) (camc314) - 98ea5c5 parser: Use TS8016 for type assertions in JS files diagnostic (#19818) (camc314) - 1710f56 codegen: Remove double indentation for enum inside namespace (#19775) (Dunqing) - 9e4995c codegen: Print type annotation on `CatchParameter` (#19790) (camc314) - 297b2bb codegen: Wrap `TSConditionalType` in parens when necessary (#19788) (camc314) - cec7878 codegen: Print `definite` property on AccessorProperty (#19786) (camc314) - 6f395cf codegen: Print `definite` property on PropertyDefinition (#19785) (camc314) - b749373 codegen: Correctly parenthesise TSArrayType (#19784) (camc314) - 876dc1b codegen: Print object property `this` param (#19783) (camc314) - 93bb861 formatter: Trim trailing whitespace before breaking line (#19740) (leaysgur) - ed17bbf codegen: Print `override` keyword for method and property definitions (#19753) (Dunqing) - 6a59a76 parser: Improve error recovery for private identifiers in property names (#19710) (Boshen) - 3b96f41 codegen: Print comments in JSX expression containers and spread attributes (#19701) (Boshen) - f5694ce estree/tokens: Reverse field order of `regex` object in tokens (#19679) (overlookmotel) - b2b7a55 estree/tokens: Generate tokens for files with BOM (#19535) (overlookmotel) - 50a7514 estree: Fix tokens for JSX (#19524) (overlookmotel) - a35063e minifier: Preserve side effects for meta property url reads (#19668) (Boshen) - 8ad3430 semantic/jsdoc: Handle even-numbered backtick sequences in JSDoc parsing (#19664) (Boshen) ### ⚡ Performance - 05ccf9f linter/plugins: Transfer tokens via raw transfer (#19893) (overlookmotel) - c1bfdcf estree/tokens: Preallocate sufficient space for tokens JSON (#19851) (overlookmotel) - 4b0611a estree/tokens: Introduce `ESTreeTokenConfig` trait (#19842) (overlookmotel) - 81bab90 estree/tokens: Do not JSON-encode keyword, punctuator, etc tokens (#19814) (overlookmotel) - 6260ddd estree/tokens: Do not JSON-encode `this` identifiers (#19813) (overlookmotel) - b378f4a estree/tokens: Do not JSON-encode JSX identifiers (#19812) (overlookmotel) - 5016d92 estree/tokens: Handle regex tokens separately (#19796) (overlookmotel) - 780a68e estree/tokens: Use strings from AST for identifier tokens (#19744) (overlookmotel) - dc9c2e3 estree: Use `CodeBuffer::print_strs_array` to reduce bounds checks (#19766) (overlookmotel) - 845da35 estree: Use `CodeBuffer::print_indent` (#19727) (overlookmotel) - ec88f6a estree/tokens: Serialize tokens while visiting AST (#19726) (overlookmotel) - bc6507f estree/tokens: Serialize with `ESTree` not `serde` (#19725) (overlookmotel) - ec24859 estree/tokens: Do not branch on presence of override twice (#19721) (overlookmotel) - dac14be estree/tokens: Replace hash map with `Vec` (#19718) (overlookmotel) - b9d2443 estree/tokens: Replace multiple hash sets into a single hash map (#19716) (overlookmotel) - 7233548 parser: Remove branches from `finish_next_inner` (#19695) (overlookmotel) - b5d9845 parser: Remove const generic param from `finish_next_inner` (#19684) (overlookmotel) - 8940f66 estree/tokens: Serialize tokens to compact JSON (#19572) (overlookmotel) - 136e39b parser/tokens: Pre-allocate capacity for tokens (#19543) (overlookmotel) - 6a6513c linter/plugins: Use Oxc tokens in plugins (#19498) (camc314) - b3b2d30 parser: Introduce `ParserConfig` (#19637) (overlookmotel) ### 📚 Documentation - b2b7a64 estree/tokens: Correct comment (#19873) (overlookmotel) - 0399311 estree/tokens: Improve comments (#19836) (overlookmotel) - 1b392de minifier: Add `Function.prototype.toString` assumption (#19758) (sapphi-red) - 75c9cd8 parser: Improve doc comments for `ParserConfig` and `LexerConfig` (#19682) (overlookmotel) - 2fa936f README.md: Map npm package links to npmx.dev (#19666) (Boshen) Co-authored-by: Boshen <1430279+Boshen@users.noreply.github.com>

# Oxlint ### 🚀 Features - 2e0e1d0 linter/no-unused-vars: Add experimental fix mode controls (off|suggestion|fix) (#19774) (camc314) - f34f6fa linter: Introduce typeCheck config option (#19764) (camc314) - 694be7d linter: Introduce typeAware as config options (#19614) (camc314) - 655c38f semantic: Add "did you mean?" suggestions to undefined name errors (#19102) (copilot-swe-agent) - e97a57e linter/id-length: Use serde to deserialize rule options (#19636) (camc314) - c4a3677 parser: Report error for initializer in ambient context (#19187) (camc314) - 346045a linter/id-length: Add `checkGeneric` option (#19634) (camc314) ### 🐛 Bug Fixes - 1b7a937 linter: Correct double-comparisons fix with swapped operands (#19846) (camc314) - c308857 linter/consistent_type_imports: Add missing help and notes to diagnostics (#19827) (Daniel Osmond) - 7682e5a linter/plugins: Decode escapes in identifier tokens (#19838) (overlookmotel) - f368fcd linter/consistent_type_assertions: Add missing with_help and with_note to diagnostics (#19826) (Daniel Osmond) - 04e6223 npm: Add `preferUnplugged` for Yarn PnP compatibility (#19829) (Boshen) - 86d5037 linter: Add help text to no-extend-native, no-useless-backreference (#19733) (Anthony Amaro) - 50e8eff linter: Add .with_help() to operator-assignment, no-nonoctal-decimal-escape (#19732) (Anthony Amaro) - 1417bdc linter/no-wrapper-object-types: Add help messages to missing diagnostics (#19771) (Daniel Osmond) - 0838477 linter/ban_ts_comment: Add help and notes to missing diagnostics (#19781) (Daniel Osmond) - e8c77cf linter/adjacent_overload_signatures: Add missing diagnostics (#19780) (Daniel Osmond) - 28834ac linter/ban_types: Add missing help and note to diagnostics (#19782) (Daniel Osmond) - fd938d3 linter/prefer-enum-initializers: Add help messages to missing diagnostics (#19772) (Daniel Osmond) - eb928ee linter/no-dynamic-delete: Add help messages to missing diagnostics (#19768) (Daniel Osmond) - a985666 linter/no-empty-interface: Add help messages to missing diagnostics (#19769) (Daniel Osmond) - 2dc0ceb linter/no-extra-non-null-assertion: Add help messages to missing diagnostics (#19770) (Daniel Osmond) - 95d5d66 linter/no-dupe-keys: Handle `__proto__` proto setters in (#19762) (camc314) - 24ff0db linter/exhaustive-deps: False positive for member expressions in IIFEs (#19751) (Dennis Chen) - 7243a58 linter/no-use-before-define: Honor `ignoreTypeReferences` when value and type name collisions (#19747) (Dimava) - eefd818 linter/explicit-module-boundary-types: Add help messages to missing diagnostics (#19736) (Daniel Osmond) - 0440e9a linter: Add help text to no_control_regex, no_fallthrough, no_param_reassign (#19655) (Anthony Amaro) - e84cb2f react/display-name: Handle merged type+value context symbols (#19608) (camc314) - ce7e253 linter/prefer-object-from-entries: Require exact path match in unicorn helper (#19687) (camc314) - f5694ce estree/tokens: Reverse field order of `regex` object in tokens (#19679) (overlookmotel) - b2b7a55 estree/tokens: Generate tokens for files with BOM (#19535) (overlookmotel) - 0722721 linter/jsx-curly-brace-presence: False positive with prop & mixed quotes (#19674) (camc314) - 3496acd linter: Enhance diagnostic help messages for eslint rules (#19653) (Anthony Amaro) - e384e94 linter: Enhance help diagnostic messages for more eslint rules (#19658) (Anthony Amaro) - a4d5b34 linter: Avoid non-promise catch false positives (#19574) (camc314) - 5706f38 linter: `unicorn/no-array-callback-reference` skip `Effect.*` array-like methods name. (#19633) (Said Atrahouch) ### ⚡ Performance - 05ccf9f linter/plugins: Transfer tokens via raw transfer (#19893) (overlookmotel) - 4b0611a estree/tokens: Introduce `ESTreeTokenConfig` trait (#19842) (overlookmotel) - ec88f6a estree/tokens: Serialize tokens while visiting AST (#19726) (overlookmotel) - d4dcf26 linter/plugins: Remove `typescript` from bundle (#19531) (overlookmotel) - 6a6513c linter/plugins: Use Oxc tokens in plugins (#19498) (camc314) ### 📚 Documentation - d86f59e linter: Improve docs for no-useless-concat, mark as pending fixer. (#19859) (connorshea) - caa091d linter/plugins: Correct doc comments for `initTokens` (#19530) (overlookmotel) - 2fa936f README.md: Map npm package links to npmx.dev (#19666) (Boshen) - dc0ff73 linter/no-useless-constructor: Warn for parameter properties as well (#19638) (Ole Asteo) # Oxfmt ### 🚀 Features - 5141bc2 formatter: Support trailing ignore comments (#19304) (Andreas Lubbe) - 4888a99 oxfmt/lsp: Support other schemes beside `file://` and `untitled://` (#19872) (Sysix) - 14a0181 oxfmt: Support `graphql()` variant for gql-in-js (#19703) (leaysgur) - ca68ea6 oxfmt: Support gql-in-js substitution (#19670) (leaysgur) - 035933c formatter,oxfmt: Support js-in-vue (partially) (#19514) (leaysgur) - 9e11dc6 parser,estree,coverage: Collect tokens in parser and convert to ESTree format (#19497) (camc314) ### 🐛 Bug Fixes - 8e3842d oxfmt: Avoid embedded TSFN crash by returning errors as data (take2) (#19806) (Yuji Sugiura) - 04e6223 npm: Add `preferUnplugged` for Yarn PnP compatibility (#19829) (Boshen) - e540585 oxfmt: Support tailwind sort for CSS/LESS/SCSS (#19803) (leaysgur) - 93bb861 formatter: Trim trailing whitespace before breaking line (#19740) (leaysgur) - b85f97b formatter: Drop blank line between terminal call and first chain member (#19659) (Dunqing) ### ⚡ Performance - b3b2d30 parser: Introduce `ParserConfig` (#19637) (overlookmotel) ### 📚 Documentation - 2fa936f README.md: Map npm package links to npmx.dev (#19666) (Boshen) Co-authored-by: Boshen <1430279+Boshen@users.noreply.github.com>

camc314 mentioned this pull request Feb 17, 2026

feat(coverage): add estree token conformance scaffolding #19496

Merged

github-actions bot added A-parser Area - Parser A-formatter Area - Formatter labels Feb 17, 2026

camc314 mentioned this pull request Feb 17, 2026

perf(linter/plugins): use Oxc tokens in plugins #19498

Merged

github-actions bot added the C-enhancement Category - New feature or request label Feb 17, 2026

camc314 force-pushed the c/02-17-feat_parser_estree_coverage_add_parser_tokens_and_shared_estree_conversion branch from 4dc586d to 4d88151 Compare February 17, 2026 16:39

camc314 force-pushed the c/02-17-feat_coverage_add_estree_token_conformance_scaffolding branch from 480fb45 to ef55626 Compare February 17, 2026 16:44

camc314 force-pushed the c/02-17-feat_parser_estree_coverage_add_parser_tokens_and_shared_estree_conversion branch 2 times, most recently from eaa2671 to d67073b Compare February 17, 2026 16:56

camc314 force-pushed the c/02-17-feat_coverage_add_estree_token_conformance_scaffolding branch from ef55626 to 781641e Compare February 17, 2026 17:01

camc314 force-pushed the c/02-17-feat_parser_estree_coverage_add_parser_tokens_and_shared_estree_conversion branch 3 times, most recently from 5cb52ed to 56e8c4f Compare February 17, 2026 17:18

overlookmotel reviewed Feb 17, 2026

View reviewed changes

crates/oxc_parser/src/lexer/mod.rs Show resolved Hide resolved

crates/oxc_parser/src/lexer/punctuation.rs Show resolved Hide resolved

camc314 force-pushed the c/02-17-feat_parser_estree_coverage_add_parser_tokens_and_shared_estree_conversion branch 2 times, most recently from a7e3db8 to 5d93223 Compare February 17, 2026 21:53

camc314 commented Feb 17, 2026

View reviewed changes

crates/oxc_estree_tokens/src/lib.rs Show resolved Hide resolved

camc314 commented Feb 17, 2026

View reviewed changes

crates/oxc_estree_tokens/src/lib.rs Show resolved Hide resolved

camc314 force-pushed the c/02-17-feat_parser_estree_coverage_add_parser_tokens_and_shared_estree_conversion branch from 5d93223 to 53e2396 Compare February 17, 2026 22:26

camc314 commented Feb 17, 2026

View reviewed changes

crates/oxc_parser/src/lexer/mod.rs Show resolved Hide resolved

camc314 commented Feb 17, 2026

View reviewed changes

crates/oxc_estree_tokens/src/lib.rs Show resolved Hide resolved

camc314 force-pushed the c/02-17-feat_coverage_add_estree_token_conformance_scaffolding branch from 781641e to a52fd1b Compare February 18, 2026 09:56

camc314 force-pushed the c/02-17-feat_parser_estree_coverage_add_parser_tokens_and_shared_estree_conversion branch from 53e2396 to a13b812 Compare February 18, 2026 09:56

camc314 mentioned this pull request Feb 18, 2026

chore(submodules): upgrade estree-conformance to latest SHA #19515

Closed

camc314 force-pushed the c/02-17-feat_parser_estree_coverage_add_parser_tokens_and_shared_estree_conversion branch from a13b812 to 28676ac Compare February 18, 2026 10:12

camc314 force-pushed the c/02-17-feat_coverage_add_estree_token_conformance_scaffolding branch from a52fd1b to 5cbebb7 Compare February 18, 2026 10:12

camc314 mentioned this pull request Feb 18, 2026

fix(tasks/coverage): bring back acorn-jsx estree testing #19516

Merged

camc314 force-pushed the c/02-17-feat_coverage_add_estree_token_conformance_scaffolding branch from 5cbebb7 to 97f88f9 Compare February 18, 2026 10:54

This was referenced Mar 2, 2026

release(crates): oxc v0.116.0 #19911

Merged

release(apps): oxlint v1.51.0 && oxfmt v0.36.0 #19912

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(parser,estree,coverage): collect tokens in parser and convert to ESTree format#19497

feat(parser,estree,coverage): collect tokens in parser and convert to ESTree format#19497
graphite-app[bot] merged 1 commit intomainfrom
c/02-17-feat_parser_estree_coverage_add_parser_tokens_and_shared_estree_conversion

camc314 commented Feb 17, 2026 •

edited by overlookmotel

Loading

Uh oh!

camc314 commented Feb 17, 2026 •

edited by overlookmotel

Loading

Uh oh!

codspeed-hq bot commented Feb 17, 2026 •

edited

Loading

Uh oh!

overlookmotel left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

camc314 commented Feb 17, 2026 • edited by overlookmotel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

camc314 commented Feb 17, 2026 • edited by overlookmotel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

codspeed-hq bot commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 15.38%

Performance Changes

Footnotes

Uh oh!

overlookmotel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

camc314 commented Feb 17, 2026 •

edited by overlookmotel

Loading

camc314 commented Feb 17, 2026 •

edited by overlookmotel

Loading

codspeed-hq bot commented Feb 17, 2026 •

edited

Loading

overlookmotel left a comment •

edited

Loading