feat(gepa): GEPA/GEPA-Flow Pareto optimizers + docs alignment#341
Merged
dosco merged 19 commits intoax-llm:mainfrom Sep 10, 2025
Merged
feat(gepa): GEPA/GEPA-Flow Pareto optimizers + docs alignment#341dosco merged 19 commits intoax-llm:mainfrom
dosco merged 19 commits intoax-llm:mainfrom
Conversation
…ling Adds a sample-efficient optimizer with reflective mutation, Pareto-based candidate selection, optional crossover, and multi-objective support. Includes typed options and examples, and improves progress logging of total rounds.
Implements a Flow-aware GEPA variant that selects modules round-robin and supports system-aware merge across candidates. Adds AxFlow helpers to expose/set node instructions and exports the optimizer.
…e paper and simplify the API; refactor examples to remove basic variants and promote Pareto examples
…e GEPA-Pareto to GEPA in logs/metrics
…iPRO continues to use compilePareto
…g; align labels to GEPA-Flow
Implement paper’s Algorithm 2 candidate selection: - Track per-instance scalar scores on validation/Pareto set (S matrix) - Sample parent from non-dominated set weighted by per-instance wins - Compute scalar as mean of multi-objective metrics per instance - GEPA-Flow also samples second parent for merge via Algorithm 2 - Persist S for each accepted candidate to drive subsequent sampling Rationale: align behavior with the GEPA paper and improve exploration vs. archive crowding-distance selection; sets the stage for optional scalar acceptance gating in a follow-up.
Extract shared multi-objective utilities to paretoUtils and replace inline duplicates in gepa.ts and gepaFlow.ts. No functional changes; simplifies maintenance and sets up for paper parity.
…to size GEPA: enable periodic instruction merge with cap and progress reporting. GEPA-Flow: add merge caps, ancestry/desirability guards per Appendix F, and make Pareto set size configurable via args. Keeps acceptance via minibatch Pareto dominance.
…ards)\n\nIntroduce explicit D_feedback/D_pareto splits to control rollout budget; plumb evaluator textual feedback μ_f into reflection; default to σ-based minibatch acceptance with configurable epsilon; add scalarizer/metric-key for per-instance S and Pareto selection; implement system-aware merge guards (ancestor/outperforms, desirability, tried merges).
- Schedule merges via mergesDue/lastIterFoundNewProgram and skip reflective on merge attempts - Dominator-based pair + ancestor selection with desirability filter and duplicate-merge guard - Targeted subsample for merge acceptance (new_sum ≥ max(parent sums)); full eval on accept - Stricter minibatch acceptance; when adapter provided, also require minibatch sum(child) > sum(parent) - Parent selection via per-instance fronts; honor maxMetricCalls; preserve fallback behavior without adapter This aligns both GEPA and GEPA-Flow with the reference engine while keeping public API stable.
… source parity Seed RNG across selection/minibatching, enforce maxMetricCalls budget, and add Flow merge guards/de-dup for stable improvements; re-export adapter types and update examples.
Add domain terms (GEPA, Traj, etc.) to cspell and ignore dist to keep spelling checks green. Rename unused vars and drop unused imports in GEPA optimizers to satisfy lint without behavior changes.
…aults Align GEPA merges with source: replace LLM merge with parent-pick, add ancestor/desirability guards and de-dup, use seeded sampling, schedule merges after accepted improvements; default merges off and skipPerfectScore on to match reference behavior.
…d defaults Bring GEPA in line with source parity by tolerating score ties, removing the hard budget requirement, and defaulting skip‑perfect in flow to match single‑module; Pareto frontier now respects epsilon.
Enforce a positive `options.maxMetricCalls` in GEPA/GEPA-Flow compile loops to match the source implementation and avoid unbounded optimization runs. BREAKING CHANGE: compile now throws if `options.maxMetricCalls` is absent or non-positive.
…lign single-module merge gating with the reference engine so reflective mutation is skipped only when a merge is actually attempted, improving behavioral parity and avoiding lost reflective iterations when no valid merge pair exists.
…compile (remove compilePareto)
Closed
Collaborator
|
sorry for the delay, was travling, will take a look later today. |
joshvfleming
pushed a commit
to joshvfleming/ax
that referenced
this pull request
Oct 14, 2025
…#341) * feat(optimizer): introduce GEPA reflective evolution with Pareto sampling Adds a sample-efficient optimizer with reflective mutation, Pareto-based candidate selection, optional crossover, and multi-objective support. Includes typed options and examples, and improves progress logging of total rounds. * feat(optimizer): add GEPA-Flow for multi-module reflective evolution Implements a Flow-aware GEPA variant that selects modules round-robin and supports system-aware merge across candidates. Adds AxFlow helpers to expose/set node instructions and exports the optimizer. * feat(optimizer): make GEPA and GEPA-Flow Pareto-only to align with the paper and simplify the API; refactor examples to remove basic variants and promote Pareto examples * refactor(optimizer): remove legacy single-objective compile and rename GEPA-Pareto to GEPA in logs/metrics * docs(optimize): clarify that GEPA/GEPA-Flow use compile for Pareto; MiPRO continues to use compilePareto * chore(gepa-flow): use flow() factory and add OptimizationStart logging; align labels to GEPA-Flow * feat(gepa,gepa-flow): adopt per-instance Pareto selection (Alg. 2) Implement paper’s Algorithm 2 candidate selection: - Track per-instance scalar scores on validation/Pareto set (S matrix) - Sample parent from non-dominated set weighted by per-instance wins - Compute scalar as mean of multi-objective metrics per instance - GEPA-Flow also samples second parent for merge via Algorithm 2 - Persist S for each accepted candidate to drive subsequent sampling Rationale: align behavior with the GEPA paper and improve exploration vs. archive crowding-distance selection; sets the stage for optional scalar acceptance gating in a follow-up. * refactor(optimizer): centralize Pareto helpers for GEPA/GEPA-Flow Extract shared multi-objective utilities to paretoUtils and replace inline duplicates in gepa.ts and gepaFlow.ts. No functional changes; simplifies maintenance and sets up for paper parity. * feat(gepa,gepa-flow): add Merge strategy and guards; parametrize Pareto size GEPA: enable periodic instruction merge with cap and progress reporting. GEPA-Flow: add merge caps, ancestry/desirability guards per Appendix F, and make Pareto set size configurable via args. Keeps acceptance via minibatch Pareto dominance. * feat(gepa,gepa-flow): align with GEPA paper (splits, μf, σ-accept, guards)\n\nIntroduce explicit D_feedback/D_pareto splits to control rollout budget; plumb evaluator textual feedback μ_f into reflection; default to σ-based minibatch acceptance with configurable epsilon; add scalarizer/metric-key for per-instance S and Pareto selection; implement system-aware merge guards (ancestor/outperforms, desirability, tried merges). * feat(gepa,gepa-flow): source-parity merges, acceptance, and adapter path - Schedule merges via mergesDue/lastIterFoundNewProgram and skip reflective on merge attempts - Dominator-based pair + ancestor selection with desirability filter and duplicate-merge guard - Targeted subsample for merge acceptance (new_sum ≥ max(parent sums)); full eval on accept - Stricter minibatch acceptance; when adapter provided, also require minibatch sum(child) > sum(parent) - Parent selection via per-instance fronts; honor maxMetricCalls; preserve fallback behavior without adapter This aligns both GEPA and GEPA-Flow with the reference engine while keeping public API stable. * feat(gepa,gepa-flow): deterministic selection + strict acceptance for source parity Seed RNG across selection/minibatching, enforce maxMetricCalls budget, and add Flow merge guards/de-dup for stable improvements; re-export adapter types and update examples. * chore(cspell,gepa,gepa-flow): add GEPA terms; resolve lint warnings Add domain terms (GEPA, Traj, etc.) to cspell and ignore dist to keep spelling checks green. Rename unused vars and drop unused imports in GEPA optimizers to satisfy lint without behavior changes. * feat(gepa): source-parity single-module merges, guards, and safer defaults Align GEPA merges with source: replace LLM merge with parent-pick, add ancestor/desirability guards and de-dup, use seeded sampling, schedule merges after accepted improvements; default merges off and skipPerfectScore on to match reference behavior. * feat(gepa,gepa-flow,optimizer): epsilon ties, optional budget, aligned defaults Bring GEPA in line with source parity by tolerating score ties, removing the hard budget requirement, and defaulting skip‑perfect in flow to match single‑module; Pareto frontier now respects epsilon. * feat(gepa,gepa-flow): require maxMetricCalls for strict parity Enforce a positive `options.maxMetricCalls` in GEPA/GEPA-Flow compile loops to match the source implementation and avoid unbounded optimization runs. BREAKING CHANGE: compile now throws if `options.maxMetricCalls` is absent or non-positive. * fix(gepa): only skip reflective after an evaluated merge attempt\n\nAlign single-module merge gating with the reference engine so reflective mutation is skipped only when a merge is actually attempted, improving behavioral parity and avoiding lost reflective iterations when no valid merge pair exists. * docs(optimize): migrate multi-objective docs to GEPA/GEPA-Flow using compile (remove compilePareto) --------- Co-authored-by: Spacy <832235+dosco@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
What is the current behavior? (You can also link to an open issue here)
What is the new behavior (if this is a feature change)?
Other information:
References:
Example runs (for reproducibility):
GEPA example (src/examples/gepa.ts)
GEPA-Flow example (src/examples/gepa-flow.ts)