Fix path for devNull on Windows and Unix #1
+101
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
use a cross-platform approach using the nul device on Windows and /dev/null on Unix
- Add _merge_with_defaults() to auto-fill missing alpha values in phase config - If user config lacks alpha values, fill them from defaults or estimate based on rank - Add null checks in _validate_alpha_ratios() to prevent TypeError on None division - Add null checks in get_current_alpha() and get_current_scale() methods - Log which alpha values were auto-filled or estimated for user visibility Fixes: TypeError when phase alpha is None in configuration Improves: User experience with incomplete alpha scheduler configs by auto-completing them This ensures the alpha scheduler gracefully handles incomplete configurations while informing users which values were auto-filled.
relaxis
pushed a commit
that referenced
this pull request
Nov 4, 2025
…d EMA **ROOT CAUSES:** 1. NO boundary realignment when resuming from checkpoint - Training always reset to boundary_index=0, steps_this_boundary=0 - Caused incorrect expert labeling in metrics after every resume 2. Codex's attempted fix had off-by-one error - Used: steps_this_boundary = effective_step % switch_boundary_every - Should be: steps_this_boundary = (effective_step % switch_boundary_every) + 1 - After completing a step, steps_this_boundary has been incremented 3. Missing EMA calculations (user's #1 requested metric) - UI only showed simple averages, not exponential moving averages **EVIDENCE FROM METRICS:** - Steps 200-400: stayed high_noise (should switch at 300) - resume at 201/301 - Steps 500-700+: stayed high_noise (should switch at 600) - resume at 701 - Timestamp gaps confirmed resumes without realignment - Expert labels completely wrong after resume **FIXES:** jobs/process/BaseSDTrainProcess.py: - Fixed off-by-one error in boundary realignment - Added correct formula: (effective_step % switch_boundary_every) + 1 - Added debug logging for realignment state - Comprehensive comments explaining the math extensions_built_in/sd_trainer/SDTrainer.py: - Added boundary switch logging at multiples of 100 steps - Logs old_expert → new_expert transitions for debugging ui/src/components/JobMetrics.tsx: - Implemented EMA calculations with proper smoothing factor - Added per-expert EMA: highNoiseLossEMA, lowNoiseLossEMA - Added per-expert gradient stability EMA - Created dedicated EMA Loss display card - Updated expert comparison cards to show both simple avg and EMA - EMA weights recent values more heavily (α = 2/(N+1)) **TESTING:** - Next resume will log realignment state - Metrics will show correct expert labels - EMA values provide better training trend indicators - Window sizes 10/50/100 all have proper EMA calculations 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
use a cross-platform approach using the nul device on Windows and /dev/null on Unix
If you run your fork on Windows, the queue worker errors out due to unix syntax