gh-111474: Add color and a fancy output mode to regrtest by Yhg1s · Pull Request #111475 · python/cpython

Yhg1s · 2023-10-29T22:16:54Z

Refactor regrtest to push more decisions about formatting to the Logger class. Add color support to Logger, and add a FancyLogger subclass that uses vt100 terminal control to make the test output much more condensed and easier to visually scan.

The color support for logger uses just bold green, bold red and normal yellow to print "good", "bad" and "notable" things. This makes anomalous things and interleaved test output stand out a lot more:

Color support is enabled by passing --color to regrtest (for example by setting the EXTRATESTOPTS environment variable in the make test call.) It doesn't try to detect color support. It obeys the NO_COLOR environment variable to disable color.

The fancy logger is enabled by passing --progress_reporter=fancy (or by passing --progress_reporter=detect and having a functioning terminal with a known terminal type). It enables color by default, but it still obeys NO_COLOR. The fancy logger will keep redrawing the same lines on the screen rather than printing new lines, reporting all running tests. Test output, test failures, and output that is longer than can be printed on a single line will still appear on their own lines, without being overwritten. (The exception is still-running tests; if those are too long they will get truncated as long as they are running, but are printed in full when they finish.) An additional option, --fancy_report_skip_reason, can be passed to make the fancy reporter print test skips with full test reason on a line of their own.

Here is a short demo of the fancy reporter running:

fancy-regrtest-short-demo.mp4

Here's what it looks like when test output needs to be printed:

And here's what it looks like when using the --fancy_report_skip_reason option is added:

The refactoring to support the fancy output logger are probably enough to also enable a Tkinter-based logger, although that's more work than I'm willing to put into it (since I would not use it myself).

With a very small exception, existing behaviour is not changed. The new options (--color and --progress_reporter) have to explicitly be passed, via the EXTRATESTOPTS environment variable or the TESTOPTS make variable, to enable them. The exception is the regrtest output when it's been waiting on running tests and doesn't have any test results to report for more than 30 seconds. Before the refactoring this looked like:

0:00:00 load avg: 0.09 Run 4 tests in parallel using 4 worker processes
0:00:00 load avg: 0.09 [1/4] test_keyword passed
0:00:00 load avg: 0.09 [2/4] test_list passed
0:00:00 load avg: 0.09 [3/4] test_dict passed
0:00:30 load avg: 0.06 running (1): test_multiprocessing_forkserver.test_threads (30.3 sec)
0:00:42 load avg: 0.05 [4/4] test_multiprocessing_forkserver.test_threads passed (42.4 sec)

but now it looks like:

0:00:00 load avg: 0.03 Run 4 tests in parallel using 4 worker processes
0:00:00 load avg: 0.03 [1/4] test_keyword passed
0:00:00 load avg: 0.03 [2/4] test_list passed
0:00:00 load avg: 0.03 [3/4] test_dict passed
0:00:30 load avg: 0.07 [3/4] running (1): test_multiprocessing_forkserver.test_threads 30.3 sec
0:00:42 load avg: 0.19 [4/4] test_multiprocessing_forkserver.test_threads passed (42.5 sec)

Changing the output back is not impossible, but to my eyes the new output makes more sense anyway.

Issue: Improve regrtest output #111474

Yhg1s · 2023-10-29T22:21:15Z

(Making the PR a draft one because I have yet to add tests for the new code.)

gaogaotiantian · 2023-10-29T22:21:57Z

Just curious - what does the output look like if the background is white-ish? I would assume that's a very common scenario.

Yhg1s · 2023-10-29T22:40:12Z

It depends on the terminal's colors. I don't have a dark-on-light terminal configuration handy myself (I only ever use light-on-dark), but I would generally expect such configurations to have reasonable contrasting colors for green, yellow and red. (I could make the colors configurable, but I'd still want to stick to the basic 16 vt100 colors because of their ubiquity.)

terryjreedy · 2023-10-30T07:45:54Z

On Windows, new -h output is

  --color, --no-color   use color in the progress reports
  --progress_reporter {plain,fancy,detect}
  --fancy_report_skip_reason
                        in the fancy reporter, report the skip reason

The progress_reporter set should be indented 2 more spaces. argparse bug?

I tried running on Windows since I believe CommandPrompt and/or PowerShell either do or can be
made to recognize at least a subset of VT100 sequences. In logger.FancyLogger.setup_terminal (221),
the termios import and SIGWINCH accessess fail. When I commented those out and just set
self.line/columns for my window it ran apparently fine -- except that VT100 commands are simply printed.
Ditto with PowerShell (which does color coding itself of executable names and options in command
and of error messages). Perhaps someone else (such as Eryk Sun) could say how to turn on recognition
and how to query current lines and columns and whether resizes can be signaled.

sobolevn

Just looking at annotations right now :)

The feature itself looks great! Thank you!

sobolevn · 2023-10-30T07:43:02Z

+            self.color = False
+        self.load_threshold = os.process_cpu_count()
+
+    def error(self, s) -> str:


Suggested change

def error(self, s) -> str:

def error(self, s: str) -> str:

sobolevn · 2023-10-30T07:43:11Z

+            return s
+        return f'{self.ERROR_COLOR}{s}{self.RESET_COLOR}'
+
+    def warning(self, s) -> str:


Suggested change

def warning(self, s) -> str:

def warning(self, s: str) -> str:

sobolevn · 2023-10-30T07:43:20Z

+        return f'{self.INFO_COLOR}{s}{self.RESET_COLOR}'

-    def log(self, line: str = '') -> None:
+    def good(self, s) -> str:


Suggested change

def good(self, s) -> str:

def good(self, s: str) -> str:

sobolevn · 2023-10-30T07:43:32Z

+            return s
+        return f'{self.GOOD_COLOR}{s}{self.RESET_COLOR}'
+
+    def load_color(self, load_avg: float):


Suggested change

def load_color(self, load_avg: float):

def load_color(self, load_avg: float) -> str:

sobolevn · 2023-10-30T07:44:12Z

+            load = self.error(load)
+        return load
+
+    def state_color(self, text: str, state: str | None):


Suggested change

def state_color(self, text: str, state: str | None):

def state_color(self, text: str, state: State | str | None) -> str:

sobolevn · 2023-10-30T07:44:29Z

-        if self._quiet:
-            return
+    def display_progress(self, test_index: int, text: str,
+                         stdout: str|None = None) -> None:


nit

Suggested change

stdout: str|None = None) -> None:

stdout: str | None = None) -> None:

(and in other places)

sobolevn · 2023-10-30T07:46:20Z

+class StatusRemovingWrapper(io.TextIOWrapper):
+    # Set status_size here to avoid having to override __init__.
+    status_size = 0
+    def write(self, msg):


Suggested change

def write(self, msg):

def write(self, msg: str) -> None:

https://github.com/python/typeshed/blob/f7aa7b709a826ed34f52b1de3f7095f90f349a9c/stdlib/io.pyi#L143

sobolevn · 2023-10-30T07:46:41Z

+        super().write(msg)
+
+
+def replace_stdout():


Suggested change

def replace_stdout():

def replace_stdout() -> None:

sobolevn · 2023-10-30T07:46:53Z

+        run_workers.PROGRESS_UPDATE = 5
+        super().__init__(results, ns)
+
+    def setup_terminal(self):


Suggested change

def setup_terminal(self):

def setup_terminal(self) -> None:

sobolevn · 2023-10-30T07:47:12Z

+        sys.stdout.status_size = report.count('\n') + 1  # type: ignore[attr-defined]
+
+
+def detect_vt100_capability():


Suggested change

def detect_vt100_capability():

def detect_vt100_capability() -> bool:

terryjreedy · 2023-10-30T08:20:43Z

One way to implement a tkinter alternative in a followup issue/PR would be to add a 'tkgui' (or whatever) option to the reporter options. If selected, it could initialize a GuiLogger subclass of FancyLogger with a different setup_terminal() that would initialize a TestOutput window and do a different stdout replacement. Then the VT100 commands could be removed (with REs) in that replacement and replaced with terminal actions. I believe logic in the replacement write() function could replace the status removal stuff.

hugovk · 2023-10-30T15:58:10Z

Color support is enabled by passing --color to regrtest (for example by setting the EXTRATESTOPTS environment variable in the make test call.) It doesn't try to detect color support. It obeys the NO_COLOR environment variable to disable color.

Something to consider: many tools accept a three-state option for --color. For example, pytest accepts:

  --color=color         Color terminal output (yes/no/auto)

And pre-commit:

  --color {auto,always,never}
                        Whether to use color in output. Defaults to `auto`.

Following https://no-color.org/, these options override NO_COLOR (and FORCE_COLOR).

…in the logger class. Add a new (optional) logger class that uses VT100 escape sequences for colourised, multi-line status reports.

…e to print skips on a line of their own even if they would fit in a regular status report.

…to just the fancy reporter.

long-running tests finish.

github-actions · 2026-04-14T12:19:57Z

This PR is stale because it has been open for 30 days with no activity.

bedevere-app Bot mentioned this pull request Oct 29, 2023

Improve regrtest output #111474

Open

bedevere-app Bot added the awaiting core review label Oct 29, 2023

Yhg1s marked this pull request as draft October 29, 2023 22:17

bedevere-app Bot removed the awaiting core review label Oct 29, 2023

sobolevn reviewed Oct 30, 2023

View reviewed changes

Yhg1s added 10 commits September 4, 2024 14:42

Refactor libregrtest's test status reporting so more of the logic is …

db59200

…in the logger class. Add a new (optional) logger class that uses VT100 escape sequences for colourised, multi-line status reports.

Display more things as skipped.

a0de2bc

Refactor more of the reporting.

554a0e0

Add an explicit option to print skip reasons, and if passed, make sur…

3d8beab

…e to print skips on a line of their own even if they would fit in a regular status report.

Reorder the color options so it's more obvious that it doesn't apply …

a4fdf92

…to just the fancy reporter.

Make 'env changed' color red, not like a skip.

c970094

Add a few comments and fix (new) mypy errors.

55183c2

Clarify some comments.

da4b6b3

In the regular logger, also colorize the test duration reported when

d7b139a

long-running tests finish.

Use _colorize.

6941b25

Yhg1s force-pushed the fancy-regrtest branch from 673c3dd to 6941b25 Compare September 4, 2024 12:53

github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 14, 2026

	def warning(self, s) -> str:
	def warning(self, s: str) -> str:

	def load_color(self, load_avg: float):
	def load_color(self, load_avg: float) -> str:

	def state_color(self, text: str, state: str \| None):
	def state_color(self, text: str, state: State \| str \| None) -> str:

	stdout: str\|None = None) -> None:
	stdout: str \| None = None) -> None:

		sys.stdout.status_size = report.count('\n') + 1 # type: ignore[attr-defined]


		def detect_vt100_capability():

	def detect_vt100_capability():
	def detect_vt100_capability() -> bool:

Uh oh!

Conversation

Yhg1s commented Oct 29, 2023 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Yhg1s commented Oct 29, 2023

Uh oh!

gaogaotiantian commented Oct 29, 2023

Uh oh!

Yhg1s commented Oct 29, 2023

Uh oh!

terryjreedy commented Oct 30, 2023 • edited by AlexWaygood Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

terryjreedy commented Oct 30, 2023

Uh oh!

hugovk commented Oct 30, 2023

Uh oh!

github-actions Bot commented Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Yhg1s commented Oct 29, 2023 •

edited by bedevere-app Bot

Loading

terryjreedy commented Oct 30, 2023 •

edited by AlexWaygood

Loading