TRT-1678: write the install upgrade and e2e duration metrics #30534

jianlinliu · 2025-11-26T04:08:40Z

No description provided.

openshift-ci-robot · 2025-11-26T04:08:43Z

Pipeline controller notification
This repository is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

openshift-ci-robot · 2025-11-26T07:47:01Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-upgrade-rollback

openshift-ci-robot · 2025-11-26T10:14:56Z

@jianlinliu: This pull request references TRT-1678 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-11-28T10:01:49Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-upgrade-rollback

petr-muller · 2025-12-04T15:55:31Z

/cc

neisw · 2025-12-04T21:26:59Z

pkg/test/ginkgo/cmd_runsuite.go

 		logrus.Infof("Retry strategy %s decided not to retry %d failing tests", o.RetryStrategy.Name(), fail)
 	}

+	end := time.Now()


This is interesting, I wonder if there is any value in tracking the time for e2e before the retries as well as the total e2e time with the retries...

This morning, I misunderstood the comment a bit. Now I introduced a new set of variables for retries to track the overall e2e duration with retries together with the e2e duration before retries. Now it should make everyone be happy.

For the total time duration of a run suite, you can probably just move end to incorporate retries in the total duration.

If you wanted to create a new metric to count retries in their own block that might be interesting to see how much time we spend on them

neisw · 2025-12-04T21:39:03Z

pkg/dataloader/types.go

 	return nil
 }
+
+// WriteDurations writes multiple duration metrics to a file in the autodol format.


typo autodol -> autodl

I feel like this function would fit better under e2eanalysis either as a separate util or in e2eanalysis.go

Sure. Moved it to e2eanalysis.go

openshift-ci-robot · 2025-12-05T10:24:47Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-upgrade-rollback

openshift-ci-robot · 2025-12-05T11:07:30Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-upgrade-rollback

openshift-ci-robot · 2025-12-05T11:59:41Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-upgrade-rollback

openshift-trt · 2025-12-05T16:11:22Z

Job Failure Risk Analysis for sha: eae382f

Job Name	Failure Risk
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade	IncompleteTests Tests for this run (26) are below the historical average (2107): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

jianlinliu · 2025-12-08T02:54:28Z

/test e2e-vsphere-ovn

jianlinliu · 2025-12-08T02:54:36Z

/test e2e-gcp-ovn-upgrade

jianlinliu · 2025-12-08T06:58:18Z

Per the test result, install duration (install_bootstrap and install_overall), e2e duration (e2e and e2e_with_retries), upgrade duration are collected sucessfully.

jianlinliu · 2025-12-08T06:58:36Z

/verified by @jianlinliu

openshift-ci-robot · 2025-12-08T06:58:46Z

@jianlinliu: This PR has been marked as verified by @jianlinliu.

In response to this:

/verified by @jianlinliu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

neisw · 2025-12-08T12:29:19Z

/test e2e-gcp-ovn-upgrade

enabled the cloud function to pull in these artifacts for validation

neisw · 2025-12-08T12:30:18Z

pkg/e2eanalysis/e2e_analysis.go

+		},
+		Rows: rows,
+	}
+	fileName := filepath.Join(artifactDir, fmt.Sprintf("duration-metric-%s%s-%s", name, timeSuffix, dataloader.AutoDataLoaderSuffix))


duration-metrics here as well

neisw · 2025-12-08T12:30:44Z

pkg/e2eanalysis/e2e_analysis.go

+	}
+
+	dataFile := dataloader.DataFile{
+		TableName: "duration-metric",


apologies for missing this before, could you make this plural duration-metrics

stbenjam · 2025-12-08T14:15:32Z

pkg/test/ginkgo/cmd_runsuite.go

 		logrus.Infof("Retry strategy %s decided not to retry %d failing tests", o.RetryStrategy.Name(), fail)
 	}

+	end_with_retries := time.Now()


endWithRetries

Please follow https://go.dev/doc/effective_go#mixed-caps

openshift-ci-robot · 2025-12-08T15:04:02Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-upgrade-rollback

neisw · 2025-12-08T20:02:52Z

/lgtm
I see the data showing up in duration-metrics

openshift-ci · 2025-12-08T20:03:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jianlinliu, neisw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [neisw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jianlinliu · 2025-12-09T01:18:58Z

/verified by @jianlinliu

openshift-ci-robot · 2025-12-09T01:19:09Z

@jianlinliu: This PR has been marked as verified by @jianlinliu.

In response to this:

/verified by @jianlinliu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

neisw · 2025-12-09T12:42:21Z

/label acknowledge-critical-fixes-only

this is data we want to start collecting for 4.21

openshift-ci · 2025-12-09T13:15:17Z

@jianlinliu: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot requested review from p0lyn0mial and sjenning November 26, 2025 04:10

jianlinliu changed the title ~~write the install upgrade and e2e duration metrics~~ TRT-1678: write the install upgrade and e2e duration metrics Nov 26, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 26, 2025

jianlinliu force-pushed the TRT-1678 branch from a4ac02a to 4efc9c6 Compare November 28, 2025 09:40

openshift-ci bot requested a review from petr-muller December 4, 2025 15:55

neisw reviewed Dec 4, 2025

View reviewed changes

jianlinliu force-pushed the TRT-1678 branch from 4efc9c6 to 1b82e96 Compare December 5, 2025 09:58

jianlinliu force-pushed the TRT-1678 branch from 1b82e96 to 27a7c0a Compare December 5, 2025 10:46

jianlinliu force-pushed the TRT-1678 branch from 27a7c0a to eae382f Compare December 5, 2025 11:34

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Dec 8, 2025

neisw reviewed Dec 8, 2025

View reviewed changes

stbenjam reviewed Dec 8, 2025

View reviewed changes

write the install upgrade and e2e duration metrics

0ab5c63

jianlinliu force-pushed the TRT-1678 branch from eae382f to 0ab5c63 Compare December 8, 2025 14:44

openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Dec 8, 2025

openshift-ci bot assigned neisw Dec 8, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 8, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 8, 2025

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Dec 9, 2025

openshift-ci bot added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Dec 9, 2025

openshift-merge-bot bot merged commit 20ed8f4 into openshift:main Dec 9, 2025
21 checks passed

TRT-1678: write the install upgrade and e2e duration metrics #30534

TRT-1678: write the install upgrade and e2e duration metrics #30534

Conversation

jianlinliu commented Nov 26, 2025

Uh oh!

openshift-ci-robot commented Nov 26, 2025

Uh oh!

openshift-ci-robot commented Nov 26, 2025

Uh oh!

openshift-ci-robot commented Nov 26, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 28, 2025

Uh oh!

petr-muller commented Dec 4, 2025

Uh oh!

neisw Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

jianlinliu Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stbenjam Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

neisw Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

jianlinliu Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Dec 5, 2025

Uh oh!

openshift-ci-robot commented Dec 5, 2025

Uh oh!

openshift-ci-robot commented Dec 5, 2025

Uh oh!

openshift-trt bot commented Dec 5, 2025

Uh oh!

jianlinliu commented Dec 8, 2025

Uh oh!

jianlinliu commented Dec 8, 2025

Uh oh!

jianlinliu commented Dec 8, 2025

Uh oh!

jianlinliu commented Dec 8, 2025

Uh oh!

openshift-ci-robot commented Dec 8, 2025

Uh oh!

neisw commented Dec 8, 2025

Uh oh!

neisw Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

neisw Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

stbenjam Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Dec 8, 2025

Uh oh!

neisw commented Dec 8, 2025

Uh oh!

openshift-ci bot commented Dec 8, 2025

Uh oh!

jianlinliu commented Dec 9, 2025

Uh oh!

openshift-ci-robot commented Dec 9, 2025

Uh oh!

neisw commented Dec 9, 2025

Uh oh!

openshift-ci bot commented Dec 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

openshift-ci-robot commented Nov 26, 2025 •

edited by openshift-ci bot

Loading

jianlinliu Dec 5, 2025 •

edited

Loading

neisw Dec 8, 2025 •

edited

Loading