feat: Implement SARIF output (open-policy-agent#1042)

* feat: add SARIF output format support

Add Static Analysis Results Interchange Format (SARIF) v2.1.0 output support
to conftest. SARIF is a standard JSON format for static analysis tools.

- SARIF v2.1.0 schema compliance
- Includes file locations and rule metadata
- Tracks execution timing and status
- Test coverage
- Documentation

Signed-off-by: Ville Vesilehto <[email protected]>

* feat(output): implement SARIF output using go-sarif library

Add Static Analysis Results Interchange Format (SARIF) v2.1.0 output support
using the go-sarif library. This provides a standard JSON format for static
analysis results with proper schema compliance.

Key changes:
- Use go-sarif/v2 library instead of custom implementation
- Support all result types (failures, warnings, exceptions, successes)
- Add comprehensive test coverage with JSON comparison
- Document new output format in options.md

The SARIF output includes:
- File locations and rule metadata
- Proper result levels (error/warning/note/none)
- Execution status and exit codes
- Rule properties from result metadata

Signed-off-by: Ville Vesilehto <[email protected]>

* refactor: address pr comments

- refactor: remove getRuleIndex
  Use direct map lookups instead

- refactor: succinct map lookups
  Map lookup with a fallback

- refactor: move result type logic to addResult func
  Cleaner code, while not really idiomatic due to go-sarif
  library design.

- fix: treat exceptions as success
  A file with only exceptions will be treated as a success. Exceptions
  will still be logged (with level "note") for visibility. The exit code
  will be 0 (success) when there are only exceptions.

- refactor: simplify hasFailures and hasWarnings
  Risk of typo is too high

- refactor: treat exceptions as successes in SARIF output
  Exceptions are now treated as successes in the SARIF output, removing
  the separate exception handling.

- test: type safe test input for SARIF
  Probably helps writing further test cases, instead of bare JSON

- refactor: use google/go-cmp for json diff
  Based on PR comment

Signed-off-by: Ville Vesilehto <[email protected]>

---------

Signed-off-by: Ville Vesilehto <[email protected]>

docs/options.md

@@ -173,6 +173,7 @@ As of today Conftest supports the following output types:
 - JUnit `--output=junit`
 - GitHub `--output=github`
 - AzureDevOps `--output=azuredevops`
+- SARIF `--output=sarif`
 
 ### Plaintext
 
@@ -322,6 +323,13 @@ success file=examples/kubernetes/deployment.yaml 1
 5 tests, 1 passed, 0 warnings, 4 failures, 0 exceptions
 ```
 
+### SARIF
+
+```console
+$ conftest test --proto-file-dirs examples/textproto/protos -p examples/textproto/policy examples/textproto/fail.textproto -o sarif
+{"version":"2.1.0","$schema":"https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json","runs":[{"tool":{"driver":{"informationUri":"https://github.com/open-policy-agent/conftest","name":"conftest","rules":[{"id":"main/deny","shortDescription":{"text":"Policy violation"}}]}},"invocations":[{"executionSuccessful":true,"exitCode":1,"exitCodeDescription":"Policy violations found"}],"results":[{"ruleId":"main/deny","ruleIndex":0,"level":"error","message":{"text":"fail: Power level must be over 9000"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"examples/textproto/fail.textproto"}}}]}]}]}
+```
+
 ## `--parser`
 
 Conftest normally detects which parser to used based on the file extension of the file, even when multiple input files are passed in. However, it is possible force a specific parser to be used with the `--parser` flag.

go.mod

@@ -85,6 +85,7 @@ require (
 	github.com/moby/docker-image-spec v1.3.1 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/opencontainers/go-digest v1.0.0 // indirect
+	github.com/owenrumney/go-sarif/v2 v2.3.3 // indirect
 	github.com/pelletier/go-toml/v2 v2.2.3 // indirect
 	github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
 	github.com/prometheus/client_golang v1.20.5 // indirect

go.sum

@@ -1007,6 +1007,10 @@ github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8
 github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
 github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug=
 github.com/opencontainers/image-spec v1.1.0/go.mod h1:W4s4sFTMaBeK1BQLXbG4AdM2szdn85PY75RI83NrTrM=
+github.com/owenrumney/go-sarif v1.1.1 h1:QNObu6YX1igyFKhdzd7vgzmw7XsWN3/6NMGuDzBgXmE=
+github.com/owenrumney/go-sarif v1.1.1/go.mod h1:dNDiPlF04ESR/6fHlPyq7gHKmrM0sHUvAGjsoh8ZH0U=
+github.com/owenrumney/go-sarif/v2 v2.3.3 h1:ubWDJcF5i3L/EIOER+ZyQ03IfplbSU1BLOE26uKQIIU=
+github.com/owenrumney/go-sarif/v2 v2.3.3/go.mod h1:MSqMMx9WqlBSY7pXoOZWgEsVB4FDNfhcaXDA1j6Sr+w=
 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
 github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M=
 github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc=
@@ -1146,6 +1150,7 @@ github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1
 github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
 github.com/zclconf/go-cty v1.2.0/go.mod h1:hOPWgoHbaTUnI5k4D2ld+GRpFJSCe6bCM7m1q/N4PQ8=
 github.com/zclconf/go-cty v1.6.1/go.mod h1:VDR4+I79ubFBGm1uJac1226K5yANQFHeauxPBoP54+o=
+github.com/zclconf/go-cty v1.10.0/go.mod h1:vVKLxnk3puL4qRAv72AO+W99LUD4da90g3uUAzyuvAk=
 github.com/zclconf/go-cty v1.13.2 h1:4GvrUxe/QUDYuJKAav4EYqdM47/kZa672LwmXFmEKT0=
 github.com/zclconf/go-cty v1.13.2/go.mod h1:YKQzy/7pZ7iq2jNFzy5go57xdxdWoLLpaEp4u238AE0=
 github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
@@ -1871,6 +1876,7 @@ google.golang.org/protobuf v1.36.3 h1:82DV7MYdb8anAVi3qge1wSnMDrnKK7ebr+I0hHRN1B
 google.golang.org/protobuf v1.36.3/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
 gopkg.in/cheggaaa/pb.v1 v1.0.27/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw=

output/output.go

@@ -34,6 +34,7 @@ const (
 	OutputJUnit       = "junit"
 	OutputGitHub      = "github"
 	OutputAzureDevOps = "azuredevops"
+	OutputSARIF       = "sarif"
 )
 
 // Get returns a type that can render output in the given format.
@@ -57,6 +58,8 @@ func Get(format string, options Options) Outputter {
 		return NewGitHub(options.File)
 	case OutputAzureDevOps:
 		return NewAzureDevOps(options.File)
+	case OutputSARIF:
+		return NewSARIF(options.File)
 	default:
 		return NewStandard(options.File)
 	}
@@ -72,5 +75,6 @@ func Outputs() []string {
 		OutputJUnit,
 		OutputGitHub,
 		OutputAzureDevOps,
+		OutputSARIF,
 	}
 }

output/output_test.go

@@ -39,6 +39,10 @@ func TestGetOutputter(t *testing.T) {
 			input:    OutputAzureDevOps,
 			expected: NewAzureDevOps(os.Stdout),
 		},
+		{
+			input:    OutputSARIF,
+			expected: NewSARIF(os.Stdout),
+		},
 		{
 			input:    "unknown_format",
 			expected: NewStandard(os.Stdout),

output/sarif.go

@@ -0,0 +1,210 @@
+package output
+
+import (
+	"fmt"
+	"io"
+	"path/filepath"
+	"strings"
+
+	"github.com/open-policy-agent/opa/tester"
+	"github.com/owenrumney/go-sarif/v2/sarif"
+	"golang.org/x/exp/slices"
+)
+
+const (
+	// Tool information
+	toolName     = "conftest"
+	toolURI      = "https://github.com/open-policy-agent/conftest"
+	sarifVersion = sarif.Version210
+
+	// Result descriptions
+	successDesc   = "Policy was satisfied successfully"
+	skippedDesc   = "Policy check was skipped"
+	failureDesc   = "Policy violation"
+	warningDesc   = "Policy warning"
+	exceptionDesc = "Policy exception"
+
+	// Exit code descriptions
+	exitNoViolations = "No policy violations found"
+	exitViolations   = "Policy violations found"
+	exitWarnings     = "Policy warnings found"
+)
+
+// SARIF represents an Outputter that outputs results in SARIF format.
+type SARIF struct {
+	writer io.Writer
+}
+
+// NewSARIF creates a new SARIF with the given writer.
+func NewSARIF(w io.Writer) *SARIF {
+	return &SARIF{
+		writer: w,
+	}
+}
+
+// getRuleID generates a stable rule ID based on namespace and rule type
+func getRuleID(namespace string, ruleType string) string {
+	return fmt.Sprintf("%s/%s", namespace, ruleType)
+}
+
+// getRuleDescription returns the appropriate description based on the rule type
+func getRuleDescription(ruleID string) string {
+	switch {
+	case strings.HasSuffix(ruleID, "/success"):
+		return successDesc
+	case strings.HasSuffix(ruleID, "/skip"):
+		return skippedDesc
+	case strings.HasSuffix(ruleID, "/allow"):
+		return exceptionDesc
+	case strings.HasSuffix(ruleID, "/warn"):
+		return warningDesc
+	default:
+		return failureDesc
+	}
+}
+
+// addRuleIndex adds a new rule to the SARIF run and returns its index.
+func addRuleIndex(run *sarif.Run, ruleID string, result Result, indices map[string]int) int {
+	addRule(run, ruleID, result)
+	idx := len(run.Tool.Driver.Rules) - 1
+	indices[ruleID] = idx
+	return idx
+}
+
+// addRule adds a new rule to the SARIF run with the given ID and result metadata.
+func addRule(run *sarif.Run, ruleID string, result Result) {
+	desc := getRuleDescription(ruleID)
+	rule := run.AddRule(ruleID).
+		WithDescription(desc).
+		WithShortDescription(&sarif.MultiformatMessageString{
+			Text: &desc,
+		})
+
+	if result.Metadata != nil {
+		props := sarif.NewPropertyBag()
+		for k, v := range result.Metadata {
+			props.Add(k, v)
+		}
+		rule.WithProperties(props.Properties)
+	}
+}
+
+// addResult adds a result to the SARIF run
+func addResult(run *sarif.Run, result Result, namespace, ruleType, level, fileName string, indices map[string]int) {
+	ruleID := getRuleID(namespace, ruleType)
+	idx, ok := indices[ruleID]
+	if !ok {
+		idx = addRuleIndex(run, ruleID, result, indices)
+	}
+
+	run.CreateResultForRule(ruleID).
+		WithRuleIndex(idx).
+		WithLevel(level).
+		WithMessage(sarif.NewTextMessage(result.Message)).
+		AddLocation(
+			sarif.NewLocationWithPhysicalLocation(
+				sarif.NewPhysicalLocation().
+					WithArtifactLocation(
+						sarif.NewSimpleArtifactLocation(filepath.ToSlash(fileName)),
+					),
+			),
+		)
+}
+
+// Output outputs the results in SARIF format.
+func (s *SARIF) Output(results []CheckResult) error {
+	report, err := sarif.New(sarifVersion)
+	if err != nil {
+		return fmt.Errorf("create sarif report: %w", err)
+	}
+
+	run := sarif.NewRunWithInformationURI(toolName, toolURI)
+	indices := make(map[string]int)
+
+	for _, result := range results {
+		// Process failures
+		for _, failure := range result.Failures {
+			addResult(run, failure, result.Namespace, "deny", "error", result.FileName, indices)
+		}
+
+		// Process warnings
+		for _, warning := range result.Warnings {
+			addResult(run, warning, result.Namespace, "warn", "warning", result.FileName, indices)
+		}
+
+		// Process exceptions (treated as successes)
+		hasSuccesses := result.Successes > 0
+		for _, exception := range result.Exceptions {
+			addResult(run, exception, result.Namespace, "allow", "note", result.FileName, indices)
+			hasSuccesses = true
+		}
+
+		// Don't add success/skip results if there are failures or warnings
+		hasErrors := len(result.Failures) > 0 || len(result.Warnings) > 0
+		if hasErrors {
+			continue
+		}
+
+		// Add success/exception results if there are no failures or warnings
+		if hasSuccesses {
+			statusResult := Result{
+				Message: successDesc,
+				Metadata: map[string]interface{}{
+					"description": successDesc,
+				},
+			}
+			addResult(run, statusResult, result.Namespace, "success", "none", result.FileName, indices)
+		} else {
+			statusResult := Result{
+				Message: skippedDesc,
+				Metadata: map[string]interface{}{
+					"description": skippedDesc,
+				},
+			}
+			addResult(run, statusResult, result.Namespace, "skip", "none", result.FileName, indices)
+		}
+	}
+
+	// Add run metadata
+	exitCode := 0
+	exitDesc := exitNoViolations
+	if hasFailures(results) {
+		exitCode = 1
+		exitDesc = exitViolations
+	} else if hasWarnings(results) {
+		exitDesc = exitWarnings
+	}
+
+	successful := true
+	invocation := sarif.NewInvocation()
+	invocation.ExecutionSuccessful = &successful
+	invocation.ExitCode = &exitCode
+	invocation.ExitCodeDescription = &exitDesc
+
+	run.Invocations = []*sarif.Invocation{invocation}
+
+	// Add the run to the report
+	report.AddRun(run)
+
+	// Write the report
+	return report.Write(s.writer)
+}
+
+// Report is not supported in SARIF output
+func (s *SARIF) Report(_ []*tester.Result, _ string) error {
+	return fmt.Errorf("report is not supported in SARIF output")
+}
+
+// hasFailures returns true if any of the results contain failures
+func hasFailures(results []CheckResult) bool {
+	return slices.ContainsFunc(results, func(r CheckResult) bool {
+		return len(r.Failures) > 0
+	})
+}
+
+// hasWarnings returns true if any of the results contain warnings
+func hasWarnings(results []CheckResult) bool {
+	return slices.ContainsFunc(results, func(r CheckResult) bool {
+		return len(r.Warnings) > 0
+	})
+}