-
Notifications
You must be signed in to change notification settings - Fork 1.6k
[blog] Security vulnerability in confighttp (CVE-2024-36129) #4600
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
svrnm
merged 8 commits into
open-telemetry:main
from
jpkrohling:jpkrohling/CVE-2024-36129
Jun 5, 2024
Merged
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
0c4df62
[blog] Security vulnerability in confighttp (CVE-2024-36129)
jpkrohling adfab58
lint, check links
jpkrohling d37a1d4
removed broken link, smaller tweaks
jpkrohling c44ed87
Update content/en/blog/2024/cve-2024-36129/index.md
mx-psi 3ba41f4
Update content/en/blog/2024/cve-2024-36129/index.md
mx-psi 42b4011
Results from /fix:all
opentelemetrybot 5f042fb
other review change requests
jpkrohling 32b2c43
Merge branch 'main' into jpkrohling/CVE-2024-36129
svrnm File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| --- | ||
| title: Security vulnerability in confighttp (CVE-2024-36129) | ||
| linkTitle: Security vulnerability in confighttp | ||
| date: 2024-06-05 | ||
| author: | ||
| '[Juraci Paixão Kröhling](https://github.com/jpkrohling) (OpenTelemetry, | ||
| Grafana Labs), [Pablo Baeyens](https://github.com/mx-psi) (OpenTelemetry, | ||
| Datadog)' | ||
| cSpell:ignore: confighttp Baeyens OSTIF zstd configgrpc Miroslav Stampar | ||
| --- | ||
|
|
||
| On our path toward graduation, the OpenTelemetry project is currently undergoing | ||
| a security audit sponsored by the [CNCF](https://www.cncf.io/), facilitated by | ||
| [OSTIF](https://ostif.org/), and performed by | ||
| [7ASecurity](https://7asecurity.com/). During this process, we have received a | ||
| few ideas about things that we could do better, like using specific compiler | ||
| flags when preparing our [OpenTelemetry Collector | ||
| binaries](https://github.com/open-telemetry/opentelemetry-collector-releases/releases). | ||
| On 31 May 2024, we received [a more serious | ||
| report](https://github.com/open-telemetry/opentelemetry-collector/security/advisories/GHSA-c74f-6mfw-mm4v): | ||
| a malicious user could cause a denial of service (DoS) when using a specially | ||
| crafted HTTP or gRPC request. The advisory was assigned the following CVE | ||
| identifier: [CVE-2024-36129](https://nvd.nist.gov/vuln/detail/CVE-2024-36129). | ||
|
|
||
| When sending an HTTP request with a compressed payload, the Collector would | ||
| verify only whether the compressed payload is beyond a certain limit, but not | ||
| its uncompressed version. A malicious payload could then send a “[compressed | ||
| bomb](https://en.wikipedia.org/wiki/Zip_bomb)”, causing the Collector to crash. | ||
|
|
||
| Similarly, when sending a gRPC request using zstd compression, the decompression | ||
| mechanism would not respect the limits imposed by gRPC, also causing the | ||
| Collector to crash while decompressing the malicious payload. | ||
|
|
||
| A few business hours after the report, on 03 June 2024, Collector developers | ||
| were able to reproduce the report related to HTTP and worked together on a fix | ||
| that [got merged the same | ||
| day](https://github.com/open-telemetry/opentelemetry-collector/pull/10289). | ||
| Given the high score of this issue, we’ve decided to hold the release that would | ||
| have happened on the same day, [completing | ||
| it](https://github.com/open-telemetry/opentelemetry-collector-releases/releases/tag/v0.102.0) | ||
| on 04 June 2024 instead. | ||
|
|
||
| After the release, we got a confirmation that gRPC with zstd was also affected. | ||
| Within a few business hours of the confirmation, we worked on a fix that also | ||
| [got merged the same | ||
| day](https://github.com/open-telemetry/opentelemetry-collector/pull/10323). [We | ||
| released | ||
| v0.102.1](https://github.com/open-telemetry/opentelemetry-collector-releases/releases/tag/v0.102.1) | ||
| right after that. | ||
|
|
||
| ## Am I affected? | ||
|
|
||
| You are affected by this vulnerability if you have an OpenTelemetry Collector | ||
| with one or more HTTP or gRPC receivers on a public port, such as the OTLP | ||
| Receiver with the “HTTP” or “gRPC” protocol enabled (typically on ports 4318 and | ||
| 4317, respectively) AND the receiver has version 0.101.0 or below. The | ||
jpkrohling marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| vulnerability is exploitable only by attackers who can send payloads to your | ||
| HTTP/gRPC endpoint(s). | ||
mx-psi marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| Note that if you require authentication, an attacker would need to have valid | ||
| credentials in order to exploit the vulnerability using the HTTP protocol. For | ||
| gRPC, the exploitable code is executed before authentication. | ||
|
|
||
| ## What does it mean to me? | ||
|
|
||
| If you manage a Collector that has an interface to the public internet, you | ||
| should upgrade it as soon as feasible, and consider setting the parameter | ||
| “MaxRequestBodySize” on HTTP receivers, such as the OTLP receiver, to a value | ||
jpkrohling marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| that makes sense to your workload. Up to v0.101.0, this setting applied only to | ||
| the payload size sent by the client, which could often be compressed. | ||
|
|
||
| Starting from v0.102.0, this setting applies to uncompressed, compressed, and | ||
| decompressed payload sizes and we are establishing a default value of 20 MiB for | ||
| this. This new default characterizes a breaking change, as clients sending | ||
| payloads bigger than 20 MiB will start seeing an error. While we believe most | ||
| authentic requests will be way within this limit, it’s still wise to monitor | ||
| your Collector for increased error rates after this update. Here’s an example of | ||
| a configuration setting a different limit to this field: | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| otlp: | ||
| protocols: | ||
| http: | ||
| endpoint: localhost:4318 | ||
| max_request_body_size: 10485760 # 10 MiB | ||
| ``` | ||
|
|
||
| For gRPC receivers, it’s sufficient to upgrade to v0.102.1, as there’s a default | ||
| value being applied to the message size already. | ||
jpkrohling marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| If your Collector instances are receiving data only from trusted clients, like | ||
| your own applications, you are still encouraged to upgrade to the latest | ||
| Collector version but you can do it at your regular pace. | ||
|
|
||
| If you are using a custom distribution and building it with the [OpenTelemetry | ||
| Collector Builder | ||
| (ocb)](https://opentelemetry.io/docs/collector/custom-collector/), you can add a | ||
| [“replaces”](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder#configuration) | ||
| entry pointing to the latest version of the | ||
| [confighttp](http://go.opentelemetry.io/collector/config/confighttp) and | ||
| [configgrpc](http://go.opentelemetry.io/collector/config/configgrpc) Go module. | ||
| If your base Collector version is at v0.96.0 or higher, we do not expect any | ||
| compatibility issues by just bumping to the latest version. | ||
|
|
||
| ## Lessons learned | ||
|
|
||
| During this process, we found a couple of gaps in the telemetry for the | ||
| Collector, as well as to the options we provide to Collector admins. Concretely, | ||
| we noticed that we do not have a good way to verify what’s the distribution of | ||
| request sizes received by the Collector, which would have been useful to | ||
| determine whether the change would break clients for a given Collector. We also | ||
| noticed that we don’t provide a way for admins to completely disable | ||
| compression, which would be a good way to mitigate an attack without having to | ||
| upgrade the Collector. We are working to fill those gaps over the next releases. | ||
arminru marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| We are also working on stabilizing the component.UseLocalHostAsDefaultHost | ||
jpkrohling marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| feature gate to reduce exposure of all Collector endpoints by default. This | ||
| feature gate was motivated by a previous, [similar vulnerability on Go's | ||
| standard library](https://github.com/advisories/GHSA-4374-p667-p6c8) and has | ||
| been in alpha for several months. You can follow the discussion surrounding | ||
| stabilization at [issue | ||
| 8510](https://github.com/open-telemetry/opentelemetry-collector/issues/8510). | ||
|
|
||
| This issue was identified by [Miroslav Stampar](https://github.com/stamparm/), | ||
jpkrohling marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| from 7ASecurity. We’d like to thank 7ASecurity for the responsible reporting of | ||
| this vulnerability. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.