Error indexing in elasticsearchexporter when duplicate attributes recieved for log records

[stevejgordon]

Component(s)

exporter/elasticsearch

What happened?

Description

The OTel spec states that attributes with the same key should not be exported. However, duplicates can occur with the .NET SDK when using the IncludeScopes option. I'll raise an issue for that separately. When these are sent to a collector using the Elasticsearch exporter, the indexing fails with:

2025-04-10T13:56:00.069+0100    error   elasticsearchexporter@v0.120.1/bulkindexer.go:343       bulk indexer flush error        {"otelcol.component.id": "elasticsearch/otel", "otelcol.component.kind": "Exporter", "otelcol.signal": "logs", "error": "flush failed (400): {\"error\":{\"type\":\"x_content_parse_exception\",\"caused_by\":{\"type\":\"json_parse_exception\"}}}"}

Steps to Reproduce

Send a log record with duplicate attributes to the collector. In an ASP.NET Core app, this can occur when using this registration, which enables IncludeScopes.

See open-telemetry/opentelemetry-dotnet#4324, which acknowledges this behaviour and non-compliance with the spec.

Expected Result

The log is exported to Elasticsearch

Actual Result

An error occurs, and the log (batch) is lost.

Collector version

0.120.1

Environment information

OpenTelemetry Collector configuration

Log output

Additional context

No response

[carsonip]

/label exporter/elasticsearch -needs-triage

[github-actions]

Pinging code owners for exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearchexporter: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[carsonip]

/label -stale

[ericywl]

@carsonip What would be the expected resolution here? For the duplicate attribute to be removed when indexing?

[carsonip]

In otel spec,

Resources, Instrumentation Scopes, Metric points, Spans, Span Events, Span Links and Log Records may contain a collection of attributes. The keys in each such collection are unique, i.e. there MUST NOT exist more than one key-value pair with the same key. The enforcement of uniqueness may be performed in a variety of ways as it best fits the limitations of the particular implementation.

I'm inclined to not fix it in ES exporter, as .NET violated the spec and should be seen as a bug in .NET: elastic/elastic-otel-dotnet#263 and open-telemetry/opentelemetry-dotnet#4324

[stevejgordon]

We're working on contributing a fix upstream for the .NET SDK. One issue I can still foresee if customers using any prior version across their estate may run into this and lose data without it being obvious. Would an opt-in de-dupe setting be a possibility? Or could that be done with a custom processor, perhaps? I'm just trying to think of a workaround we can also offer at the collector.