Does OpenTelemetry need "enum" metric type?

[jsuereth]

How are users expected to encode 'enumerated' values in OpenTelemetry? Could be a gauge w/ string value?

We need to figure this out.

[sirianni]

+1 for adding first-class support for enumerated values (or a String-valued gauge).

I have some internal customers trying to model this in their applications. Currently they are setting the enumerated value as an attribute and setting the metric value to 0 or 1 depending on which enum value is active. One of the downsides of the "value-as-attribute" approach is that it bloats the volume of metrics flowing through the pipeline (in each collection interval, all but one of the metrics have a 0 value). We are getting pushback from the teams that own our metrics sinks and being asked to minimize this volume.

An alternative pattern would be to encode each enumerated value as a numeric integer. This has the downside that the metric is not self-describing and you'd need additional context to decode the integer value in the sink.

In OpenMetrics, enums can be encoded using the more general StateSet type which uses a bit-vector to encode the values. StateSets represent a series of related boolean values, also called a bitset.

If encoded as a StateSet, ENUMs MUST have exactly one Boolean which is true within a MetricPoint.
This is suitable where the enum value changes over time, and the number of States isn't much more than a handful.

[bertysentry]

+1

We're developing a hardware monitoring solution in OpenTelemetry and many of the components report a status that is an enumeration (e.g. simply "OK", "Degraded", or "Failed").

Using the "value-as-attribute" approach is non-sense when the enumeration has many possible values.

A simple implementation would be to map each string value to an integer value and describe this mapping in the unit in a standard way. Example: { 0 = OK; 1 = Degraded; 2 = Failed }

This way the current specification and implementation of metrics don't need to change much and it's the UI's responsibility to translate the integer value into the corresponding string value (very much like it's already the UI's responsibility to convert bytes to GB to display something that the end user can read easily).

[dyladan]

A simple implementation would be to map each string value to an integer value and describe this mapping in the unit in a standard way. Example: { 0 = OK; 1 = Degraded; 2 = Failed }

Keep in mind that the unit field is limited to 63 characters which is extremely limiting for an enum like this. Also, this not-quite-json would require a custom parser if a backend wanted to display it more clearly and a more machine readable true JSON format would be more annoying for users of systems that don't do any parsing.

If we don't want to have a proper enum type, I would argue that a simple counter/gauge/etc with attribute status=OK, status=DEGRADED, or status=FAILED would be a more elegant solution.

[bertysentry]

I agree that the { 0 = OK; 1 = Degraded; 2 = Failed } unit suggestion is more of a workaround than a proper enum implementation, as it defers the implementation of the conversion to the UI, with loose specifications. Also, good point on the 63 chars limit in the Unit field.

Using an Attribute is not ideal either. With a long Enum (say like the very common OperatingStatus property of the CIM_ManagedSystemElement WBEM class), you would need to send:

managed_system_element.operating_status{state="Unknown"} 0.0
managed_system_element.operating_status{state="Not Available"} 0.0
managed_system_element.operating_status{state="Servicing"} 1.0
managed_system_element.operating_status{state="Starting"} 0.0
managed_system_element.operating_status{state="Stopping"} 0.0
managed_system_element.operating_status{state="Stopped"} 0.0
managed_system_element.operating_status{state="Aborted"} 0.0
# and many more
managed_system_element.operating_status{state="Vendor Reserved"} 0.0

...just to say that the managed system element is "Servicing". In terms of storage, memory and network, it's very inefficient.

Also, it allows invalid states where several values are "enabled" at once, while only one is possible at the same time (the service can't be both "Starting" and "Stopping" at the same time.

DMTF's CIM model has had ValueMap from the beginning (see page 89 - ValueMap in CIM specifications).

Heck, SNMP has had enumeration types since for more than a couple decades. See RFC-2579 which defines textual conventions syntax:

INTEGER {
                     other(1),       -- eh?
                     ok(2),    -- everything is fine
                     degraded(3), -- still working, but for how long?
                     failed(4),   -- no longer working
                     dead(5)     -- you won't fix this one
                 }

My conclusion is:

1. We should use an Attribute to identify various states of an enum when they can be combined (like the StateSet in OpenMetrics)
2. We need an Enum value type (where values are mutually exclusive) for metrics in OpenTelemetry that specifies a mapping from int to string, just like SNMP and DMTF's CIC were doing
3. It will be the exporter's responsibility to export the mapping information to the receiving platform (as a unit field, as a description, as a comment, or as a true enum).

Note: In the meantime, assuming OpenTelemetry adds an Enum value type in the future, I would recommend that developers use a single Gauge int metric to represent mutually exclusive enums, as it will be much easier to convert later a single int metric to an Enum.

[pirgeo]

Thanks for elaborating on this. The example you have given above feels a bit more like an Event to me (implying a state transition). Maybe there is a entirely different way of modeling this that does not necessarily involve metrics if performance is critical?

Some considerations regarding using a gauge:

• How would you aggregate these gauges? The UI needs to know that this is not actually a gauge, but an enum, and act accordingly. This actually raises the question of how you would do space aggregation for an enum data type, but I think its applicable here too: if you remove the label distinguishing multiple instances of this metric, what do you show?
• I am not sure, but I think using a gauge for now and later switching to an enum data type (if it is introduced) would break data continuity in OTLP. Once the semantic convention is marked as stable, I don't think we can just change the data type exported in a backwards compatible way.

It will be the exporter's responsibility to export the mapping information to the receiving platform (as a unit field, as a description, as a comment, or as a true enum).

I am not sure I understand this. Are you suggesting that the int-to-string mapping information needs to be transferred with the enum data type during metrics message export? If this is the case, I am not sure if sending the different data points as labels would really increase the size of the message a lot, since we transfer all of the strings anyway (just already separated instead of as a description that needs to be parsed by the backend), and I think adding a few more integers would not really increase the relative size of the message a lot. Also: can the enum definition change over time? What happens if different services report different mappings - which one is given priority?

[bertysentry]

Sorry for the late reply, I did not receive the notification... Thank you for the questions and comments!

The example you have given above feels a bit more like an Event to me

You're right, metrics or events can be used to report "discrete" values (the status of a component for example), they both have pros and cons. In the end, you still need a proper Enum value to represent the current status (metric) or the new status (event).

How would you aggregate these gauges? The UI needs to know that this is not actually a gauge, but an enum, and act accordingly.

Grafana Dashboards already has a "State timeline" panel designed just for that. The mapping in this case needs to be specified in the UI. If the metric itself could indicate the mapping in the future, that would make everybody's life much easier.

The timeseries database storing the metrics doesn't need to handle Enum metrics as a specific type. They can process them just as simple integers. The value mapping could be stored as metadata. Spatial aggregation should prefer max() and min() functions, rather than average().

using a gauge for now and later switching to an enum data type (if it is introduced) would break data continuity in OTLP

True. To avoid this, an enum type could be defined as a subtype of Gauge, like EnumGauge. The only difference between EnumGauge and a simple Gauge is the value mapping (instead of the Unit field). Not sure this is feasible in Go.

Also, breaking data continuity doesn't really happen in OTLP itself, but rather in the exporters. If the Prometheus exporter handled EnumGauge just like Gauge (as numbers), it wouldn't make any difference in Prometheus (no break in data continuity).

The same cannot be said if we have Attributes describing the states. They end up as different labels in Prometheus and thus different timeseries.

It will be the exporter's responsibility to export the mapping information to the receiving platform (as a unit field, as a description, as a comment, or as a true enum).

I am not sure I understand this.

I'm simply saying that it's the exporter's responsibility to adapt the OpenTelemetry metrics to the receiving end. In the example of Prometheus, which doesn't support Enum values, the exporter would simply export the int value, and maybe the mapping in the metric description (in # comments when polling /metrics).

If a timeseries backend supports Enum values natively in the future, the corresponding exporter may have the option to export OTLP's EnumGauge as the corresponding native Enum.

Also: can the enum definition change over time? What happens if different services report different mappings - which one is given priority?

Good question! The value mapping of an Enum must not change over time, very much like the unit of a metric is not supposed to change over time. EnumGauge DataPoints can be send in bulk, without resending the value mapping (just like we don't need to resend the Gauge unit for each DataPoint).

[bertysentry]

As a matter of fact (if I'm not mistaken 😅), Grafana doesn't have any visualization that can display StateSet metrics, where the different states are represented with an Attribute (label). See this question on StackOverflow links to a 3rd-party plugin to do that (but it's not made by Grafana).

I haven't seen anything that could handle this on Datadog either.

[pirgeo]

Thanks for your reply. I think I see your point about the enum-as-gauge (and please correct me if I am wrong):

It seems that using attributes could lead to (potentially unwanted) StateSet semantics, and it leads to having to export zeros for states that the device is currently not in (at least the ones that were already seen before). Using the Gauge is an efficient, but much less user friendly alternative. Also, using a Gauge currently does not allow adding the mapping on the client, since there is no way of transmitting that mapping. One would have to add it on the server side manually (as is the case in Grafana, if I understand correctly). Am I missing something here?

The upside of using attributes would be that you could easily sum the counts from different producers and see stats like "how many disks are in a running state, how many disks fail on average per hour?". This is possible today and would not require any changes to the protocol. I think I am more familiar with this representation, because it better resembles my understanding of metrics (being able to "do math" with them), while this is not the case for the gauge representation.

Either way, I don't really see a way around breaking semantic conventions when introducing an Enum type later. If I am not mistaken, there is no inheritance in Protobuf, so in OTLP the Enum would have to be a separate type. That means that unless the consumer decides to explicitly convert OTLP Enums to whatever internal representation they used before, there will be two separate data streams. I see your point about Prometheus in that regard, and that it should be possible to do that easily there, but I am not sure about a lot of the other consumers.

[bertysentry]

I understand your point regarding the ability to do math with the metrics, where the states described with an attribute is much more user friendly.

Also at this point, I think it is unlikely we will see timeseries server vendors embrace a new Enum type. Even though I think I have a ton of excellent reasons to want this EnumGauge, I'll have to cave in and do with what we have now.

I'll update the hardware-metrics semantic convention to reflect that. Thank you for the discussion!

[jmacd]

There is a lengthy discussion about this topic in #2518 (comment). In that comment, I propose that OTel is better off having string-valued attributes and 0/1-valued UpDownCounters to convey state sets.

With more specification work, it should be possible to avoid transmitting zeros for inactive states. For example, could we define a a data point flag set on the 1-valued attribute set that marks prior values for the same attribute keys as stale?