Guest tracing improvements to use tracing crate

[dblnz]

Description

This PR closes #723, #704 and partially addresses #318.
These changes modify the way we perform guest tracing to use the tracing crate and its macros (instrument, trace).

How it works

Guest

What makes this possible is the implementation of the Subscriber trait in the hyperlight-guest-tracing crate. By implementing it, we can now handle the capturing of spans and events and choose how to store them and when to export them to the host.

The GuestSubscriber type that implements Subscriber keeps an internal TraceState that holds all the needed information.
Whenever a new span is created, entered or exited, a callback on the subscriber is called so that we can handle the functionality. The same happens with the events also.

Each time a new span or event is added to the internal state, we check whether the buffer got full and send them to the host to process.

Host

When the host detects a VM exit from the guest, it checks whether it contains tracing information in the OutB instruction.
When tracing information is found, the host starts going through it and check against the local storage of spans.

• If the spans have previously been created, just update the end timestamp (if present) and add new events (if any).
• If they haven't been created, create and store them.

The spans parents are set based on the information got from the host.

TODO

• The current issue is with the guest calls that end up calling back into the host.
  These do not correctly set the parents of the spans created in the host to the last one created in the guest before doing the VM exit
  I need to find a way to propagate the context into the guest and back whenever it is needed. But using the Opentelemetry propagators needs std support which we do not have in the guest.
• There is a corner case with calculating the timestamp for the span that is open when a VM exit is done

Jaeger picture of a Guest call that calls back into the host

[jprendes]

I made a high level review, and what I've seen looks good :-)

[ludfjig]

First round review looks good to me. I am curious what the performance looks like with tracing vs without, maybe we could add a benchmark or something for this?

Also, is there any possibility that we don't flush spans/records after exiting the guest, and that some end up not being emitted?

Another thing to consider is log crate vs tracing crate. Should we ditch one? Or is there some mechanism that allows regular logs to be consumed by tracing crate: And should we expose any of these in guest C-api?

Also is the tracing buffer sizes configurable? Maybe it should be if it isn't, so users can tweak it in case it affects performance.

[jsturtevant]

I am curious what the performance looks like with tracing vs without, maybe we could add a benchmark or something for this?

+1

[dblnz]

First round review looks good to me. I am curious what the performance looks like with tracing vs without, maybe we could add a benchmark or something for this?

Ok, I can do that.

Also, is there any possibility that we don't flush spans/records after exiting the guest, and that some end up not being emitted?

Hmm, in my limited testing I haven't seen this case, but I wouldn't exclude the possibility.
One scenario that comes to mind is cancellation, where the vCPU forced to stop, so no out instruction to deliver the info.
Any ideas how we can treat these scenarios?

Another thing to consider is log crate vs tracing crate. Should we ditch one? Or is there some mechanism that allows regular logs to be consumed by tracing crate: And should we expose any of these in guest C-api?

I am not sure about the best approach is.
The solution I added for guest spans/events only works with opentelemetry Subscribers on the host, so if no opentelemetry subscribers, the they won't be captured (haven't tried it yet, but this is how it should work)

Also is the tracing buffer sizes configurable? Maybe it should be if it isn't, so users can tweak it in case it affects performance.

The tracing buffer is compile time configurable, which I agree is not ideal for customers.
But I needed a fixed size buffer so that I could reliably give the pointer to the host to access memory (current approach, I don't know if it is the best, but it is certainly faster than using the input/output buffer which copies data twice. This way we only copy once, on the host).

[dblnz]

First round review looks good to me. I am curious what the performance looks like with tracing vs without, maybe we could add a benchmark or something for this?

I've run some benchmarks locally and here are the results.
Relative to the work done in the dummy guest, the tracing logic could look like it takes a lot, but we need a realistic guest scenario to correctly assess how relevant the numbers are.

Runtime	Strategy	Flavour	RPS	p50 (s)	p95 (s)	p99 (s)	p99.99 (s)	Peak RSS (MB)
hyperlight-dummy	new		69.96	0.7065	1.1569	1.2509	1.3412	12.79
hyperlight-dummy-tracing	new	nolog	70.92	0.6728	1.1247	1.2065	1.3029	20.21
hyperlight-dummy-tracing	new	log	70.14	0.7354	1.0132	1.0953	1.1922	18.54
hyperlight-dummy	reload		93506.17	0.0004	0.0011	0.0018	0.0067	15.00
hyperlight-dummy-tracing	reload	nolog	56152.01	0.0006	0.0023	0.0049	0.0575	23.42
hyperlight-dummy-tracing	reload	log	38585.95	0.0009	0.0035	0.0069	0.0507	21.90
hyperlight-dummy	reuse		80821.05	0.0005	0.0016	0.0026	0.0065	14.85
hyperlight-dummy-tracing	reuse	nolog	54745.61	0.0007	0.0020	0.0049	0.0581	23.28
hyperlight-dummy-tracing	reuse	log	41301.24	0.0009	0.0028	0.0056	0.1097	21.24

Runtimes:

• hyperlight-dummy - Hyperlight Sandbox running a dummy guest (no TracingProvider)
• hyperlight-dummy-tracing - Hyperlight Sandbox running a dummy guest with trace_guest enabled on both guest and host and a TracingProvider instantiated
  • nolog - the max log level is none, which means the guest doesn't send tracing info, but the Host logic to handle them is enabled
  • log - the max log level is trace for the guest, which makes the guest send tracing info and the host to handle them

Some thoughts:

• This feature started as a development phase improvement to help figure out where we have bottlenecks on the guest.
  This is a huge improvement over the previous solution we had.
• At this point, I do not think this should be used in production by default because the performance is not ideal. It could be useful on an error path, when something doesn't work as expected, it could be turned on
• These numbers do not perfectly reflect only the tracing logic comparison because there are multiple variables such as: the communication with the Jaeger Collector to send the tracing data (when the TracingProvider is installed).
• Some user feedback would be perfect
• This would benefit from other generic performance improvements, such as (sharing memory with the guest in such a way that doesn't imply copying memory - guest immutable references).
  I think we should first discuss where we stand in terms of these big changes in Hyperlight, which may also positively impact the tracing performance.