feat(replay): add feedback to LLM context

[michellewzhang]

closes https://linear.app/getsentry/issue/REPLAY-418/add-error-and-user-feedback-context-to-llm

adds feedback context to the LLM input for AI breadcrumbs. we do this somewhat differently from how we fetch errors for error context; feedback breadcrumbs are readily available in the rrweb payloads, so we can get the feedback (event) ID from the breadcrumb. from there, we fetch the feedback event and use the info in the event (timestamp, message) to populate the log message.

[michellewzhang]

@sentry review

[seer-by-sentry]

On it! We are reviewing the PR and will provide feedback shortly.

[seer-by-sentry]

PR Description

This pull request introduces the ability to include user feedback events in the AI-powered summaries of replay breadcrumbs. This allows the AI to consider user-reported issues and positive feedback when generating a summary of a user's session, providing a more complete picture of the user experience.

Click to see more

Key Technical Changes

1. Renamed ErrorEvent to GroupEvent: This change broadens the type to encompass both error and feedback events, promoting code reuse and clarity.
2. Implemented fetch_feedback_details: This function retrieves user feedback event details from nodestore, similar to how error details are fetched.
3. Updated as_log_message: This function now handles EventType.FEEDBACK events, extracting the feedback message and formatting it into a log message suitable for AI consumption.
4. Modified get_request_data and gen_request_data: These functions were updated to accept and process both error and feedback events, ensuring chronological ordering.
5. Added FEEDBACK event type to EventType enum: This new enum value represents user feedback events.
6. Added comprehensive tests: New tests were added to verify the correct handling of feedback events, including feature flag checks and integration with the AI summarization endpoint.

Architecture Decisions

The decision to reuse the GroupEvent type and existing data fetching patterns for feedback events promotes consistency and reduces code duplication. The as_log_message function is extended to handle the new event type, centralizing the logic for converting events into AI-readable log messages. The use of nodestore for retrieving feedback events aligns with the existing architecture for event storage.

Dependencies and Interactions

This pull request depends on the sentry and nodestore services for event storage and retrieval. It interacts with the Seer AI summarization endpoint to generate the final summary. The changes are gated by feature flags (organizations:session-replay, organizations:replay-ai-summaries, and organizations:gen-ai-features) to allow for controlled rollout and experimentation.

Risk Considerations

1. Performance impact: Fetching feedback details from nodestore adds an additional data retrieval step, which could impact the overall performance of the summarization process. Performance testing should be conducted to ensure acceptable latency.
2. Data security: The feedback messages may contain sensitive user information. Appropriate measures should be taken to ensure data privacy and security.
3. AI interpretation: The AI model needs to be trained to effectively interpret and incorporate feedback messages into the summary. Insufficient training could lead to inaccurate or misleading summaries.
4. Error handling: The code includes basic error handling, but more robust error handling and logging may be needed to ensure resilience in production.

Notable Implementation Details

The fetch_feedback_details function returns None if project_id is None. The as_log_message function includes a try-except block to handle potential KeyError exceptions when parsing feedback events. The tests include checks for feature flag enablement and the presence of feedback messages in the generated logs.

[michellewzhang]

bugbot run

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ KeyError: 'to' in prod

_{Did you find this useful? React with a 👍 or 👎}