Skip to content

IMPLEMENTATION.md

Latest commit

 

History

History
234 lines (176 loc) · 12.1 KB

IMPLEMENTATION.md

File metadata and controls

234 lines (176 loc) · 12.1 KB

Travel App Implementation

This document provides a comprehensive overview of the architecture, purpose, and implementation of the Travel App example.

Purpose

The Travel App is a demonstration of a generative UI application built using the flutter_genui package. It functions as a conversational travel agent assistant. Instead of a static, predefined user interface, the app's UI is dynamically generated by a large language model (LLM) in response to the user's prompts. This allows for a highly flexible and context-aware user experience, where the UI adapts to the flow of the conversation.

Architecture

The application is structured into several distinct layers, each with a specific responsibility. The following diagram illustrates the high-level architecture and the relationships between the main components.

classDiagram
    direction TB

    namespace TravelApp {
        class TravelPlannerPage {
            <<StatefulWidget>>
            +build()
            - _genUiManager
            - _aiClient
            - _conversation
            - _triggerInference()
        }
        class ConversationWidget {
            <<Widget>>
        }
        class GenUiSurface {
            <<Widget>>
        }
    }

    namespace flutter_genui {
        class GenUiManager {
            <<Component>>
            +getTools()
            +surfaceUpdates
        }
        class AiClient {
            <<Component>>
            +generateContent()
        }
        class WidgetCatalog {
            <<Component>>
        }
    }

    namespace Backend {
        class LLM {
            <<System>>
        }
    }

    TravelPlannerPage --o flutter_genui.GenUiManager : "Initializes and holds"
    TravelPlannerPage --o flutter_genui.AiClient : "Initializes and holds"
    TravelPlannerPage --> ConversationWidget : "Builds"
    ConversationWidget --> GenUiSurface : "Renders AI UI messages"

    TravelPlannerPage --> flutter_genui.AiClient : "Calls generateContent()"

    flutter_genui.GenUiManager --> TravelPlannerPage : "Notifies of updates via Stream"
    GenUiSurface --> flutter_genui.WidgetCatalog : "Uses to build widgets"

    flutter_genui.AiClient --> Backend.LLM : "Communicates with"
Loading

1. UI Layer (main.dart)

This is the main entry point for the application. Its responsibilities are:

  • Initialization: It initializes Firebase and loads the asset image catalog.
  • UI Scaffolding: It sets up the root MaterialApp and a Scaffold with a TabBar. The app has two main tabs:
    • "Travel": This tab contains the TravelPlannerPage, which is the primary interface for the conversational travel agent.
    • "Widget Catalog": This tab displays a CatalogView from the flutter_genui_dev package, allowing developers to browse and inspect all the available UI components in the app's catalog.

2. App Logic (lib/src/travel_planner_page.dart)

The core application logic resides in the TravelPlannerPage widget. Its responsibilities are:

  • Initialization: It initializes the GenUiManager and the AiClient.
  • State Management: It manages the _conversation history list, sends prompts to the AI via _triggerInference, and handles UI events returned from the GenUiSurface widgets.
  • UI Rendering: It provides the structure for the chat interface, which consists of a Conversation widget to render the conversation history and a custom chat input field.

3. UI State Management Layer (package:flutter_genui)

The components from the flutter_genui package are the central orchestrators of the dynamic UI state.

  • GenUiManager: The core state manager. It maintains the definitions for all active UI "surfaces", provides the UI manipulation tools (addOrUpdateSurface, deleteSurface) to the app logic, and notifies listening widgets (like TravelPlannerPage) of changes via a stream.
  • Conversation widget: A widget defined in the app (lib/src/widgets/conversation.dart) that renders a list of ChatMessages. It is responsible for displaying the conversation history, including text messages and dynamically rendered UI surfaces via GenUiSurface.

4. AI/Model Layer (package:flutter_genui_firebase_ai)

The FirebaseAiClient class, from the flutter_genui_firebase_ai package, abstracts the communication with the underlying generative AI model (a Google Gemini model via Firebase). It handles the API calls to the model, sending prompts and receiving the model's responses, including the tool calls that drive the UI generation.

5. Widget Catalog (The "Tools")

This is the collection of predefined UI components that the AI can use to construct the interface. It acts as the "API" that the AI targets.

  • Definition: The catalog is defined in lib/src/catalog.dart as a Catalog instance, which is a list of CatalogItems.
  • Custom Components: The travel app defines several custom CatalogItems in the lib/src/catalog/ directory. Key components include:
    • TravelCarousel: For displaying a horizontal list of selectable options.
    • ItineraryWithDetails, ItineraryDay, ItineraryEntry: For building structured travel plans.
    • InputGroup: A container for grouping various input widgets.
    • OptionsFilterChipInput, CheckboxFilterChipsInput, TextInputChip, DateInputChip: Different types of input chips for user selections.
    • InformationCard: For displaying detailed information about a topic.
    • Trailhead: For suggesting follow-up prompts to the user.
    • ListingsBooker: For displaying a list of bookings to checkout.
    • PaddedBodyText: For displaying a block of text with padding.
    • SectionHeader: For displaying a section header.
    • TabbedSections: For displaying a set of tabbed sections.
  • Standard Components: It also uses standard, pre-built components from flutter_genui like column, text, image, etc.

Data Flow: The Generative UI Cycle

The diagram below shows the sequence of events from user input to UI rendering. The application logic in lib/src/travel_planner_page.dart is responsible for driving this cycle.

sequenceDiagram
    actor User
    participant AppLogic as "App Logic (TravelPlannerPage)"
    participant AiClient
    participant LLM
    participant GenUiManager
    participant ConversationWidget

    activate User
    User->>AppLogic: Enters prompt "Find flights to Paris"
    deactivate User

    activate AppLogic
    AppLogic->>AppLogic: Adds UserMessage to _conversation list
    AppLogic->>AiClient: generateContent(_conversation, uiTools)
    activate AiClient
    AiClient->>LLM: Sends prompt and tool schemas
    activate LLM
    LLM-->>AiClient: Responds with tool call (e.g. addOrUpdateSurface)
    deactivate LLM
    AiClient-->>AppLogic: Returns result (AI tool calls are executed internally)
    deactivate AiClient

    note right of AppLogic: The AiClient invokes the tool, which calls a method on GenUiManager.

    activate GenUiManager
    GenUiManager->>GenUiManager: Updates internal state
    GenUiManager-->>AppLogic: Notifies of state change via Stream
    deactivate GenUiManager

    AppLogic->>AppLogic: Updates _conversation list with AiUiMessage

    activate ConversationWidget
    AppLogic->>ConversationWidget: Rebuilds with new message list
    ConversationWidget->>ConversationWidget: Renders new UI surface
    ConversationWidget-->>User: Displays new UI
    deactivate ConversationWidget
    deactivate AppLogic
Loading

Implementation Details

The System Prompt

The interaction with the model is heavily guided by a detailed system prompt defined in lib/src/travel_planner_page.dart. This prompt is critical to the application's success. It instructs the model on:

  • Persona: To act as a helpful travel agent.
  • Conversation Flow: To first ask clarifying questions and then present results.
  • Tool Usage: Which widgets (tools) to use in different situations (e.g., use travel_carousel for options, itinerary_with_details for final results).
  • UI Style: How to compose widgets for a good user experience (e.g., breaking up itineraries with other content).
  • Available Data: It includes a JSON string (_imagesJson) containing a list of available asset image paths and descriptions, ensuring the model uses relevant and high-quality images from the application's assets.

The Widget Catalog

The catalog is the cornerstone of the dynamic UI generation. It's a registry of all possible UI components the model can render.

Each component in the catalog is a CatalogItem with three key parts:

  1. name: A unique string that the AI uses to identify the widget (e.g., 'travel_carousel').
  2. dataSchema: A Schema object that defines the structure and types of data the widget expects. This schema is provided to the model so it knows what parameters to generate for the tool call. The descriptions within the schema are crucial for the model to understand the purpose of each parameter.
  3. widgetBuilder: A Dart function that takes the data generated by the model (as a Map<String, Object?>) and returns a Flutter Widget.

Catalog Item Definition

Each custom widget in lib/src/catalog/ follows a consistent pattern:

  1. Schema Definition: A _schema variable defines the data structure using S.object, S.string, S.list, etc.
  2. Data Accessor: A Dart extension type is defined to provide type-safe access to the Map<String, Object?> data received from the model. This avoids manual casting and error-prone map access.
  3. CatalogItem Instance: A final variable creates the CatalogItem, passing the name, dataSchema, and widgetBuilder.
  4. Widget Implementation: A standard Flutter StatelessWidget or StatefulWidget that contains the actual UI code for the component.

Dynamic UI Composition

Widgets can be nested to create complex layouts. This is achieved by having a widget's schema accept the ID of another widget as a parameter (e.g., the child property of itinerary_with_details).

The widgetBuilder for a container-like widget receives a buildChild function as an argument. It can call buildChild(childId) to recursively ask the GenUiSurface to build and return the child widget, which is then placed in the parent's widget tree.

graph TD
    subgraph "UI Definition from LLM"
        RootDef["root: 'itinerary_1'"]
        WidgetsDef["widgets: [...]"]
    end

    subgraph "Rendered UI"
        A(GenUiSurface) -- Renders --> B(ItineraryWithDetails Widget)
        B -- contains --> C(Column Widget)
        C -- contains --> D(Text Widget)
        C -- contains --> E(TravelCarousel Widget)
    end

    subgraph "Recursive Build Process"
        F["GenUiSurface.build<br>('itinerary_1')"] -->|Calls builder for Itinerary| G["ItineraryBuilder"]
        G -->|"calls<br>buildChild('column_1')"| H["GenUiSurface.build<br>('column_1')"]
        H -->|calls builder<br>for Column| I["ColumnBuilder"]
        I -->|"calls<br>buildChild('text_1')"| J["...builds Text"]
        I -->|"calls<br>buildChild('carousel_1')"| K["...builds TravelCarousel"]
    end

    RootDef --> F
    WidgetsDef --> F

    %% Add links from build process to rendered UI
    F -- "creates" --> B
    H -- "creates" --> C
    J -- "creates" --> D
    K -- "creates" --> E
Loading

User Interaction and Events

The UI is not just for display; it's interactive. Widgets like InputGroup, OptionsFilterChipInput, and TravelCarousel can capture user input.

  • Event Dispatching: The widgetBuilder for each catalog item receives a dispatchEvent function. This function is called in response to user actions (like a button press or item selection), creating a UiEvent (e.g., UiActionEvent).
  • Event Handling: The TravelPlannerPage listens to the genUiManager.onSubmit stream. When an event is dispatched from a widget, the GenUiManager processes it and emits a UserMessage on this stream.
  • Conversation Update: The TravelPlannerPage's stream listener (_handleUserMessageFromUi) receives the UserMessage, adds it to the conversation history, and triggers a new inference call to the model. This informs the model of the user's actions, allowing it to respond accordingly (e.g., refining a search based on a selected filter).