TestGenius

TestGenius is an AI agent that leverages the Google Gemini API to automatically generate comprehensive end-to-end test cases and their corresponding executable test scripts for both API and User Interface (UI) functionalities.

Features

• AI-Powered Test Case Generation: Uses the Gemini API to generate detailed test cases from API specifications (OpenAPI/Swagger JSON) or UI descriptions (user stories, key elements).
• Executable Test Scripts: Creates Python test scripts.
  • API Testing: Uses the requests library.
  • UI Testing: Uses the playwright library (targeting Chromium by default, headless).
• Pytest Framework: All generated scripts are designed to be run with pytest.
• CLI Interface: Provides a command-line interface to specify inputs and generate tests.
• Configurable: Supports environment-specific configurations via .env files (e.g., for API keys, base URLs).
• Modular Design: Developed with a focus on modularity and maintainability.

Project Structure

.
├── AGENTS.md              # Instructions for AI development assistance
├── examples/              # Example inputs (OpenAPI specs, UI descriptions) and outputs
├── requirements.txt       # Project dependencies
├── testgenius/            # Main application package
│   ├── __init__.py
│   ├── cli/               # Command-line interface logic (Typer)
│   │   ├── __init__.py
│   │   └── main.py
│   ├── core/              # Core logic (Gemini interaction, test case models, config)
│   │   ├── __init__.py
│   │   ├── config.py
│   │   ├── models.py
│   │   └── test_case_generator.py
│   ├── generators/        # Test script generation logic
│   │   ├── __init__.py
│   │   ├── api_test_generator.py
│   │   └── ui_test_generator.py
│   ├── templates/         # Placeholders for script templates (currently not extensively used)
│   │   └── __init__.py
│   └── utils/             # Common utility functions
│       ├── __init__.py
│       └── common.py
├── tests/                 # Pytest tests for TestGenius itself
│   ├── __init__.py
│   ├── core/
│   │   ├── __init__.py
│   │   └── test_config.py
│   └── utils/
│       ├── __init__.py
│       └── test_common.py
└── README.md

Setup & Installation

1. Clone the repository (if applicable):

   git clone <repository_url>
   cd testgenius-project-directory

2. Create a virtual environment (recommended):

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Install Playwright browsers (for UI testing): The generated UI tests use Playwright. You'll need to install browser drivers, at least Chromium for the default:

   playwright install chromium
   # Or install all: playwright install

5. Set up Environment Variables: TestGenius requires a Google Gemini API key. Create a .env file in the project root directory:

   GEMINI_API_KEY=your_actual_google_gemini_api_key_here
   
   # Optional: Default target URLs for generated tests if not overridden by CLI options
   # TESTGENIUS_TARGET_BASE_URL=http://default-api.example.com/api
   # TESTGENIUS_TARGET_APP_URL=http://default-app.example.com
   

   Replace your_actual_google_gemini_api_key_here with your valid API key.

Usage

TestGenius is operated via its command-line interface.

General Help

python -m testgenius.cli.main --help
# or if testgenius is installed as a package:
# testgenius --help

(Note: To run with python -m testgenius.cli.main, ensure your Python path is set up correctly or run from the project's root directory.)

Generating API Tests

python -m testgenius.cli.main generate-api-tests [OPTIONS]

Key Options:

• --spec FILE_PATH or -s FILE_PATH: Path to your OpenAPI JSON specification file. If not provided, TestGenius will use internal placeholder data to demonstrate test case generation (useful for testing TestGenius itself without a live API spec).
• --output-dir DIRECTORY_PATH or -o DIRECTORY_PATH: Directory where the generated API test scripts will be saved. (Default: generated_tests/api)
• --base-url URL: Override the base URL that the generated API test scripts will target. If not provided, it might use TESTGENIUS_TARGET_BASE_URL from your .env or a hardcoded default.

Example:

python -m testgenius.cli.main generate-api-tests -s ./examples/openapi_petstore.json -o ./my_api_tests --base-url "https://petstore.swagger.io/v2"

Generating UI Tests

python -m testgenius.cli.main generate-ui-tests [OPTIONS]

Key Options:

• --url APP_BASE_URL or -u APP_BASE_URL: Required. The base URL of the web application you want to test. This URL is used to inform Gemini.
• --story USER_STORY or -S USER_STORY: Required. A user story or a high-level description of the feature to be tested (e.g., "As a user, I can log in with valid credentials.").
• --elements "element1,element2" or -e "element1,element2": Comma-separated list of key UI elements involved in the story (e.g., "Login button,Username input field").
• --flow "detailed flow description" or -f "detailed flow description": A more detailed description of the expected user flow and outcome.
• --output-dir DIRECTORY_PATH or -o DIRECTORY_PATH: Directory where the generated UI test scripts will be saved. (Default: generated_tests/ui)
• --app-base-url URL: Override the application base URL that the generated UI test scripts will target. This is useful if the URL used for test execution is different from the one provided to Gemini (e.g., local dev vs. staging). If not set, it defaults to the value of --url.

Example:

python -m testgenius.cli.main generate-ui-tests \
    -u "https://www.example.com" \
    -S "User registration and login" \
    -e "Username field,Password field,Submit button,Registration link" \
    -f "User clicks registration, fills form, submits, then tries to log in." \
    -o ./my_ui_tests \
    --app-base-url "http://localhost:3000"

Running Generated Tests

Generated tests are standard pytest files.

1. Navigate to the output directory (or a directory from which pytest can discover the tests).
2. Ensure you have pytest, requests (for API tests), and playwright (for UI tests) installed in your environment.
3. Run pytest:

   pytest

   You might need to specify the directory if running from elsewhere:

   pytest your_output_directory/

AGENTS.md File

This repository includes an AGENTS.md file. This file contains specific instructions and guidelines for AI agents (like Jules) assisting in the development of TestGenius. It helps ensure consistency and adherence to project standards when AI contributes to the codebase.

Future Enhancements

• OAuth Handling for API Tests: Implement robust OAuth (client credentials, authorization code flows) token generation and management within generated API tests.
• Data-Driven Testing:
  • Allow parameterization of generated tests using pytest.mark.parametrize.
  • Support reading test data from external files (CSV, JSON).
• Enhanced Test Reporting: Integrate with reporting tools like pytest-html or Allure for better visualization of test results.
• Advanced UI Interactions: Support for more complex Playwright actions (drag and drop, file uploads, handling iframes, shadow DOM).
• Page Object Model (POM) for UI Tests: Automatically generate or scaffold page objects for UI tests to improve maintainability.
• Configuration for Gemini Model: Allow selection of different Gemini models or parameters.
• More Sophisticated OpenAPI Parsing: Deeper analysis of OpenAPI specs to extract more context for test generation.
• Refined Error Handling and Logging: More granular error reporting and logging throughout TestGenius.
• Comprehensive Self-Testing: Expand the test suite for TestGenius itself, including integration tests.
• Web Interface: A basic web UI for interacting with TestGenius as an alternative to the CLI.