Skip to content

plugin-system.md

Latest commit

 

History

History
837 lines (670 loc) · 23.2 KB

plugin-system.md

File metadata and controls

837 lines (670 loc) · 23.2 KB
Raw

Ansible Plugin System Architecture

Overview

Ansible's plugin system is the foundation of its extensibility, providing well-defined APIs for extending functionality across 15+ different plugin types. The system combines dynamic loading, polymorphic inheritance, configuration management, and collection-aware distribution.

Plugin Architecture Overview

graph TB
    subgraph "Core Engine"
        Core[Ansible Core]
        Loader[Plugin Loader]
    end
    
    subgraph "Plugin Types"
        Action[Action Plugins]
        Connection[Connection Plugins]
        Strategy[Strategy Plugins]
        Callback[Callback Plugins]
        Filter[Filter Plugins]
        Test[Test Plugins]
        Lookup[Lookup Plugins]
        Vars[Vars Plugins]
        Inventory[Inventory Plugins]
        Cache[Cache Plugins]
        Become[Become Plugins]
        Shell[Shell Plugins]
        Terminal[Terminal Plugins]
        Netconf[Netconf Plugins]
        HttpApi[HttpApi Plugins]
    end
    
    subgraph "Distribution"
        Collections[Collections]
        BuiltIn[Built-in Plugins]
        Community[Community Plugins]
    end
    
    Core --> Loader
    Loader --> Action
    Loader --> Connection
    Loader --> Strategy
    Loader --> Callback
    Loader --> Filter
    Loader --> Test
    Loader --> Lookup
    Loader --> Vars
    Loader --> Inventory
    Loader --> Cache
    Loader --> Become
    Loader --> Shell
    Loader --> Terminal
    Loader --> Netconf
    Loader --> HttpApi
    
    Collections --> Loader
    BuiltIn --> Loader
    Community --> Loader
    
    style Core fill:#e1f5fe
    style Loader fill:#f3e5f5
    style Collections fill:#e8f5e8
Loading

Plugin Loader Architecture

Core Loading System

Location: lib/ansible/plugins/loader.py

The PluginLoader class provides the foundation for all plugin loading operations.

flowchart TD
    Request[Plugin Request] --> Cache{In Cache?}
    Cache -->|Yes| Return[Return Cached Plugin]
    Cache -->|No| Discover[Discover Plugin Paths]
    
    Discover --> BuiltIn[Built-in Paths]
    Discover --> Config[Config Paths]
    Discover --> Collections[Collection Paths]
    Discover --> Extra[Extra Paths]
    
    BuiltIn --> Search[Search for Plugin]
    Config --> Search
    Collections --> Search
    Extra --> Search
    
    Search --> Found{Plugin Found?}
    Found -->|Yes| Load[Load Python Module]
    Found -->|No| NotFound[Plugin Not Found]
    
    Load --> Validate[Validate Plugin Class]
    Validate --> Configure[Configure Plugin]
    Configure --> CacheStore[Store in Cache]
    CacheStore --> Return
    
    style Cache fill:#e8f5e8
    style Return fill:#f3e5f5
    style NotFound fill:#ffebee
Loading

Discovery Algorithm:

def _find_plugin(self, name, suffixes=None, collection_list=None):
    """Multi-phase plugin discovery"""
    
    # Phase 1: Check cache
    if name in self._plugin_path_cache:
        return self._plugin_path_cache[name]
    
    # Phase 2: Search paths in order
    search_paths = self._get_paths()  # Built-in, config, collections, extra
    
    for path in search_paths:
        for suffix in suffixes:
            plugin_path = self._find_plugin_in_path(path, name, suffix)
            if plugin_path:
                return self._load_plugin_from_path(plugin_path)
    
    # Phase 3: Collection-aware search
    if collection_list:
        return self._find_plugin_in_collections(name, collection_list)
    
    return None

Caching Strategy

graph LR
    subgraph "Plugin Loader Caches"
        PathCache[Path Cache<br/>_plugin_path_cache]
        ModuleCache[Module Cache<br/>_module_cache]
        InstanceCache[Instance Cache<br/>_plugin_instance_cache]
    end
    
    subgraph "Cache Keys"
        PathKey[Plugin Name + Suffix]
        ModuleKey[File Path]
        InstanceKey[Plugin Class + Config]
    end
    
    PathKey --> PathCache
    ModuleKey --> ModuleCache
    InstanceKey --> InstanceCache
    
    style PathCache fill:#fff3e0
    style ModuleCache fill:#e8f5e8
    style InstanceCache fill:#f3e5f5
Loading

Cache Levels:

  1. Path Cache: Maps plugin names to file locations
  2. Module Cache: Caches loaded Python modules
  3. Instance Cache: Caches configured plugin instances (for stateless plugins)

Plugin Types and Hierarchies

Action Plugins

Location: lib/ansible/plugins/action/ Base Class: ActionBase

Action plugins execute on the controller before modules run on target hosts.

classDiagram
    class ActionBase {
        +connection: Connection
        +templar: Templar
        +play_context: PlayContext
        +task: Task
        +run(tmp, task_vars) dict
        +_execute_module(module_name, module_args)
        +_configure_module(module_name, module_args)
        +_transfer_file(local_path, remote_path)
    }
    
    class CopyAction {
        +run(tmp, task_vars) dict
        +_copy_file(source, dest)
    }
    
    class TemplateAction {
        +run(tmp, task_vars) dict
        +_template_file(source, dest, task_vars)
    }
    
    class CommandAction {
        +run(tmp, task_vars) dict
        +_build_command(command_args)
    }
    
    ActionBase <|-- CopyAction
    ActionBase <|-- TemplateAction  
    ActionBase <|-- CommandAction
Loading

Integration Points:

  • Connection Management: Direct access to connection plugins
  • Module Execution: Configures and executes modules on targets
  • File Transfer: Handles put/fetch operations with permission management
  • Template Processing: Resolves variables and templates task arguments

Connection Plugins

Location: lib/ansible/plugins/connection/ Base Class: ConnectionBase

Connection plugins manage transport to target systems.

classDiagram
    class ConnectionBase {
        <<abstract>>
        +_connect() Connection
        +exec_command(cmd, in_data, sudoable) tuple
        +put_file(in_path, out_path)
        +fetch_file(in_path, out_path)
        +close()
    }
    
    class SSHConnection {
        +ssh_executable: str
        +control_path: str
        +_connect() SSHConnection
        +exec_command(cmd, in_data, sudoable) tuple
        +_build_command(binary, *args) list
    }
    
    class LocalConnection {
        +_connect() LocalConnection
        +exec_command(cmd, in_data, sudoable) tuple
        +put_file(in_path, out_path)
        +fetch_file(in_path, out_path)
    }
    
    class WinRMConnection {
        +endpoint: str
        +transport: str
        +_connect() WinRMConnection
        +exec_command(cmd, in_data, sudoable) tuple
    }
    
    ConnectionBase <|-- SSHConnection
    ConnectionBase <|-- LocalConnection
    ConnectionBase <|-- WinRMConnection
Loading

Advanced Features:

  • Persistent Connections: Socket-based connection reuse
  • Pipelining: Optimized execution for compatible connections
  • Network Connections: Specialized base classes for network devices
  • Become Integration: Privilege escalation support

Strategy Plugins

Location: lib/ansible/plugins/strategy/ Base Class: StrategyBase

Strategy plugins control task execution flow and parallelization.

stateDiagram-v2
    [*] --> Initialize
    Initialize --> LoadStrategy
    LoadStrategy --> CreateIterator
    CreateIterator --> QueueTasks
    
    state QueueTasks {
        [*] --> GetNextTask
        GetNextTask --> AssignWorker
        AssignWorker --> ExecuteTask
        ExecuteTask --> ProcessResult
        ProcessResult --> GetNextTask
        ProcessResult --> [*]: No More Tasks
    }
    
    QueueTasks --> HandlersPhase
    HandlersPhase --> Complete
    Complete --> [*]
Loading

Strategy Implementations:

  • Linear: Lockstep execution across all hosts
  • Free: Independent host execution with load balancing
  • Debug: Interactive debugging with user prompts

Callback Plugins

Location: lib/ansible/plugins/callback/ Base Class: CallbackBase

Callback plugins handle execution events and output formatting.

sequenceDiagram
    participant Engine as Execution Engine
    participant Manager as Callback Manager
    participant Plugin1 as Default Callback
    participant Plugin2 as JSON Callback
    participant Plugin3 as Custom Callback
    
    Engine->>Manager: v2_runner_on_ok(result)
    Manager->>Plugin1: v2_runner_on_ok(result)
    Manager->>Plugin2: v2_runner_on_ok(result)
    Manager->>Plugin3: v2_runner_on_ok(result)
    
    Plugin1-->>Manager: Console Output
    Plugin2-->>Manager: JSON Logging
    Plugin3-->>Manager: Custom Processing
Loading

Event Types:

  • Runner Events: Task execution (ok, failed, skipped, retry)
  • Play Events: Play start/end, stats
  • Playbook Events: Playbook start/end
  • Handler Events: Handler execution and notification

Filter and Test Plugins

Location: lib/ansible/plugins/filter/, lib/ansible/plugins/test/

Extend Jinja2 templating with custom functions.

graph LR
    Template[Jinja2 Template] --> FilterLoader[Filter Loader]
    Template --> TestLoader[Test Loader]
    
    FilterLoader --> BuiltInFilters[Built-in Filters<br/>to_yaml, regex_replace]
    FilterLoader --> CustomFilters[Custom Filters<br/>my_custom_filter]
    
    TestLoader --> BuiltInTests[Built-in Tests<br/>defined, match]
    TestLoader --> CustomTests[Custom Tests<br/>my_custom_test]
    
    BuiltInFilters --> Render[Template Rendering]
    CustomFilters --> Render
    BuiltInTests --> Render
    CustomTests --> Render
    
    style FilterLoader fill:#e8f5e8
    style TestLoader fill:#f3e5f5
Loading

Plugin Structure:

class FilterModule:
    def filters(self):
        return {
            'my_filter': self.my_filter,
            'another_filter': self.another_filter
        }
    
    def my_filter(self, value, arg1, arg2=None):
        # Filter implementation
        return processed_value

Lookup Plugins

Location: lib/ansible/plugins/lookup/ Base Class: LookupBase

Retrieve data from external sources during templating.

flowchart TD
    Template[Template Rendering] --> LookupCall[lookup('plugin_name', args)]
    LookupCall --> LoadPlugin[Load Lookup Plugin]
    LoadPlugin --> RunMethod[plugin.run(terms, variables)]
    RunMethod --> ExternalSource[External Data Source]
    
    subgraph "Data Sources"
        File[Files]
        ENV[Environment Variables]
        DNS[DNS Records]
        API[REST APIs]
        DB[Databases]
        Vault[HashiCorp Vault]
    end
    
    ExternalSource --> File
    ExternalSource --> ENV
    ExternalSource --> DNS
    ExternalSource --> API
    ExternalSource --> DB
    ExternalSource --> Vault
    
    ExternalSource --> ReturnData[Return Data]
    ReturnData --> Template
    
    style ExternalSource fill:#fff3e0
    style ReturnData fill:#e8f5e8
Loading

Common Patterns:

class LookupModule(LookupBase):
    def run(self, terms, variables=None, **kwargs):
        results = []
        for term in terms:
            # Fetch data from external source
            data = self._fetch_data(term)
            results.append(data)
        return results

Vars Plugins

Location: lib/ansible/plugins/vars/ Base Class: BaseVarsPlugin

Load variables from external sources.

graph TB
    VarManager[Variable Manager] --> VarsPlugins[Vars Plugins]
    
    subgraph "Plugin Types"
        HostGroup[host_group_vars]
        Advanced[advanced_host_list]
        Auto[auto_inventory]
        Custom[Custom Vars Plugins]
    end
    
    VarsPlugins --> HostGroup
    VarsPlugins --> Advanced
    VarsPlugins --> Auto
    VarsPlugins --> Custom
    
    subgraph "Variable Sources"
        Files[YAML/JSON Files]
        Directories[Variable Directories]
        External[External APIs]
        Computed[Computed Variables]
    end
    
    HostGroup --> Files
    HostGroup --> Directories
    Advanced --> External
    Auto --> Computed
    Custom --> Files
    Custom --> External
    
    style VarManager fill:#e1f5fe
    style Files fill:#e8f5e8
    style External fill:#fff3e0
Loading

Loading Stages:

  • Inventory Stage: Load variables during inventory construction
  • Task Stage: Load variables during task execution
  • All Stages: Load variables at both stages

Inventory Plugins

Location: lib/ansible/plugins/inventory/ Base Class: BaseInventoryPlugin

Dynamic inventory loading from various sources.

flowchart LR
    InventoryManager[Inventory Manager] --> Verify[verify_file()]
    Verify --> CanHandle{Can Handle?}
    CanHandle -->|Yes| Parse[parse(inventory, loader, path)]
    CanHandle -->|No| NextPlugin[Try Next Plugin]
    
    Parse --> PopulateInventory[Populate Inventory]
    PopulateInventory --> AddHosts[Add Hosts]
    PopulateInventory --> AddGroups[Add Groups]
    PopulateInventory --> SetVars[Set Variables]
    
    AddHosts --> InventoryData[Inventory Data]
    AddGroups --> InventoryData
    SetVars --> InventoryData
    
    style InventoryManager fill:#e1f5fe
    style InventoryData fill:#e8f5e8
Loading

Plugin Examples:

  • YAML: Static YAML inventory files
  • Constructed: Dynamic groups based on variables
  • Script: Legacy script-based inventory
  • AWS EC2: Dynamic AWS instance inventory
  • VMware: VMware vSphere inventory

Collection Integration

Collection Plugin Distribution

graph TB
    subgraph "Collection Structure"
        Collection[ansible_collections/namespace/collection]
        Plugins[plugins/]
        Action[plugins/action/]
        Connection[plugins/connection/]
        Modules[plugins/modules/]
        Filter[plugins/filter/]
        Other[plugins/...]
    end
    
    subgraph "Plugin Loading"
        Loader[Plugin Loader]
        FQCN[Fully Qualified Collection Name]
        Resolver[Collection Resolver]
    end
    
    Collection --> Plugins
    Plugins --> Action
    Plugins --> Connection
    Plugins --> Modules
    Plugins --> Filter
    Plugins --> Other
    
    Loader --> FQCN
    FQCN --> Resolver
    Resolver --> Collection
    
    style Collection fill:#e8f5e8
    style Loader fill:#f3e5f5
Loading

FQCN Resolution:

def _find_fq_plugin(self, fq_name, extension, plugin_load_context):
    """Resolve fully-qualified collection names"""
    # Example: community.general.setup -> 
    # ansible_collections.community.general.plugins.action.setup
    
    acr = AnsibleCollectionRef.from_fqcr(fq_name, self.type)
    collection_pkg = f'ansible_collections.{acr.collection}'
    plugin_pkg = f'{collection_pkg}.plugins.{self.subdir}'
    
    return self._load_plugin_from_package(plugin_pkg, acr.resource)

Plugin Metadata and Configuration

# Collection metadata (galaxy.yml)
namespace: community
name: general
version: 1.0.0
dependencies:
  ansible.posix: ">=1.0.0"

# Plugin metadata (plugin docstring)
DOCUMENTATION = '''
---
module: my_module
short_description: Example module
options:
  name:
    description: Name parameter
    type: str
    required: true
'''

Plugin Configuration System

Configuration Hierarchy

graph TD
    PluginConfig[Plugin Configuration] --> TaskLevel[Task Parameters]
    PluginConfig --> VarLevel[Variable Configuration]
    PluginConfig --> EnvLevel[Environment Variables]
    PluginConfig --> ConfigFile[Configuration Files]
    PluginConfig --> Defaults[Plugin Defaults]
    
    TaskLevel --> Merge[Configuration Merge]
    VarLevel --> Merge
    EnvLevel --> Merge
    ConfigFile --> Merge
    Defaults --> Merge
    
    Merge --> FinalConfig[Final Plugin Configuration]
    
    style TaskLevel fill:#ffebee
    style FinalConfig fill:#e8f5e8
Loading

Configuration Resolution:

def set_options(self, task_keys=None, var_options=None, direct=None):
    """Multi-source configuration resolution"""
    
    # Start with plugin defaults
    options = self.get_option_defaults()
    
    # Apply configuration file settings
    config_options = C.config.get_plugin_options(self.plugin_type, self._load_name)
    options.update(config_options)
    
    # Apply environment variables
    env_options = self._get_env_options()
    options.update(env_options)
    
    # Apply variable-based configuration (ansible_*)
    if var_options:
        options.update(var_options)
    
    # Apply task-level parameters (highest precedence)
    if task_keys:
        options.update(task_keys)
    
    return options

Plugin Options Definition

class MyConnectionPlugin(ConnectionBase):
    documentation = '''
    options:
        timeout:
            description: Connection timeout
            type: int
            default: 30
            env:
                - name: ANSIBLE_MY_TIMEOUT
            vars:
                - name: ansible_my_timeout
    '''

Plugin Lifecycle Management

Plugin Loading Lifecycle

sequenceDiagram
    participant Core as Core Engine
    participant Loader as Plugin Loader
    participant Plugin as Plugin Class
    participant Instance as Plugin Instance
    
    Core->>Loader: get('plugin_name')
    Loader->>Loader: _find_plugin()
    Loader->>Loader: _load_plugin_from_file()
    Loader->>Plugin: import module
    Plugin-->>Loader: plugin class
    Loader->>Instance: __init__(options)
    Instance-->>Loader: configured instance
    Loader-->>Core: plugin instance
    
    Core->>Instance: plugin_method()
    Instance-->>Core: result
Loading

Lifecycle Phases:

  1. Discovery: Find plugin files in search paths
  2. Loading: Import Python module and extract plugin class
  3. Validation: Verify plugin inherits from correct base class
  4. Configuration: Apply configuration from multiple sources
  5. Instantiation: Create configured plugin instance
  6. Execution: Call plugin methods as needed
  7. Cleanup: Resource cleanup and connection closure

Plugin Caching and Performance

Multi-Level Caching:

class PluginLoader:
    def __init__(self):
        # Level 1: Path cache (name -> file path)
        self._plugin_path_cache = {}
        
        # Level 2: Module cache (file path -> Python module)
        self._module_cache = {}
        
        # Level 3: Instance cache (for stateless plugins)
        self._plugin_instance_cache = {}
    
    @lru_cache(maxsize=100)
    def _get_plugin_stage_list(self, plugin_type):
        """Cache plugin stage decisions"""
        return [p for p in self._plugins if plugin_type in p.REQUIRES]

Error Handling and Debugging

Plugin Error Context

flowchart TD
    PluginError[Plugin Error] --> Context[Error Context]
    Context --> LoadContext[Plugin Load Context]
    Context --> RuntimeContext[Runtime Context]
    
    LoadContext --> PluginPath[Plugin File Path]
    LoadContext --> LoadHistory[Load History]
    LoadContext --> RedirectInfo[Redirect Information]
    
    RuntimeContext --> TaskInfo[Current Task]
    RuntimeContext --> HostInfo[Current Host]
    RuntimeContext --> VarContext[Variable Context]
    
    PluginPath --> ErrorReport[Detailed Error Report]
    LoadHistory --> ErrorReport
    RedirectInfo --> ErrorReport
    TaskInfo --> ErrorReport
    HostInfo --> ErrorReport
    VarContext --> ErrorReport
    
    style PluginError fill:#ffebee
    style ErrorReport fill:#f3e5f5
Loading

Plugin Load Context:

@dataclass
class PluginLoadContext:
    resolved_fqcn: str = None
    plugin_resolved_name: str = None  
    plugin_resolved_path: str = None
    plugin_resolved_collection: str = None
    deprecation_warnings: list = field(default_factory=list)
    redirect_list: list = field(default_factory=list)
    error_list: list = field(default_factory=list)

Debugging Support

Plugin Debugging:

def _display_plugin_load_context(self, plugin_load_context):
    """Display detailed plugin loading information"""
    if plugin_load_context.redirect_list:
        self._display.vvv(f"Plugin redirects: {plugin_load_context.redirect_list}")
    
    if plugin_load_context.deprecation_warnings:
        for warning in plugin_load_context.deprecation_warnings:
            self._display.deprecated(warning['msg'], version=warning['version'])

Plugin Development Guidelines

Base Class Implementation

from ansible.plugins.action import ActionBase

class ActionModule(ActionBase):
    """Custom action plugin implementation"""
    
    def run(self, tmp=None, task_vars=None):
        """Main plugin execution method"""
        # Call parent for common setup
        result = super().run(tmp, task_vars)
        
        # Plugin-specific logic here
        try:
            # Perform action
            result['changed'] = True
            result['msg'] = 'Action completed successfully'
        except Exception as e:
            result['failed'] = True
            result['msg'] = str(e)
        
        return result

Configuration Options

class MyPlugin(BasePlugin):
    """Plugin with configuration options"""
    
    # Define plugin options
    DOCUMENTATION = '''
    options:
        api_endpoint:
            description: API endpoint URL
            type: str
            required: true
            env:
                - name: MY_PLUGIN_API_ENDPOINT
            vars:
                - name: my_plugin_api_endpoint
    '''
    
    def __init__(self):
        super().__init__()
        self.set_options()

Testing and Validation

# Unit test structure
class TestMyPlugin(unittest.TestCase):
    def setUp(self):
        self.plugin = MyPlugin()
        self.mock_task = Mock()
        self.mock_connection = Mock()
    
    def test_plugin_execution(self):
        result = self.plugin.run(task_vars={})
        self.assertIn('changed', result)
        self.assertTrue(result['changed'])

Performance Considerations

Plugin Loading Optimization

  1. Lazy Loading: Plugins loaded only when needed
  2. Caching: Multi-level caching reduces redundant operations
  3. Collection Optimization: Efficient FQCN resolution
  4. Instance Reuse: Stateless plugins cached for reuse

Memory Management

  1. Cache Size Limits: LRU caching with configurable limits
  2. Instance Cleanup: Proper resource cleanup in plugin destructors
  3. Module Unloading: Selective module cache clearing
  4. Connection Pooling: Shared connections across plugin instances

Scalability Features

  1. Parallel Loading: Concurrent plugin discovery where safe
  2. Path Optimization: Efficient search path ordering
  3. Collection Indexing: Fast collection plugin lookup
  4. Metadata Caching: Cache plugin metadata for repeated access

Summary

Ansible's plugin system demonstrates exceptional architectural design with:

  1. Extensibility: 15+ plugin types covering all major functionality areas
  2. Dynamic Loading: Runtime plugin discovery and loading with comprehensive caching
  3. Configuration Flexibility: Multi-source configuration with clear precedence rules
  4. Collection Integration: Modern plugin distribution and lifecycle management
  5. Performance Optimization: Multi-level caching and lazy loading strategies
  6. Developer Experience: Clear APIs, documentation standards, and debugging support
  7. Error Handling: Rich error context and detailed debugging information

This architecture enables Ansible to maintain a clean core while supporting unlimited extensibility through community and enterprise plugins, making it suitable for diverse automation requirements across different environments and use cases.