Skip to content

Commit 808040c

Browse files
chandrasekharan-zipstackclaudepre-commit-ci[bot]coderabbitai[bot]
authored
UN-2930 [MISC] Add plugin infrastructure to unstract-core (#1618)
* UN-2930 [FEAT] Add core plugin infrastructure Introduce a generic, framework-agnostic plugin system that supports Django, Flask, and Workers. This foundation enables dynamic loading of plugins with metadata validation, singleton/non-singleton patterns, and support for compiled extensions. Key components: - Generic PluginManager with plugin discovery and validation - DjangoPluginManager wrapper for Django apps - FlaskPluginManager wrapper for Flask apps - Redis client enhancement (REDIS_USERNAME support) - Plugin directory exclusions in .gitignore - Development dependency: debugpy for debugging This is the first PR in a series for UN-2930, establishing the infrastructure needed for subscription usage tracking and other plugin-based features. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactored plugin gitignore to avoid explicitly stating plugin names * Added new plugin loading mechanism to BE NOTE: Existing mechanism is left untouched to avoid breaking anything * Addressed code rabbit's review comment * Minor code rabbit comment addressed, updated docs on adding and using plugins * misc: Updated plugin discovery to supported nested plugins upto 2 levels * UN-2930 [FEAT] Enhance plugin_loader with required parameter validation - Update plugin_loader docstring to clarify plugins_dir and plugins_pkg are required - Add parameter validation with helpful ValueError message - Include practical example showing correct usage pattern - Improves developer experience with clear setup requirements Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * UN-2930 [FIX] Add thread-safety to FlaskPluginManager singleton - Add threading.Lock at class level to protect singleton instance - Guard instance creation and state mutations with lock - Prevent race conditions in multi-threaded Flask environments - Ensure thread-safe initialization of app, plugins_dir, and plugins_pkg 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * UN-2930 [REFACTOR] Merge nested conditions and simplify exception handling - Merge nested if statements in FlaskPluginManager.getInstance() into single compound condition - Remove redundant PermissionError from plugin_manager (OSError parent class covers it) 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> * [MISC] Refactor plugin loading for BE and prompt-service to use centralized get_plugin() mechanism (#1626) Refactor plugin loading to use centralized get_plugin() mechanism Replaced manual plugin loading implementations across backend, prompt-service, and core modules with the new centralized get_plugin() function. This aligns all plugin loading with the new infrastructure, reduces code duplication, and simplifies maintenance. Removed obsolete loader files (subscription_loader, modifier_loader, processor_loader) that are now replaced by get_plugin(). Generated with Claude Code Co-authored-by: Claude <noreply@anthropic.com> * UN-2930 [FIX] Track subscription usage only for successful runs (#1620) UN-2930 [FEAT] Implement subscription tracking for successful executions only This PR implements the complete UN-2930 fix to track subscription usage only for successful API calls and workflow executions. Key changes: - Add subscription usage tracking decorator for Django views - Apply decorator to Prompt Studio index and answer-prompt operations - Add workflow completion/failure handlers for defer/commit/discard pattern - Implement batch tracking in workers for completed workflows - Add platform-service plugin integration for subscription tracking - Remove hardcoded SQL from platform-service, use plugin-based approach Components: 1. Decorator (backend/utils/subscription_usage_decorator.py): - Auto-commits usage on success - Auto-discards usage on failure - Non-blocking error handling 2. Prompt Studio Integration: - Track usage for index_document() - Track usage for prompt_responder() 3. Workflow Integration: - Completion handler commits deferred usage for COMPLETED workflows - Failure handler discards all deferred usage 4. Workers Integration: - Batch commit for successful file executions - Calls internal API for efficient batch processing 5. Platform Service: - Plugin-based subscription tracking - Cleaner separation of concerns This ensures usage is only recorded when operations actually succeed, fixing the issue where failed operations were incorrectly tracked. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update backend/configuration/config_registry.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com> --------- Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
1 parent 7db8766 commit 808040c

File tree

43 files changed

+1479
-1074
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+1479
-1074
lines changed

.gitignore

Lines changed: 0 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -620,29 +620,6 @@ $RECYCLE.BIN/
620620

621621
### Unstract ###
622622

623-
# Authentication Plugins
624-
backend/plugins/authentication/*
625-
!backend/plugins/authentication/auth_sample
626-
627-
# Processor Plugins
628-
backend/plugins/processor/*
629-
630-
# Subscription Plugins
631-
backend/plugins/subscription/*
632-
633-
634-
# API Deployment Plugins
635-
backend/plugins/api/**
636-
637-
# Notification Plugin
638-
backend/plugins/notification/**
639-
640-
# Configuration Plugin
641-
backend/plugins/configuration/**
642-
643-
# Verticals Usage Plugin
644-
backend/plugins/verticals_usage/**
645-
646623
# BE pluggable-apps
647624
backend/pluggable_apps/*
648625

backend/account_v2/subscription_loader.py

Lines changed: 0 additions & 129 deletions
This file was deleted.

backend/adapter_processor_v2/views.py

Lines changed: 33 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -342,37 +342,40 @@ def partial_update(
342342
# Send email notifications to newly shared users
343343
if response.status_code == 200 and AdapterKeys.SHARED_USERS in request.data:
344344
try:
345-
from plugins.notification.constants import ResourceType
346-
from plugins.notification.sharing_notification import (
347-
SharingNotificationService,
348-
)
349-
350-
adapter.refresh_from_db()
351-
new_shared_users = set(adapter.shared_users.all())
352-
newly_shared_users = new_shared_users - current_shared_users
353-
354-
if newly_shared_users:
355-
# Map adapter type to specific resource type
356-
adapter_type_to_resource = {
357-
"LLM": ResourceType.LLM.value,
358-
"EMBEDDING": ResourceType.EMBEDDING.value,
359-
"VECTOR_DB": ResourceType.VECTOR_DB.value,
360-
"X2TEXT": ResourceType.X2TEXT.value,
361-
}
362-
363-
resource_type = adapter_type_to_resource.get(
364-
adapter.adapter_type, ResourceType.LLM.value
365-
)
345+
from plugins import get_plugin
346+
347+
notification_plugin = get_plugin("notification")
348+
if notification_plugin:
349+
from plugins.notification.constants import ResourceType
350+
351+
adapter.refresh_from_db()
352+
new_shared_users = set(adapter.shared_users.all())
353+
newly_shared_users = new_shared_users - current_shared_users
354+
355+
if newly_shared_users:
356+
# Map adapter type to specific resource type
357+
adapter_type_to_resource = {
358+
"LLM": ResourceType.LLM.value,
359+
"EMBEDDING": ResourceType.EMBEDDING.value,
360+
"VECTOR_DB": ResourceType.VECTOR_DB.value,
361+
"X2TEXT": ResourceType.X2TEXT.value,
362+
}
363+
364+
resource_type = adapter_type_to_resource.get(
365+
adapter.adapter_type, ResourceType.LLM.value
366+
)
366367

367-
notification_service = SharingNotificationService()
368-
notification_service.send_sharing_notification(
369-
resource_type=resource_type,
370-
resource_name=adapter.adapter_name,
371-
resource_id=str(adapter.id),
372-
shared_by=request.user,
373-
shared_to=list(newly_shared_users),
374-
resource_instance=adapter,
375-
)
368+
# Get notification service from plugin
369+
service_class = notification_plugin["service_class"]
370+
notification_service = service_class()
371+
notification_service.send_sharing_notification(
372+
resource_type=resource_type,
373+
resource_name=adapter.adapter_name,
374+
resource_id=str(adapter.id),
375+
shared_by=request.user,
376+
shared_to=list(newly_shared_users),
377+
resource_instance=adapter,
378+
)
376379
except Exception as e:
377380
logger.exception(f"Failed to send sharing notification: {e}")
378381

backend/api_v2/api_deployment_dto_registry.py

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import logging
22
from typing import Any
33

4-
from plugins.api.dto import metadata
4+
from plugins import get_plugin
55

66
from api_v2.postman_collection.dto import PostmanCollection
77

@@ -13,10 +13,16 @@ class ApiDeploymentDTORegistry:
1313

1414
@classmethod
1515
def load_dto(cls) -> Any | None:
16-
class_name = PostmanCollection.__name__
17-
if metadata.get(class_name):
18-
return metadata[class_name].class_name
19-
return PostmanCollection # Return as soon as we find a valid DTO
16+
"""Load DTO from plugin or return default PostmanCollection.
17+
18+
Checks if the api_dto plugin is available and gets the Postman DTO
19+
via the service, otherwise returns the base PostmanCollection class.
20+
"""
21+
plugin = get_plugin("api_dto")
22+
if plugin:
23+
service = plugin["service_class"]()
24+
return service.get_postman_dto()
25+
return PostmanCollection
2026

2127
@classmethod
2228
def get_dto(cls) -> type | None:

backend/api_v2/api_deployment_views.py

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
from django.db.models import QuerySet
88
from django.http import HttpResponse
99
from permissions.permission import IsOwner, IsOwnerOrSharedUserOrSharedToOrg
10+
from plugins import get_plugin
1011
from prompt_studio.prompt_studio_registry_v2.models import PromptStudioRegistry
1112
from rest_framework import serializers, status, views, viewsets
1213
from rest_framework.decorators import action
@@ -33,15 +34,12 @@
3334
SharedUserListSerializer,
3435
)
3536

36-
try:
37-
from plugins.notification.constants import ResourceType
38-
from plugins.notification.sharing_notification import SharingNotificationService
37+
# Check if notification plugin is available
38+
notification_plugin = get_plugin("notification")
3939

40-
NOTIFICATION_PLUGIN_AVAILABLE = True
41-
sharing_notification_service = SharingNotificationService()
42-
except ImportError:
43-
NOTIFICATION_PLUGIN_AVAILABLE = False
44-
sharing_notification_service = None
40+
# Import constants from notification plugin if available
41+
if notification_plugin:
42+
from plugins.notification.constants import ResourceType
4543

4644
logger = logging.getLogger(__name__)
4745

@@ -319,15 +317,17 @@ def partial_update(self, request: Request, *args: Any, **kwargs: Any) -> Respons
319317
if (
320318
response.status_code == 200
321319
and "shared_users" in request.data
322-
and NOTIFICATION_PLUGIN_AVAILABLE
320+
and notification_plugin
323321
):
324322
try:
325323
instance.refresh_from_db()
326324
new_shared_users = set(instance.shared_users.all())
327325
newly_shared_users = new_shared_users - current_shared_users
328326

329327
if newly_shared_users:
330-
notification_service = SharingNotificationService()
328+
# Get notification service from plugin
329+
service_class = notification_plugin["service_class"]
330+
notification_service = service_class()
331331
notification_service.send_sharing_notification(
332332
resource_type=ResourceType.API_DEPLOYMENT.value,
333333
resource_name=instance.display_name,

backend/backend/settings/base.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -308,7 +308,7 @@ def filter(self, record):
308308
"drf_yasg",
309309
"docs",
310310
# Plugins
311-
"plugins",
311+
"plugins.apps.PluginsConfig",
312312
"feature_flag",
313313
"django_celery_beat",
314314
# For additional helper commands

0 commit comments

Comments
 (0)