Commit 808040c
UN-2930 [MISC] Add plugin infrastructure to unstract-core (#1618)
* UN-2930 [FEAT] Add core plugin infrastructure
Introduce a generic, framework-agnostic plugin system that supports
Django, Flask, and Workers. This foundation enables dynamic loading
of plugins with metadata validation, singleton/non-singleton patterns,
and support for compiled extensions.
Key components:
- Generic PluginManager with plugin discovery and validation
- DjangoPluginManager wrapper for Django apps
- FlaskPluginManager wrapper for Flask apps
- Redis client enhancement (REDIS_USERNAME support)
- Plugin directory exclusions in .gitignore
- Development dependency: debugpy for debugging
This is the first PR in a series for UN-2930, establishing the
infrastructure needed for subscription usage tracking and other
plugin-based features.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Refactored plugin gitignore to avoid explicitly stating plugin names
* Added new plugin loading mechanism to BE
NOTE: Existing mechanism is left untouched to avoid breaking anything
* Addressed code rabbit's review comment
* Minor code rabbit comment addressed, updated docs on adding and using plugins
* misc: Updated plugin discovery to supported nested plugins upto 2 levels
* UN-2930 [FEAT] Enhance plugin_loader with required parameter validation
- Update plugin_loader docstring to clarify plugins_dir and plugins_pkg are required
- Add parameter validation with helpful ValueError message
- Include practical example showing correct usage pattern
- Improves developer experience with clear setup requirements
Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* UN-2930 [FIX] Add thread-safety to FlaskPluginManager singleton
- Add threading.Lock at class level to protect singleton instance
- Guard instance creation and state mutations with lock
- Prevent race conditions in multi-threaded Flask environments
- Ensure thread-safe initialization of app, plugins_dir, and plugins_pkg
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* UN-2930 [REFACTOR] Merge nested conditions and simplify exception handling
- Merge nested if statements in FlaskPluginManager.getInstance() into single compound condition
- Remove redundant PermissionError from plugin_manager (OSError parent class covers it)
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
* [MISC] Refactor plugin loading for BE and prompt-service to use centralized get_plugin() mechanism (#1626)
Refactor plugin loading to use centralized get_plugin() mechanism
Replaced manual plugin loading implementations across backend, prompt-service,
and core modules with the new centralized get_plugin() function. This aligns
all plugin loading with the new infrastructure, reduces code duplication, and
simplifies maintenance. Removed obsolete loader files (subscription_loader,
modifier_loader, processor_loader) that are now replaced by get_plugin().
Generated with Claude Code
Co-authored-by: Claude <noreply@anthropic.com>
* UN-2930 [FIX] Track subscription usage only for successful runs (#1620)
UN-2930 [FEAT] Implement subscription tracking for successful executions only
This PR implements the complete UN-2930 fix to track subscription usage
only for successful API calls and workflow executions.
Key changes:
- Add subscription usage tracking decorator for Django views
- Apply decorator to Prompt Studio index and answer-prompt operations
- Add workflow completion/failure handlers for defer/commit/discard pattern
- Implement batch tracking in workers for completed workflows
- Add platform-service plugin integration for subscription tracking
- Remove hardcoded SQL from platform-service, use plugin-based approach
Components:
1. Decorator (backend/utils/subscription_usage_decorator.py):
- Auto-commits usage on success
- Auto-discards usage on failure
- Non-blocking error handling
2. Prompt Studio Integration:
- Track usage for index_document()
- Track usage for prompt_responder()
3. Workflow Integration:
- Completion handler commits deferred usage for COMPLETED workflows
- Failure handler discards all deferred usage
4. Workers Integration:
- Batch commit for successful file executions
- Calls internal API for efficient batch processing
5. Platform Service:
- Plugin-based subscription tracking
- Cleaner separation of concerns
This ensures usage is only recorded when operations actually succeed,
fixing the issue where failed operations were incorrectly tracked.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update backend/configuration/config_registry.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
---------
Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>1 parent 7db8766 commit 808040c
File tree
43 files changed
+1479
-1074
lines changed- backend
- account_v2
- adapter_processor_v2
- api_v2
- backend/settings
- configuration
- pipeline_v2
- plugins
- api/dto
- prompt_studio
- prompt_studio_core_v2
- prompt_studio_registry_v2
- scheduler
- utils
- workflow_manager/workflow_v2
- platform-service
- src/unstract/platform_service
- controller
- prompt-service
- src/unstract/prompt_service
- controllers
- helpers
- services
- unstract/core/src/unstract/core
- cache
- django
- flask
- plugins
- workers
- callback
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
43 files changed
+1479
-1074
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
620 | 620 | | |
621 | 621 | | |
622 | 622 | | |
623 | | - | |
624 | | - | |
625 | | - | |
626 | | - | |
627 | | - | |
628 | | - | |
629 | | - | |
630 | | - | |
631 | | - | |
632 | | - | |
633 | | - | |
634 | | - | |
635 | | - | |
636 | | - | |
637 | | - | |
638 | | - | |
639 | | - | |
640 | | - | |
641 | | - | |
642 | | - | |
643 | | - | |
644 | | - | |
645 | | - | |
646 | 623 | | |
647 | 624 | | |
648 | 625 | | |
| |||
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
342 | 342 | | |
343 | 343 | | |
344 | 344 | | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
362 | | - | |
363 | | - | |
364 | | - | |
365 | | - | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
366 | 367 | | |
367 | | - | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
376 | 379 | | |
377 | 380 | | |
378 | 381 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
20 | 26 | | |
21 | 27 | | |
22 | 28 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
36 | | - | |
37 | | - | |
38 | | - | |
| 37 | + | |
| 38 | + | |
39 | 39 | | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
45 | 43 | | |
46 | 44 | | |
47 | 45 | | |
| |||
319 | 317 | | |
320 | 318 | | |
321 | 319 | | |
322 | | - | |
| 320 | + | |
323 | 321 | | |
324 | 322 | | |
325 | 323 | | |
326 | 324 | | |
327 | 325 | | |
328 | 326 | | |
329 | 327 | | |
330 | | - | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
331 | 331 | | |
332 | 332 | | |
333 | 333 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
308 | 308 | | |
309 | 309 | | |
310 | 310 | | |
311 | | - | |
| 311 | + | |
312 | 312 | | |
313 | 313 | | |
314 | 314 | | |
| |||
0 commit comments