Multi-Cloud Storage Support

[EricXiao95]

Description

Expand Cognee's storage capabilities beyond AWS S3 to support Azure Blob Storage and Google Cloud Storage, while refactoring the existing storage architecture to be more extensible, maintainable, and cloud-agnostic. This enhancement will enable users to deploy Cognee on any major cloud platform and provide better storage flexibility.

What’s in this PR

• Storage Architecture Refactoring
• Remove hardcoded s3:// references
• Azure Blob Storage Support
• Google Cloud Storage Support
• Enhanced Kuzu Migration

Tests

• test_kuzu_migrate_cloud.py ---> Test kuzu migrate for cloud storage
• test_kuzu_migrate_local.py ---> Test kuzu migrate for local file system storage
• test_azure_blob_file_storage.py ---> This is an end-to-end integration test to ensure cognee works correctly with Azure Storage as its backend. (Prerequisite: Azure settings must be configured.)
• test_gcs_file_storage.py ---> This is an end-to-end integration test to ensure cognee works correctly with Google Cloud Storage as its backend. (Prerequisite: Google Cloud Storage settings must be configured.)

#1180

DCO Affirmation

I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin.

[pull-checklist]

Please make sure all the checkboxes are checked:

• I have tested these changes locally.
• I have reviewed the code changes.
• I have added end-to-end and unit tests (if applicable).
• I have updated the documentation and README.md file (if necessary).
• I have removed unnecessary code and debug statements.
• PR title is clear and follows the convention.
• I have tagged reviewers or team members for feedback.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Vasilije1990]

@EricXiao95 apologies for the delay. We had a bit of work for the upcoming UI release, but we will check this one next week

[gitguardian]

⚠️ GitGuardian has uncovered 3 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
9573981	Triggered	Generic Password	3acb1b6	.github/workflows/temporal_graph_tests.yml	View secret
8719688	Triggered	Generic Password	3acb1b6	.github/workflows/temporal_graph_tests.yml	View secret
7122536	Triggered	Generic High Entropy Secret	3acb1b6	.github/workflows/temporal_graph_tests.yml	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}