Remove community retrievers

[TomeHirata]

Context

As part of the strategic shift of DSPy toward a reliable framework, we will stop supporting various community retrievers within the DSPy library since 2.7.0 due to the challenges of maintaining them reliably and the inactive status of their development.

Resolves #8072

Changes

Removed the following retrievers and related doc pages. Also, the retrieve dir is unified with retrievers dir.

• AzureAISearchRM
• ChromadbRM
• ClarifaiRM
• FalkordbRM
• FaissRM
• LancedbRM
• MilvusRM
• MyScaleRM
• Neo4jRM
• QdrantRM
• RAGatouilleRM
• SnowflakeRM
• WatsonDiscovery
• YouRM

[Copilot]

Copilot reviewed 45 out of 45 changed files in this pull request and generated no comments.

[TomeHirata]

Hi, @sam-h-bean @sky-2002 @AniLeo-01 @mogith-pn @insop @kingtroga @HARISHKUMAR1112001 @no-dice-io @PrashantDixit0 @jaelgu @tomasonjo @jhyearsley @poppingtonic @hasalams @drawal1 @andreswagner @sfc-gh-alherrera @ofermend.

Thank you all for your great contributions to the DSPy retrievers. Due to the maintenance challenges and strategic shift, we will remove the community retrievers you added previously. If you use these retrievers in your system, please move the implementation into your repositories. Please let us know should you have any compelling reasons to retain them in the DSPy repository. Again, we deeply appreciate your support and look forward to your future contributions.

Sincerely,

DSPy maintenance team.

[JPonsa]

@TomeHirata I understand the decision of removing the community retrievers as an architectural decision. However, is there a way to document them as examples in the DSPy documentation so the knowledge isn't lost?

Alternatively, @tomasonjo, could I request as a "wish" that the Neo4jRM is documented in the neo4j documentation https://neo4j.com/labs/genai-ecosystem/dspy/ as a code snippet, without any commitment to update it to future DSPy versions; just as reference.

Or am I missing something an retrievers are getting obsolete and being replaced with something else?

[chenmoneygithub]

@JPonsa We don't even know if the implementation is getting stale or still working, because we don't have maintainer with the domain knowledge of these vector index providers. Documentation is more robust if maintained by the retriever providers like neo4j than us, so I do agree with your call to add the documentation to neo4j site.