Add tests that run all samples #40875
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dargilco
requested review from
jhakulin,
luigiw,
needuv,
paulshealy1 and
singankit
as code owners
Collaborator
API Change CheckAPIView identified API level changes in this PR and created the following API reviews |
dargilco
merged commit a897b73
into
feature/azure-ai-projects-beta11
16 of 20 checks passed
annatisch
pushed a commit
that referenced
this pull request
May 15, 2025
* First * Re-emit * Fix setup.py * Quality gates * Update local cspell instead of root one * Remove projects SDK stuff from root cspell.json * Add azure-ai-agents path to pyrightconfig.json * Updating evaluation object to have name field instead of id (#40779) * Address comments from SDK review meeting (#40748) * Updating Evaluator Id list (#40798) * Updaing Evaluator Id list * Updating evaluator list * Re-emitted SDK (#40794) * Re-emitted SDK * Fixed patched code following auto-gen code changes * Evaluation Review Feedback (#40830) * Evaluation Review Feedback * Fixing samples * Fixing sample * Address Azure SDK review meeting comment and other fixes / minor changes (#40839) * For now, limit the build to Projects SDK only * Switch credential scope * Add tests that run all samples (#40875) * Re-emit from TypeSpec (#40876) * First (#40881) * Adding sample for agent evaluation (#40844) * Adding sample for agent evaluation * Review comments * Create automatic report of all Samples run, and their Exception message (if any) (#40885) * Add sync and async methods `.connections.get_default()` (#40916) * Fix pylint errors * Updating Agent Evalaution Sample (#40927) * Updating Agent Evlaution Sample * Update sample_agent_evaluations.py * Re-emit from latest TypeSpec (#40935) * Re-emit * Fix Sphinx * Fix async get_azure_openai_client. Fixes in samples test * Re-emit from TypeSpec. Fix one sample * Users/singankit/evaluation sample update (#40973) * Evaluation Sample Update * Fixing agent evaluation sample * Updating evaluation samples * Review Comment * Udpdating sample with evaluator ids * Spell check * Updates to Dataset upload methods (#41020) * Take setup dependency on azure-ai-agents. Update inference route to /models (#41040) * Use /models instead of /api/models route for inference * Remove lazy import of azure-ai-agents, not that it was published * Fix quality gates * Update package README * Attempt to fix quality gate related to dependency on azure-ai-agents * Quality gates and other small fixes in prep for release (#41062) * Take inference credential scopes from AIProjectClient * Run tool 'black' * Attempt to suppress verifytypes in auto-emitted code * Suppress verifytypes after talking to Krista * Update shared_requirements.txt to fix `Analyze dependencies` job * Update implementation of .inference.get_azure_openai_client() (#41076) Support either returning an AOAI client on the parent AI Services resource, or on a connected AOAI service. We are still debating which api_version to show in the sample code. * Fix AOAI test samples * Fix missing connection name is samples test * Exclue samples test file from spelling check * Update change log and readme (#41105) * Update readme, changelog * Fix quality gates * manually remove unused services pattern classes from emitted code (#41112) TypeSpec defined a namespace Azure.AI.Projects.ServicePatterns that contains a TypeSpec template for common operations used by Datasets and Indexes. This results in an unused property service_patterns on AIProjectClient that does nothing, and emitted ServicePattern classes that do nothing. I'm trying to update the TypeSpec namespace so that will not happen, but running into some issues. So for the time being, I'm manually editing the Python emitted code to unblock release so anything to do with service patter does not appear in the public API surface. * Updates to README and CHANGELOG * Users/singankit/readme update (#41123) * Updating smaples and README * Update Changelog file * Review comments * Fixing pyright issues * Quality gates for evaluation samples * Update sample_evaluations_aoai_graders.py (from Ankit) * fix broken cspell.json * pyrightconfig.json suppression of evaluation samples does not work --------- Co-authored-by: Ankit Singhal <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add two tests, one that runs all sync samples and one that runs all async samples. The appropriate environment variables are set before running sample. A report at the end shows which samples succeeded and which failed. Failed means an exception has been thrown (e.g. when the service returns an error).
Automatically running the samples required some minor change to them.
There is still one problem to be solved, so there will likely be another PR related to this.