Skip to content

update built-in chat eval flow #34896

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
singankit merged 8 commits into from
Mar 26, 2024
Merged

update built-in chat eval flow #34896

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
6f3cd27
update built-in chat eval flow
qusongms Mar 22, 2024
27fdf77
update _metric_handler.py for built-in chat flow
qusongms Mar 22, 2024
48cce4c
update e2e tests of chat evaluation flow
qusongms Mar 25, 2024
94ea1a3
update built-in qa evaluation flow
qusongms Mar 26, 2024
5f38fc8
Update _metric_handler.py
singankit Mar 26, 2024
e87b977
Update _utils.py
singankit Mar 26, 2024
61cbd8b
Update _local_flow_handler.py
singankit Mar 26, 2024
847672a
Merge branch 'main' into qunsong/chat-eval-groundedness-service
singankit Mar 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update _metric_handler.py for built-in chat flow 
* add fallback_groundedness_evaluation node to chat node list
* keep evaluation_per_turn column in output
  • Loading branch information
@qusongms
qusongms committed Mar 22, 2024
commit 27fdf77eb368e495997220f0e71a15d02aef7fee
5 changes: 4 additions & 1 deletion sdk/ai/azure-ai-generative/azure/ai/generative/evaluate/_metric_handler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

NODE_LIST_BY_TASK = {
"qa": ["gpt_coherence", "gpt_similarity", "gpt_relevance", "gpt_fluency", "gpt_groundedness"],
"chat": ["evaluate_chat_rag", "evaluate_coherence_fluency"],
"chat": ["evaluate_chat_rag", "evaluate_coherence_fluency", "fallback_groundedness_evaluation"],
}


Expand Down Expand Up @@ -136,6 +136,9 @@ def calculate_metrics(self) -> Dict:
if col.replace("outputs.", "").startswith(metric):
is_col_to_delete = False
break
# keep the column "evaluation_per_turn" in the output
if "evaluation_per_turn" in col:
is_col_to_delete = False
if is_col_to_delete:
columns_to_drop.append(col)
result_df.drop(columns_to_drop, axis=1, inplace=True)
Expand Down