Skip to content

fix live test issues #44594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
amber-Chen-86 wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

fix live test issues #44594

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
96ad091
update tests.yml
amber-Chen-86 Jan 8, 2026
541fc6d
add skip live test in conversations
amber-Chen-86 Jan 8, 2026
1e3b2d8
updated live tests for conv authoring
amber-Chen-86 Jan 9, 2026
e56b3ad
updated assets.json
amber-Chen-86 Jan 9, 2026
1a6c15a
adjusted tests in text authoring
amber-Chen-86 Jan 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sdk/cognitivelanguage/azure-ai-language-conversations-authoring/assets.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "python",
"TagPrefix": "python/cognitivelanguage/azure-ai-language-conversations-authoring",
"Tag": "python/cognitivelanguage/azure-ai-language-conversations-authoring_94c35e32fe"
"Tag": "python/cognitivelanguage/azure-ai-language-conversations-authoring_1f8d51d76b"
}
5 changes: 3 additions & 2 deletions ...itivelanguage/azure-ai-language-conversations-authoring/tests/test_cancel_training_job.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# pylint: disable=line-too-long,useless-suppression
import functools
import pytest

from devtools_testutils import AzureRecordedTestCase, EnvironmentVariableLoader, recorded_by_proxy
from azure.core.credentials import AzureKeyCredential
Expand All @@ -18,7 +19,7 @@ class TestConversations(AzureRecordedTestCase):
def create_client(self, endpoint, key):
return ConversationAuthoringClient(endpoint, AzureKeyCredential(key))


@pytest.mark.playback_test_only
class TestConversationsCancelTrainingSync(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy
Expand All @@ -27,7 +28,7 @@ def test_cancel_training_job(self, authoring_endpoint, authoring_key):
client = self.create_client(authoring_endpoint, authoring_key)

project_client = client.get_project_client("Test-data-labels")
job_id = "0f2fa6fe-c39c-48d5-a1e1-1e549a1c2299_638985888000000000"
job_id = "e39f6ec0-9ba9-426b-989d-3e0804bdcd1c_639034272000000000"

poller = project_client.project.begin_cancel_training_job(
job_id=job_id,
Expand Down
4 changes: 2 additions & 2 deletions ...anguage/azure-ai-language-conversations-authoring/tests/test_cancel_training_job_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ class TestConversationsAsync(AzureRecordedTestCase):
async def create_client(self, endpoint: str, key: str) -> ConversationAuthoringClient:
return ConversationAuthoringClient(endpoint, AzureKeyCredential(key))


@pytest.mark.playback_test_only
class TestConversationsCancelTrainingAsync(TestConversationsAsync):
@ConversationsPreparer()
@recorded_by_proxy_async
Expand All @@ -29,7 +29,7 @@ async def test_cancel_training_job_async(self, authoring_endpoint, authoring_key
client = await self.create_client(authoring_endpoint, authoring_key)
try:
project_name = "Test-data-labels"
job_id = "122b7341-afaa-4bea-af63-f6c7e0539fcb_638985888000000000"
job_id = "fb3b428d-185f-4d6e-a777-e3fd86cab993_639034272000000000"
project_client = client.get_project_client(project_name)

poller = await project_client.project.begin_cancel_training_job(
Expand Down
5 changes: 3 additions & 2 deletions ...gnitivelanguage/azure-ai-language-conversations-authoring/tests/test_delete_deployment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# pylint: disable=line-too-long,useless-suppression
import functools
import pytest

from devtools_testutils import AzureRecordedTestCase, EnvironmentVariableLoader, recorded_by_proxy
from azure.core.credentials import AzureKeyCredential
Expand All @@ -19,15 +20,15 @@ class TestConversations(AzureRecordedTestCase):
def create_client(self, endpoint, key):
return ConversationAuthoringClient(endpoint, AzureKeyCredential(key))


@pytest.mark.playback_test_only
class TestConversationsDeleteDeploymentSync(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy
def test_delete_deployment(self, authoring_endpoint, authoring_key):
client = self.create_client(authoring_endpoint, authoring_key)

project_name = "Test-data-labels"
deployment_name = "deployment2"
deployment_name = "deployment1"

project_client = client.get_project_client(project_name)

Expand Down
4 changes: 2 additions & 2 deletions ...elanguage/azure-ai-language-conversations-authoring/tests/test_delete_deployment_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ class TestConversationsAsync(AzureRecordedTestCase):
async def create_client(self, endpoint: str, key: str) -> ConversationAuthoringClient:
return ConversationAuthoringClient(endpoint, AzureKeyCredential(key))


@pytest.mark.playback_test_only
class TestConversationsDeleteDeploymentAsync(TestConversationsAsync):
@ConversationsPreparer()
@recorded_by_proxy_async
Expand All @@ -30,7 +30,7 @@ async def test_delete_deployment_async(self, authoring_endpoint, authoring_key):
client = await self.create_client(authoring_endpoint, authoring_key)
async with client:
project_name = "Test-data-labels"
deployment_name = "deployment2"
deployment_name = "deployment1"

project_client = client.get_project_client(project_name)

Expand Down
3 changes: 2 additions & 1 deletion .../azure-ai-language-conversations-authoring/tests/test_delete_deployment_from_resources.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# pylint: disable=line-too-long,useless-suppression
import functools
import pytest

from devtools_testutils import AzureRecordedTestCase, EnvironmentVariableLoader, recorded_by_proxy
from azure.core.credentials import AzureKeyCredential
Expand All @@ -19,7 +20,7 @@ class TestConversations(AzureRecordedTestCase):
def create_client(self, endpoint, key):
return ConversationAuthoringClient(endpoint, AzureKeyCredential(key))


@pytest.mark.playback_test_only
class TestConversationsDeleteDeploymentSync(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy
Expand Down
2 changes: 1 addition & 1 deletion ...-ai-language-conversations-authoring/tests/test_delete_deployment_from_resources_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ class TestConversationsAsync(AzureRecordedTestCase):
async def create_client(self, endpoint: str, key: str) -> ConversationAuthoringClient:
return ConversationAuthoringClient(endpoint, AzureKeyCredential(key))


@pytest.mark.playback_test_only
class TestConversationsDeleteDeploymentFromResourcesAsync(TestConversationsAsync):
@ConversationsPreparer()
@recorded_by_proxy_async
Expand Down
2 changes: 1 addition & 1 deletion sdk/cognitivelanguage/azure-ai-language-conversations-authoring/tests/test_delete_project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ class TestConversationsDeleteProjectSync(TestConversations):
@recorded_by_proxy
def test_delete_project(self, authoring_endpoint, authoring_key):
authoring_client = self.create_client(authoring_endpoint, authoring_key)
project_name = "MyPythonProject0820"
project_name = "MyPythonProject1110"

# Act: begin delete (LRO)
poller = authoring_client.begin_delete_project(project_name)
Expand Down
2 changes: 1 addition & 1 deletion ...tivelanguage/azure-ai-language-conversations-authoring/tests/test_delete_project_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ class TestConversationsDeleteProjectAsync(TestConversationsAsync):
async def test_delete_project_async(self, authoring_endpoint, authoring_key):
client = await self.create_client(authoring_endpoint, authoring_key)
async with client:
project_name = "PythonImportProject0820"
project_name = "MyPythonProject1110Async"

# Begin delete; LRO now returns None on success
poller = await client.begin_delete_project(project_name)
Expand Down
2 changes: 1 addition & 1 deletion sdk/cognitivelanguage/azure-ai-language-conversations-authoring/tests/test_export_project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class TestConversationsExportCase(TestConversations):
@recorded_by_proxy
def test_export_project(self, authoring_endpoint, authoring_key):
authoring_client = self.create_client(authoring_endpoint, authoring_key)
project_name = "PythonImportProject0820"
project_name = "Test-data-labels"
project_client = authoring_client.get_project_client(project_name)

# Act: begin export (LRO)
Expand Down
4 changes: 2 additions & 2 deletions ...uage/azure-ai-language-conversations-authoring/tests/test_get_model_evaluation_results.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ class TestConversationsGetModelEvaluationResultsSync(TestConversations):
def test_get_model_evaluation_results(self, authoring_endpoint, authoring_key):
authoring_client = self.create_client(authoring_endpoint, authoring_key)

project_name = "Aurora-CLU-Prod"
trained_model_label = "m1"
project_name = "EmailApp"
trained_model_label = "Model1"

# Get trained-model scoped client and fetch the paged evaluation results
project_client = authoring_client.get_project_client(project_name)
Expand Down
4 changes: 2 additions & 2 deletions ...zure-ai-language-conversations-authoring/tests/test_get_model_evaluation_results_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ class TestConversationsGetModelEvaluationResultsAsync(TestConversationsAsync):
async def test_get_model_evaluation_results_async(self, authoring_endpoint, authoring_key):
client = await self.create_client(authoring_endpoint, authoring_key)
try:
project_name = "Aurora-CLU-Prod"
trained_model_label = "m1"
project_name = "EmailApp"
trained_model_label = "Model1"

# Get trained-model scoped client and fetch the paged evaluation results
project_client = client.get_project_client(project_name)
Expand Down
2 changes: 1 addition & 1 deletion sdk/cognitivelanguage/azure-ai-language-conversations/tests/test_orchestration_prediction.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ def create_client(self, endpoint, key):

...


@pytest.mark.playback_test_only
Copy link

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The decorator @pytest.mark.playback_test_only causes these tests to skip when running in live mode. Given that the PR title is "fix live test issues", adding this decorator seems contradictory - it prevents the tests from running live rather than fixing them.

If these tests genuinely cannot run in live mode and should only run in playback mode, this change makes sense, but the PR description should explain why. Otherwise, this may be masking the underlying issue rather than fixing it.

Suggested change
@pytest.mark.playback_test_only

Copilot uses AI. Check for mistakes.
class TestConversationsCase(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy
Expand Down
2 changes: 1 addition & 1 deletion ...tivelanguage/azure-ai-language-conversations/tests/test_orchestration_prediction_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ async def create_async_client(self, endpoint: str, key: str) -> ConversationAnal
credential = AzureKeyCredential(key)
return ConversationAnalysisClient(endpoint, credential)


@pytest.mark.playback_test_only
Copy link

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The decorator @pytest.mark.playback_test_only causes these tests to skip when running in live mode. Given that the PR title is "fix live test issues", adding this decorator seems contradictory - it prevents the tests from running live rather than fixing them.

If these tests genuinely cannot run in live mode and should only run in playback mode, this change makes sense, but the PR description should explain why. Otherwise, this may be masking the underlying issue rather than fixing it.

Suggested change
@pytest.mark.playback_test_only

Copilot uses AI. Check for mistakes.
class TestConversationsCase(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy_async
Expand Down
3 changes: 2 additions & 1 deletion sdk/cognitivelanguage/azure-ai-textanalytics-authoring/tests/test_cancel_training_job.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# pylint: disable=line-too-long,useless-suppression
import functools
import pytest

from devtools_testutils import AzureRecordedTestCase, EnvironmentVariableLoader, recorded_by_proxy
from azure.core.credentials import AzureKeyCredential
Expand All @@ -18,7 +19,7 @@ class TestConversations(AzureRecordedTestCase):
def create_client(self, endpoint, key):
return TextAuthoringClient(endpoint, AzureKeyCredential(key))


@pytest.mark.playback_test_only
class TestConversationsCancelTrainingSync(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy
Expand Down
2 changes: 1 addition & 1 deletion ...ognitivelanguage/azure-ai-textanalytics-authoring/tests/test_cancel_training_job_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
authoring_key="fake_key",
)


@pytest.mark.playback_test_only
class TestConversationsCancelTrainingAsync(AzureRecordedTestCase):
@ConversationsPreparer()
@recorded_by_proxy_async
Expand Down
3 changes: 2 additions & 1 deletion sdk/cognitivelanguage/azure-ai-textanalytics-authoring/tests/test_delete_deployment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# pylint: disable=line-too-long,useless-suppression
import functools
import pytest

from devtools_testutils import AzureRecordedTestCase, EnvironmentVariableLoader, recorded_by_proxy
from azure.core.credentials import AzureKeyCredential
Expand All @@ -19,7 +20,7 @@ class TestConversations(AzureRecordedTestCase):
def create_client(self, endpoint, key):
return TextAuthoringClient(endpoint, AzureKeyCredential(key))


@pytest.mark.playback_test_only
class TestConversationsDeleteDeploymentSync(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy
Expand Down
2 changes: 1 addition & 1 deletion sdk/cognitivelanguage/azure-ai-textanalytics-authoring/tests/test_delete_deployment_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ class TestConversations(AzureRecordedTestCase):
def create_client(self, endpoint, key):
return TextAuthoringClient(endpoint, AzureKeyCredential(key)) # type: ignore[arg-type]


@pytest.mark.playback_test_only
class TestConversationsDeleteDeploymentAsync(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy_async
Expand Down
4 changes: 3 additions & 1 deletion sdk/cognitivelanguage/azure-ai-textanalytics-authoring/tests/test_delete_project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# pylint: disable=line-too-long,useless-suppression
import functools
import pytest

from devtools_testutils import AzureRecordedTestCase, PowerShellPreparer, recorded_by_proxy
from azure.core.credentials import AzureKeyCredential
from azure.core.exceptions import HttpResponseError
Expand All @@ -17,7 +19,7 @@ class TestConversations(AzureRecordedTestCase):
def create_client(self, endpoint: str, key: str) -> TextAuthoringClient:
return TextAuthoringClient(endpoint, AzureKeyCredential(key))


@pytest.mark.playback_test_only
class TestConversationsCase(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy
Expand Down
2 changes: 1 addition & 1 deletion sdk/cognitivelanguage/azure-ai-textanalytics-authoring/tests/test_delete_project_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ class TestConversations(AzureRecordedTestCase):
def create_client(self, endpoint: str, key: str) -> TextAuthoringClient: # type: ignore[override]
return TextAuthoringClient(endpoint, AzureKeyCredential(key)) # type: ignore[arg-type]


@pytest.mark.playback_test_only
class TestConversationsCaseAsync(TestConversations):
@ConversationsPreparer()
@recorded_by_proxy_async
Expand Down
12 changes: 6 additions & 6 deletions sdk/cognitivelanguage/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ extends:
AZURE_CONVERSATIONS_DEPLOYMENT_NAME: $(clu-project-deployment)
AZURE_CONVERSATIONS_WORKFLOW_PROJECT_NAME: $(clu-workflow-project)
AZURE_CONVERSATIONS_WORKFLOW_DEPLOYMENT_NAME: $(clu-workflow-project-deployment)
CONVERSATIONS_ENDPOINT: $(clu-uri)
CONVERSATIONS_KEY: $(clu-key)
AUTHORING_ENDPOINT: $(clu-uri)
AUTHORING_KEY: $(clu-key)
TEXT_ANALYSIS_ENDPOINT: $(clu-uri)
TEXT_ANALYSIS_KEY: $(clu-key)
CONVERSATIONS_ENDPOINT: $(conversation-uri)
Copy link

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify that the Azure DevOps pipeline variable $(conversation-uri) exists and is properly configured. The variable name has changed from $(clu-uri) to $(conversation-uri), which is a different naming pattern. If this variable is not defined in the pipeline configuration, tests will fail with missing credentials.

Copilot uses AI. Check for mistakes.
CONVERSATIONS_KEY: $(conversation-key)
Copy link

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify that the Azure DevOps pipeline variable $(conversation-key) exists and is properly configured. The variable name has changed from $(clu-key) to $(conversation-key), which is a different naming pattern. If this variable is not defined in the pipeline configuration, tests will fail with missing credentials.

Suggested change
CONVERSATIONS_KEY: $(conversation-key)
CONVERSATIONS_KEY: $(clu-key)

Copilot uses AI. Check for mistakes.
AUTHORING_ENDPOINT: $(authoring-uri)
Copy link

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify that the Azure DevOps pipeline variable $(authoring-uri) exists and is properly configured. The variable name has changed from $(clu-uri) to $(authoring-uri), which is a different naming pattern. If this variable is not defined in the pipeline configuration, tests will fail with missing credentials.

Suggested change
AUTHORING_ENDPOINT: $(authoring-uri)
AUTHORING_ENDPOINT: $(clu-uri)

Copilot uses AI. Check for mistakes.
AUTHORING_KEY: $(authoring-key)
Copy link

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify that the Azure DevOps pipeline variable $(authoring-key) exists and is properly configured. The variable name has changed from $(clu-key) to $(authoring-key), which is a different naming pattern. If this variable is not defined in the pipeline configuration, tests will fail with missing credentials.

Suggested change
AUTHORING_KEY: $(authoring-key)
AUTHORING_KEY: $(clu-key)

Copilot uses AI. Check for mistakes.
TEXT_ANALYSIS_ENDPOINT: $(text_analysis-uri)
Copy link

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify that the Azure DevOps pipeline variable $(text_analysis-uri) exists and is properly configured. The variable name has changed from $(clu-uri) to $(text_analysis-uri), which is a different naming pattern (note the underscore in the middle). If this variable is not defined in the pipeline configuration, tests will fail with missing credentials.

Copilot uses AI. Check for mistakes.
TEXT_ANALYSIS_KEY: $(text_analysis-key)
Copy link

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify that the Azure DevOps pipeline variable $(text_analysis-key) exists and is properly configured. The variable name has changed from $(clu-key) to $(text_analysis-key), which is a different naming pattern (note the underscore in the middle). If this variable is not defined in the pipeline configuration, tests will fail with missing credentials.

Copilot uses AI. Check for mistakes.
Comment on lines +25 to +26
Copy link

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an inconsistency in the naming convention for the Azure DevOps pipeline variables. Lines 21-24 use hyphens without underscores (e.g., conversation-uri, authoring-uri), while lines 25-26 use underscores (e.g., text_analysis-uri, text_analysis-key). This inconsistency could lead to confusion. Consider standardizing the naming convention to either use all hyphens or all underscores consistently.

Suggested change
TEXT_ANALYSIS_ENDPOINT: $(text_analysis-uri)
TEXT_ANALYSIS_KEY: $(text_analysis-key)
TEXT_ANALYSIS_ENDPOINT: $(text-analysis-uri)
TEXT_ANALYSIS_KEY: $(text-analysis-key)

Copilot uses AI. Check for mistakes.
AZURE_SUBSCRIPTION_ID: $(COGNITIVELANGUAGE_SUBSCRIPTION_ID)
TEST_MODE: "RunLiveNoRecord" # use when allowing preparers to create the rgs for you
AZURE_TEST_RUN_LIVE: 'true'
Expand Down
Loading