[HDInsight]-Add descriptions for sparkjob related definition. #10121

sunsw1994 · 2020-07-14T02:53:23Z

MSFT employees can try out our new experience at OpenAPI Hub - one location for using our validation tools and finding your workflow.

Contribution checklist:

I have reviewed the documentation for the workflow.
Validation tools were run on swagger spec(s) and have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.

ARM API Review Checklist

Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
- adding/removing APIs.
- adding/removing properties.
- adding/removing API-version.
- adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

openapi-pipeline-app · 2020-07-14T02:53:27Z

[Staging] Swagger Validation Report

️️✔️BreakingChange [Detail]

 There are no breaking changes.

️️✔️LintDiff [Detail]

 Validation passes for LintDiff.

️️✔️Avocado [Detail]

 Validation passes for Avocado.

_{Posted by Swagger Pipeline | How to fix these errors?}

azure-pipelines · 2020-07-14T02:53:39Z

Azure Pipelines successfully started running 1 pipeline(s).

azuresdkci · 2020-07-14T02:54:17Z

Can one of the admins verify this patch?

openapi-sdkautomation · 2020-07-14T02:55:13Z

azure-sdk-for-python - Release

⚠️ warning [Logs] [Expand Details]

⚠️ Generate from 46877dc with merge commit 9fea3de. SDK Automation 13.0.17.20200619.4
```
[azure-hdinsight-job][ChangeLog] WARNING: Failed to generate ChangeLog.
```

⚠️azure-hdinsight-job [View full logs] [Release SDK Changes]

[build_conf] INFO:packaging_tools:Building template azure-hdinsight-job
[build_conf] INFO:packaging_tools.conf:Build default conf for azure-hdinsight-job
[build_conf] INFO:packaging_tools:Template done azure-hdinsight-job
[build_package] /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
[build_package]   warnings.warn(msg)
[build_package] /z/venv/lib/python3.6/site-packages/setuptools/dist.py:406: UserWarning: The version specified ('2018-11-01-preview') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
[build_package]   "details." % self.metadata.version
[build_package] warning: no files found matching '*.py' under directory 'tests'
[build_package] warning: no files found matching '*.yaml' under directory 'tests'
[build_package] /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
[build_package]   warnings.warn(msg)
[build_package] /z/venv/lib/python3.6/site-packages/setuptools/dist.py:406: UserWarning: The version specified ('2018-11-01-preview') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
[build_package]   "details." % self.metadata.version
[build_package] warning: no files found matching '*.py' under directory 'tests'
[build_package] warning: no files found matching '*.yaml' under directory 'tests'
[breaking_change_setup] Ignoring mock: markers 'python_version <= "2.7"' don't match your environment
Not exact 2 reports found:
Not generating changelog.
[ChangeLog] WARNING: Failed to generate ChangeLog.

openapi-sdkautomation · 2020-07-14T02:55:15Z

azure-sdk-for-js - Release

⚠️ warning [Logs] [Expand Details]

⚠️ Generate from 46877dc with merge commit 9fea3de. SDK Automation 13.0.17.20200619.4
```
Failed to find any diff after autorest so no changed packages was found.
```

openapi-sdkautomation · 2020-07-14T02:55:16Z

Azure CLI Extension Generation

No readme.md specification configuration files were found that are associated with the files modified in this pull request, or swagger_to_sdk section in readme.md is not configured

openapi-sdkautomation · 2020-07-14T02:55:16Z

azure-sdk-for-java - Release

🔄 inProgress [Logs] [Expand Details]

🔄 Generate from 46877dc with merge commit 9fea3de. SDK Automation 13.0.17.20200619.4

openapi-sdkautomation · 2020-07-14T02:59:21Z

azure-sdk-for-net - Release

No readme.md specification configuration files were found that are associated with the files modified in this pull request, or swagger_to_sdk section in readme.md is not configured

openapi-sdkautomation · 2020-07-14T02:59:37Z

Trenton Generation - Release

No readme.md specification configuration files were found that are associated with the files modified in this pull request, or swagger_to_sdk section in readme.md is not configured

openapi-sdkautomation · 2020-07-14T03:04:40Z

azure-sdk-for-go - Release

🔄 inProgress [Logs] [Expand Details]

🔄 Generate from 46877dc with merge commit 9fea3de. SDK Automation 13.0.17.20200619.4

aim-for-better · 2020-07-16T06:23:28Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

      "type": "object",
      "properties": {
        "from": {
+          "description": "The start index to fetch sessions.",


sessions [](start = 51, length = 8)

spark batch Job? #Closed

I add description following livy documatation(search GET /batches in page)

Is it ok for different from this doc? #Closed

I think that the minor change is acceptable and necessary to fit in our sdk. Although the class name is SparkBatchJobCollection.

In reply to: 455539008 [](ancestors = 455539008)

aim-for-better · 2020-07-16T06:24:11Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

          "type": "integer"
        },
        "sessions": {
+          "description": "Batch list.",


Batch list [](start = 26, length = 10)

The Spark Batch job list. #Closed

Thanks for your reminder. I will scan all description again to enhance the completeness. #Closed

aim-for-better · 2020-07-16T06:24:45Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

      "type": "object",
      "properties": {
        "id": {
+          "description": "The session id.",


session [](start = 30, length = 7)

The spark job's livy id.

I think livy is necessary to make it clear.
Do you think "The livy id of the spark batch job." will be better?

In reply to: 455536463 [](ancestors = 455536463)

aim-for-better · 2020-07-16T06:24:52Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

          "type": "integer"
        },
        "appId": {
+          "description": "The application id of this session.",


session [](start = 53, length = 7)

spark job #Closed

aim-for-better · 2020-07-16T06:25:22Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

        },
        "state": {
-          "type": "string"
+          "description": "The batch state.",


The batch state [](start = 26, length = 15)

The spark batch job state. #Closed

aim-for-better · 2020-07-16T06:27:31Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

        "state": {
-          "type": "string"
+          "description": "The batch state.",
+          "$ref": "#/definitions/SessionState"


SessionState [](start = 33, length = 12)

I noticed that spark batch job and spark session job share the same SessionState. Could you please double check this?
If they don't have the totally same state please split it.

aim-for-better · 2020-07-16T06:29:58Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

          }
        },
        "files": {
+          "description": "Files to be used in this session.",


session [](start = 51, length = 7)

I know that the description in livy doc used the word "session". But this property is one of SparkBatchJobRequest. So I think that we need to mitigate "session" to "spark batch job". How do you like? #Closed

aim-for-better · 2020-07-16T06:30:48Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

-            "sparkr",
-            "sql"
-          ]
+          "description": "Session kind.",


Session [](start = 26, length = 7)

Spark Session kind? #Closed

aim-for-better · 2020-07-16T06:57:26Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

      "type": "object",
      "properties": {
        "id": {
+          "description": "Session id.",


Session [](start = 26, length = 7)

The "Session" is so common. I think that we had better to add prefix for example Spark Batch, Spark Session, Spark Statement etc.
Could you please check all the "session" word to make it clear enough? #Closed

azure-pipelines · 2020-07-16T11:35:07Z

Azure Pipelines successfully started running 1 pipeline(s).

openapi-sdkautomation · 2020-07-16T11:36:39Z

azure-sdk-for-python-track2 - Release

No readme.md specification configuration files were found that are associated with the files modified in this pull request, or swagger_to_sdk section in readme.md is not configured

aim-for-better · 2020-07-27T07:50:25Z

...ification/hdinsight/data-plane/Microsoft.HDInsight/preview/2018-11-01-preview/livySpark.json

          "type": "integer"
        },
        "sessions": {
+          "description": "The Spark Batch job list.",


job [](start = 42, length = 3)

jobs

azure-pipelines · 2020-07-27T10:59:43Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2020-07-27T11:07:58Z

Azure Pipelines successfully started running 1 pipeline(s).

aim-for-better · 2020-08-05T02:41:24Z

Hi @anuchandy Could you please review the PR?

aim-for-better

LGTM

majastrz

Changes is just backfilling missing swagger (changing strings to enums to match real API). Signed off from ARM side.

Add descriptions for sparkjob related definition.

12ea82e

sunsw1994 added Do Not Generate SDK DoNotMerge <valid label in PR review process> use to hold merge after approval labels Jul 14, 2020

openapi-assignment-bot bot assigned anuchandy Jul 14, 2020

sunsw1994 requested review from aim-for-better and idear1203 July 16, 2020 06:15

sunsw1994 removed Do Not Generate SDK DoNotMerge <valid label in PR review process> use to hold merge after approval labels Jul 16, 2020

aim-for-better reviewed Jul 16, 2020

View reviewed changes

update descriptions

f15e6b7

aim-for-better reviewed Jul 27, 2020

View reviewed changes

Address comments

e28a551

Fix CI error

46877dc

aim-for-better requested a review from anuchandy August 5, 2020 02:40

aim-for-better self-requested a review August 5, 2020 02:41

aim-for-better approved these changes Aug 5, 2020

View reviewed changes

majastrz added the ARMSignedOff <valid label in PR review process>add this label when ARM approve updates after review label Aug 5, 2020

majastrz approved these changes Aug 5, 2020

View reviewed changes

anuchandy approved these changes Aug 5, 2020

View reviewed changes

anuchandy merged commit 9fea3de into Azure:master Aug 5, 2020

[HDInsight]-Add descriptions for sparkjob related definition. #10121

[HDInsight]-Add descriptions for sparkjob related definition. #10121

Uh oh!

Conversation

sunsw1994 commented Jul 14, 2020 • edited by aim-for-better Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution checklist:

ARM API Review Checklist

Uh oh!

openapi-pipeline-app bot commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

[Staging] Swagger Validation Report

Uh oh!

azure-pipelines bot commented Jul 14, 2020

Uh oh!

azuresdkci commented Jul 14, 2020

Uh oh!

openapi-sdkautomation bot commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

azure-sdk-for-python - Release

Uh oh!

openapi-sdkautomation bot commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

azure-sdk-for-js - Release

Uh oh!

openapi-sdkautomation bot commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Azure CLI Extension Generation

Uh oh!

openapi-sdkautomation bot commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

azure-sdk-for-java - Release

Uh oh!

openapi-sdkautomation bot commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

azure-sdk-for-net - Release

Uh oh!

openapi-sdkautomation bot commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Trenton Generation - Release

Uh oh!

openapi-sdkautomation bot commented Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

azure-sdk-for-go - Release

Uh oh!

aim-for-better Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunsw1994 Jul 16, 2020 • edited by aim-for-better Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aim-for-better Jul 16, 2020

Choose a reason for hiding this comment

Uh oh!

aim-for-better Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunsw1994 Jul 16, 2020 • edited by aim-for-better Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aim-for-better Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aim-for-better Jul 27, 2020

Choose a reason for hiding this comment

Uh oh!

aim-for-better Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aim-for-better Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aim-for-better Jul 16, 2020

sunsw1994 commented Jul 14, 2020 •

edited by aim-for-better

Loading

openapi-pipeline-app bot commented Jul 14, 2020 •

edited

Loading

openapi-sdkautomation bot commented Jul 14, 2020 •

edited

Loading

openapi-sdkautomation bot commented Jul 14, 2020 •

edited

Loading

openapi-sdkautomation bot commented Jul 14, 2020 •

edited

Loading

openapi-sdkautomation bot commented Jul 14, 2020 •

edited

Loading

openapi-sdkautomation bot commented Jul 14, 2020 •

edited

Loading

openapi-sdkautomation bot commented Jul 14, 2020 •

edited

Loading

openapi-sdkautomation bot commented Jul 14, 2020 •

edited

Loading

aim-for-better Jul 16, 2020 •

edited

Loading

sunsw1994 Jul 16, 2020 •

edited by aim-for-better

Loading

aim-for-better Jul 16, 2020 •

edited

Loading

sunsw1994 Jul 16, 2020 •

edited by aim-for-better

Loading

aim-for-better Jul 16, 2020 •

edited

Loading

aim-for-better Jul 16, 2020 •

edited

Loading

aim-for-better Jul 16, 2020 •

edited

Loading

aim-for-better Jul 16, 2020 •

edited

Loading

aim-for-better Jul 16, 2020 •

edited

Loading

aim-for-better Jul 16, 2020 •

edited

Loading

openapi-sdkautomation bot commented Jul 16, 2020 •

edited

Loading