Skip to content

[DataFactory]Add AzureMySqlSink and two datasets #6820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
michaeljqzq merged 3 commits into from
Aug 7, 2019
Merged

[DataFactory]Add AzureMySqlSink and two datasets #6820

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1b18c20
[DataFactory]Add AzureMySqlSink and two datasets
wenbof Jul 31, 2019
ad04152
split name
hbzhxying Aug 1, 2019
640aecc
don't split name for odbc
hbzhxying Aug 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 94 additions & 1 deletion ...factory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1433,6 +1433,74 @@
}
}
},
"AmazonRedshiftTableDataset": {
"x-ms-discriminator-value": "AmazonRedshiftTable",
Copy link
Member

@majastrz majastrz Aug 2, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AmazonRedshiftTable [](start = 35, length = 19)

So you're adding a new discriminator value to an existing stable API. The SDK generated from the swagger before the change will be able to deal with the discriminator value of "AmazonRedshiftTable" because it's modeled as a string, but it will not be able to deserialize the payload specific to AmazonRedshiftTable. This unfortunately makes it an SDK breaking change. Can you add this in a new API version? #Closed

Copy link
Author

@wenbof-zz wenbof-zz Aug 4, 2019 •
edited by majastrz
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. it will be generated in a new API version when making changes to SDK. #Closed

Copy link
Author

@wenbof-zz wenbof-zz Aug 4, 2019 •
edited by majastrz
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as similar as this SDK change where you can see that we change the API version Azure/azure-sdk-for-net#7044 #Closed

Copy link
Member

@majastrz majastrz Aug 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To generate a new API version when making changes to SDK, you need the swagger to exist for the new API version, no?

In reply to: 310406114 [](ancestors = 310406114)

Copy link
Author

@wenbof-zz wenbof-zz Aug 5, 2019 •
edited by majastrz
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right. the order I understand is also the same as you : merge swagger change first, then generate SDK based on the latest swagger. #Closed

Copy link
Author

@wenbof-zz wenbof-zz Aug 5, 2019 •
edited by majastrz
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the new SDK version is always built based on the latest swagger. which means the old SDK were built based on swagger at specific time and then kept intact there. #Closed

Copy link
Member

@majastrz majastrz Aug 5, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not assume. Can you verify the behavior? What I want to know if it will throw. If the SDK just accepts the payload even if the user can't really interact with it, I'd be ok with that.

@michaeljqzq, can you chime in here as well? What is the behavior with the generated SDK if a new discriminator value is being returned in the backend, but the SDK doesn't know about it yet? Will it throw an exception? #Closed

Copy link
Author

@wenbof-zz wenbof-zz Aug 5, 2019 •
edited by majastrz
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I double checked our changes on SDK and can confirm that we manually changed the version and appending the new changes on existing one every time to the new version. @hvermis could you please also help confirm as well? thanks. #Closed

Copy link
Contributor

@hvermis hvermis Aug 6, 2019 •
edited by majastrz
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not update the API version. This is a known behavior and we have been adding new types like this. Our customers are aware of this. #Closed

Copy link
Member

@majastrz majastrz Aug 6, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wenbof and @hvermis, you are not actually answering my question, which is delaying this review. I'm not asking whether you have done this before (you clearly have). Azure is a distributed system. There's no mechanism to ensure that all customers simultaneously upgrade to latest and greatest SDK that you have generated, which is why we have breaking changes guidance.

So I will ask the question again in a different way. Let's say you have SDK v2 that supports connectors a and b and SDK v3 that supports connectors a, b, and c. User A has been happily using SDK v2 to manage resources using connectors a and b. Suddenly user B creates resources using connector c in the same subscription or RG via SDK v3.

What will User A now see when he/she queries for all resources?

  1. Will the old SDK v2 throw an exception?
  2. Will the responses deserialize successfully but incompletely?

Please pick either answer 1 or 2. #Closed

"description": "The Amazon Redshift table dataset.",
"type": "object",
"allOf": [
{
"$ref": "#/definitions/Dataset"
}
],
"properties": {
"typeProperties": {
"description": "Amazon Redshift table dataset properties.",
"x-ms-client-flatten": true,
"$ref": "#/definitions/AmazonRedshiftTableDatasetTypeProperties"
}
}
},
"AmazonRedshiftTableDatasetTypeProperties": {
"description": "Amazon Redshift table dataset properties.",
"properties": {
"tableName": {
"type": "object",
"description": "This property will be retired. Please consider using schema + table properties instead."
},
"table": {
"type": "object",
"description": "The Amazon Redshift table name. Type: string (or Expression with resultType string)."
},
"schema": {
"type": "object",
"description": "The Amazon Redshift schema name. Type: string (or Expression with resultType string)."
}
}
},
"Db2TableDataset": {
"x-ms-discriminator-value": "Db2Table",
"description": "The Db2 table dataset.",
"type": "object",
"allOf": [
{
"$ref": "#/definitions/Dataset"
}
],
"properties": {
"typeProperties": {
"description": "Db2 table dataset properties.",
"x-ms-client-flatten": true,
"$ref": "#/definitions/Db2TableDatasetTypeProperties"
}
}
},
"Db2TableDatasetTypeProperties": {
"description": "Db2 table dataset properties.",
"properties": {
"tableName": {
"type": "object",
"description": "This property will be retired. Please consider using schema + table properties instead."
},
"schema": {
"type": "object",
"description": "The Db2 schema name. Type: string (or Expression with resultType string)."
},
"table": {
"type": "object",
"description": "The Db2 table name. Type: string (or Expression with resultType string)."
}
}
},
"RelationalTableDataset": {
"x-ms-discriminator-value": "RelationalTable",
"description": "The relational table dataset.",
Expand Down Expand Up @@ -1558,8 +1626,16 @@
"description": "PostgreSQL table dataset properties.",
"properties": {
"tableName": {
"type": "object",
"description": "This property will be retired. Please consider using schema + table properties instead."
},
"table": {
"type": "object",
"description": "The PostgreSQL table name. Type: string (or Expression with resultType string)."
},
"schema": {
"type": "object",
"description": "The PostgreSQL schema name. Type: string (or Expression with resultType string)."
}
}
},
Expand Down Expand Up @@ -2668,7 +2744,24 @@
"typeProperties": {
"description": "Properties specific to this dataset type.",
"x-ms-client-flatten": true,
"$ref": "#/definitions/GenericDatasetTypeProperties"
"$ref": "#/definitions/NetezzaTableDatasetTypeProperties"
}
}
},
"NetezzaTableDatasetTypeProperties": {
"description": "Netezza dataset properties.",
"properties": {
"tableName": {
"type": "object",
"description": "This property will be retired. Please consider using schema + table properties instead."
},
"table": {
"type": "object",
"description": "The table name of the Netezza. Type: string (or Expression with resultType string)."
},
"schema": {
"type": "object",
"description": "The schema name of the Netezza. Type: string (or Expression with resultType string)."
}
}
},
Expand Down
15 changes: 15 additions & 0 deletions ...actory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1980,6 +1980,21 @@
}
}
},
"AzureMySqlSink": {
"description": "A copy activity Azure MySql sink.",
"type": "object",
"allOf": [
{
"$ref": "#/definitions/CopySink"
}
],
"properties": {
"preCopyScript": {
"type": "object",
"description": "A query to execute before starting the copy. Type: string (or Expression with resultType string)."
}
}
},
"ConcurSource": {
"description": "A copy activity Concur Service source.",
"type": "object",
Expand Down