Skip to content

[Data Factory]Add ORC Dataset/Copy Source/Copy Sink #7040

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Juliehzl merged 4 commits into from
Sep 3, 2019
Merged

[Data Factory]Add ORC Dataset/Copy Source/Copy Sink #7040

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
de493fa
Merge pull request #1 from Azure/master
TinaHu1 Aug 9, 2019
07dcf30
Merge pull request #3 from Azure/master
TinaHu1 Aug 13, 2019
b90d8e6
Merge pull request #4 from Azure/master
TinaHu1 Aug 22, 2019
864a2b3
[DataFactory]Add support for ORC dataset
TinaHu18 Aug 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions ...factory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -719,6 +719,35 @@
"location"
]
},
"OrcDataset": {
"x-ms-discriminator-value": "Orc",
"description": "ORC dataset.",
"type": "object",
"allOf": [
{
"$ref": "#/definitions/Dataset"
}
],
"properties": {
"typeProperties": {
"description": "ORC dataset properties.",
"x-ms-client-flatten": true,
"$ref": "#/definitions/OrcDatasetTypeProperties"
}
}
},
"OrcDatasetTypeProperties": {
"description": "ORC dataset properties.",
"properties": {
"location": {
"$ref": "#/definitions/DatasetLocation",
"description": "The location of the ORC data storage."
}
},
"required": [
"location"
]
},
"BinaryDataset": {
"x-ms-discriminator-value": "Binary",
"description": "Binary dataset.",
Expand Down
30 changes: 30 additions & 0 deletions ...actory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -776,6 +776,21 @@
}
}
},
"OrcSource": {
"description": "A copy activity ORC source.",
Copy link
Contributor

@Juliehzl Juliehzl Aug 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more thing is that where OrcSource and OrcSink are used in your swagger file.

Copy link
Contributor Author

@TinaHu1 TinaHu1 Sep 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the similiar usage.
Take Source as an example:
In specification/datafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json
"OrcSource" is to add a new type for "CopySource" with the defination "discriminator" in copy source(line 914). Then "CopySource" is referenced by "CopyActivityTypeProperties"(line 885)=>"CopyActivity"(line 859)=>extend a type for "Activity"(line 21)=>"Pipeline"(line 9)

In specification/datafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/datafactory.json
"Pipeline" was reference by "PipelineResource"(line 4106)=>used in several API.

Sink is almost the same as Source.

Copy link
Contributor

@Juliehzl Juliehzl Sep 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you use x-ms-discriminator-value in "OrcDataset" not in "OrcSource"?

Copy link
Contributor Author

@TinaHu1 TinaHu1 Sep 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the real value we want for customer to use is "Orc" but not "OrcDataset".
@Juliehzl It will be very helpful if we can merge this PR before tomorrow, thanks.

Copy link
Contributor

@Juliehzl Juliehzl Sep 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TinaHu1 Thanks for your detailed explanation. Just ensure everything is in the right way. 😊

"type": "object",
"allOf": [
{
"$ref": "#/definitions/CopySource"
}
],
"properties": {
"storeSettings": {
"$ref": "#/definitions/StoreReadSettings",
"description": "ORC store settings."
}
}
},
"DelimitedTextSink": {
"description": "A copy activity DelimitedText sink.",
"type": "object",
Expand Down Expand Up @@ -814,6 +829,21 @@
}
}
},
"OrcSink": {
"description": "A copy activity ORC sink.",
"type": "object",
"allOf": [
{
"$ref": "#/definitions/CopySink"
}
],
"properties": {
"storeSettings": {
"$ref": "#/definitions/StoreWriteSettings",
"description": "ORC store settings."
}
}
},
"CopyActivity": {
"x-ms-discriminator-value": "Copy",
"description": "Copy activity.",
Expand Down