Skip to content

Conversation

@AngersZhuuuu
Copy link
Contributor

@AngersZhuuuu AngersZhuuuu commented May 10, 2021

What changes were proposed in this pull request?

RepairTableCommand respects spark.sql.addPartitionInBatch.size too

Why are the changes needed?

Make RepairTableCommand add partition batch size configurable.

Does this PR introduce any user-facing change?

User can use spark.sql.addPartitionInBatch.size to change batch size when repair table.

How was this patch tested?

Not need

@github-actions github-actions bot added the SQL label May 10, 2021
@AngersZhuuuu
Copy link
Contributor Author

ping @MaxGekk @wangyum @maropu

Copy link
Member

@MaxGekk MaxGekk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to change the config doc since it mentions the concrete command:

"`AlterTableAddPartitionCommand` to add partitions into table. The smaller " +

@MaxGekk
Copy link
Member

MaxGekk commented May 10, 2021

RepairTableCommand respect -> RepairTableCommand respects

@AngersZhuuuu AngersZhuuuu changed the title [SPARK-35360][SQL] RepairTableCommand respect spark.sql.addPartitionInBatch.size too [SPARK-35360][SQL] RepairTableCommand respects spark.sql.addPartitionInBatch.size too May 10, 2021
@AngersZhuuuu
Copy link
Contributor Author

Need to change the config doc since it mentions the concrete command:

"`AlterTableAddPartitionCommand` to add partitions into table. The smaller " +

Done

@SparkQA
Copy link

SparkQA commented May 10, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42847/

@SparkQA
Copy link

SparkQA commented May 10, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42847/

@MaxGekk
Copy link
Member

MaxGekk commented May 10, 2021

+1, LGTM. GA passed. Merging to master.
Thank you @AngersZhuuuu .

@MaxGekk MaxGekk closed this in 7182f8c May 10, 2021
@SparkQA
Copy link

SparkQA commented May 10, 2021

Test build #138325 has finished for PR 32489 at commit 918ddcd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants