Skip to content

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer #39819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wayneguow wants to merge 11 commits into from
Closed

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer #39819

wayneguow wants to merge 11 commits into from

Conversation

@wayneguow
Copy link
Contributor

@wayneguow wayneguow commented Jan 31, 2023 •
edited
Loading

What changes were proposed in this pull request?

Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config spark.shuffle.localDisk.file.output.buffer instead.

Why are the changes needed?

The old config is desgined to be used in UnsafeShuffleWriter, but now it has been used in all local shuffle writers through LocalDiskShuffleMapOutputWriter, introduced by #25007.

Does this PR introduce any user-facing change?

Old still works, advised to use new.

How was this patch tested?

Passed existing tests.

@wayneguow wayneguow changed the title [SPARK-42252][Core] Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config [SPARK-42252][CORE] Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config Jan 31, 2023
@wayneguow
Copy link
Contributor Author

wayneguow commented Jan 31, 2023

cc @mccheah and @HyukjinKwon

@wayneguow wayneguow force-pushed the shuffle_output_buffer branch from 4696a36 to bd0c1b6 Compare February 3, 2023 13:04
dongjoon-hyun
dongjoon-hyun previously requested changes Feb 3, 2023
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you don't mind, let's target this proposal for Apache Spark 3.5.0.

@wayneguow
Copy link
Contributor Author

wayneguow commented Feb 4, 2023

If you don't mind, let's target this proposal for Apache Spark 3.5.0.

Ok, it's good. I will make some changes.

@wayneguow
Copy link
Contributor Author

wayneguow commented Feb 7, 2023

Gentle ping @dongjoon-hyun, the target version of this proposal has been updated. Could you go on reviewing it when you have time? Thanks.

@wayneguow wayneguow force-pushed the shuffle_output_buffer branch from fc1b3bf to fe8729d Compare February 14, 2023 08:10
@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented May 5, 2023

Thank you for your patience. Could you rebase this PR?

dongjoon-hyun
dongjoon-hyun reviewed May 5, 2023
View reviewed changes
docs/core-migration-guide.md Outdated
Copy link
Member

@dongjoon-hyun dongjoon-hyun May 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be Since Spark 3.5.

Copy link
Contributor Author

@wayneguow wayneguow May 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm here, updated it, and waiting for GA.

@wayneguow wayneguow force-pushed the shuffle_output_buffer branch from fe8729d to 6fafbc5 Compare May 6, 2023 08:08
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-42252][CORE] Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config [SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer May 7, 2023
@wayneguow wayneguow requested a review from dongjoon-hyun May 12, 2023 06:33
dongjoon-hyun
dongjoon-hyun reviewed May 12, 2023
View reviewed changes
docs/core-migration-guide.md Outdated
Copy link
Member

@dongjoon-hyun dongjoon-hyun May 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove thought still works. deprecated means it still works.

Copy link
Contributor Author

@wayneguow wayneguow May 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

solved

@HyukjinKwon
Copy link
Member

HyukjinKwon commented May 15, 2023

@wayneguow which PR made this change:

The old config is desgined to be used in UnsafeShuffleWriter, but now it has been used in all local shuffle writers through LocalDiskShuffleMapOutputWriter.

? Would be easier for me to review if you can point it out.

@wayneguow
Copy link
Contributor Author

wayneguow commented May 15, 2023

@wayneguow which PR made this change:

The old config is desgined to be used in UnsafeShuffleWriter, but now it has been used in all local shuffle writers through LocalDiskShuffleMapOutputWriter.

? Would be easier for me to review if you can point it out.

@HyukjinKwon The PR is #25007 .

dongjoon-hyun
dongjoon-hyun approved these changes May 18, 2023
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM from my side.

cc @HyukjinKwon , @cloud-fan , @Ngone51 , @mridulm

@github-actions
Copy link

github-actions bot commented Aug 27, 2023

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

@github-actions github-actions bot added the Stale label Aug 27, 2023
@github-actions github-actions bot closed this Aug 28, 2023
@dongjoon-hyun dongjoon-hyun reopened this Aug 28, 2023
@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Aug 28, 2023

Too bad. Sorry but it seems that we forgot this PR, @wayneguow .

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Aug 28, 2023

It seems too late because Apache Spark 3.5 is RC3 stage. If we want this still, we can do for Apache Spark 4.0.0.

@github-actions github-actions bot closed this Aug 29, 2023
@LuciferYang LuciferYang reopened this May 30, 2024
@LuciferYang LuciferYang removed the Stale label May 30, 2024
@wayneguow wayneguow force-pushed the shuffle_output_buffer branch from 5f19b3d to 02843cf Compare May 30, 2024 12:01
wayneguow
wayneguow commented May 30, 2024
View reviewed changes
@wayneguow
Copy link
Contributor Author

wayneguow commented May 30, 2024

Gentle ping @dongjoon-hyun , a pr from some time ago.

@LuciferYang
Copy link
Contributor

LuciferYang commented Jun 4, 2024

friendly ping @yaooqinn Do you have time to help take a look?

@yaooqinn yaooqinn closed this in dd8b05f Jun 14, 2024
@yaooqinn
Copy link
Member

yaooqinn commented Jun 14, 2024

Merged to master, thank you @dongjoon-hyun @LuciferYang @wayneguow

@wayneguow wayneguow deleted the shuffle_output_buffer branch February 11, 2025 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LuciferYang LuciferYang LuciferYang approved these changes

@yaooqinn yaooqinn yaooqinn approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees

No one assigned

Labels

CORE DOCS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@wayneguow @dongjoon-hyun @HyukjinKwon @LuciferYang @yaooqinn