Skip to content

[SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile #45232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wbo4958 wants to merge 14 commits into from
Closed

[SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile #45232

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
11930c9
Support Stage-level scheduling for connect
wbo4958 Jan 3, 2024
a7ccd5e
add versionchanged
wbo4958 Feb 26, 2024
95a75af
reuse some tests
wbo4958 Feb 29, 2024
ca41951
comments
wbo4958 Mar 6, 2024
30531ae
comments
wbo4958 Mar 6, 2024
5e61437
comments
wbo4958 Mar 8, 2024
ba5c25a
pylint
wbo4958 Mar 8, 2024
3899ada
fix test
wbo4958 Mar 10, 2024
0e0f91f
use Command to create ResourceProfile
wbo4958 Mar 11, 2024
fb6d721
Merge remote-tracking branch 'upstream/master' into connect-rp
wbo4958 Mar 11, 2024
192e59e
Merge remote-tracking branch 'upstream/master' into connect-rp
wbo4958 Mar 19, 2024
1bdb284
Merge remote-tracking branch 'upstream/master' into connect-rp
wbo4958 Mar 20, 2024
7d1523f
Merge remote-tracking branch 'upstream/master' into connect-rp
wbo4958 Mar 27, 2024
7efa33b
comments
wbo4958 Mar 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
pylint
  • Loading branch information
@wbo4958
wbo4958 committed Mar 8, 2024
commit ba5c25a3e02e90affd50f6835f3a1b896bab7f79
5 changes: 3 additions & 2 deletions python/pyspark/resource/tests/test_connect_resources.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,10 @@ def assert_request_contents(exec_reqs, task_reqs):

with self.assertRaisesRegex(
Exception,
"ResourceProfiles are only supported on YARN and Kubernetes and Standalone with dynamic allocation enabled.",
"ResourceProfiles are only supported on YARN and Kubernetes and "
"Standalone with dynamic allocation enabled.",
):
df.mapInArrow(lambda x: x, df.schema, False, rp).collect()
rp.id

spark.stop()

Expand Down