Skip to content

[SPARK-47199][PYTHON][TESTS] Add prefix into TemporaryDirectory to avoid flakiness #45298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 4 commits into from
Closed

[SPARK-47199][PYTHON][TESTS] Add prefix into TemporaryDirectory to avoid flakiness #45298

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c3eb6d3
Add prefix into TemporaryDirectory to avoid flakiness
HyukjinKwon Feb 28, 2024
66d7215
Address comments
HyukjinKwon Feb 28, 2024
ac758e7
typo
HyukjinKwon Feb 28, 2024
84e7502
typo
HyukjinKwon Feb 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
typo
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Feb 28, 2024
commit 84e7502bea186e1b4b1f7fff8ad5e52814281421
4 changes: 2 additions & 2 deletions python/pyspark/sql/streaming/readwriter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -985,7 +985,7 @@ def format(self, source: str) -> "DataStreamWriter":
>>> import tempfile
>>> import time
>>> with tempfile.TemporaryDirectory(prefix="format1") as d:
... with tempfile.TemporaryDirectory(prefix="format2") as cp):
... with tempfile.TemporaryDirectory(prefix="format2") as cp:
... df = spark.readStream.format("rate").load()
... q = df.writeStream.format("csv").option("checkpointLocation", cp).start(d)
... time.sleep(5)
Expand Down Expand Up @@ -1106,7 +1106,7 @@ def partitionBy(self, *cols: str) -> "DataStreamWriter": # type: ignore[misc]
>>> import tempfile
>>> import time
>>> with tempfile.TemporaryDirectory(prefix="partitionBy1") as d:
... with tempfile.TemporaryDirectory(prefix="partitionBy2") as cp):
... with tempfile.TemporaryDirectory(prefix="partitionBy2") as cp:
... df = spark.readStream.format("rate").option("rowsPerSecond", 10).load()
... q = df.writeStream.partitionBy(
... "timestamp").format("parquet").option("checkpointLocation", cp).start(d)
Expand Down