Skip to content

[SPARK-48567][SS] StreamingQuery.lastProgress should return the actual StreamingQueryProgress #46921

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
WweiL wants to merge 14 commits into from
+227 −99
Closed

[SPARK-48567][SS] StreamingQuery.lastProgress should return the actual StreamingQueryProgress #46921

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c73ebfc
?
WweiL Jun 8, 2024
d967119
wip
WweiL Jun 8, 2024
ef7a116
ready for CI, pending query.name fix
WweiL Jun 8, 2024
f8d7403
more
WweiL Jun 8, 2024
707d5bd
minor
WweiL Jun 8, 2024
917adb6
merge master, lint
WweiL Jun 10, 2024
cab14ac
Done
WweiL Jun 10, 2024
8355342
should pass
WweiL Jun 12, 2024
b43dc96
lint
WweiL Jun 12, 2024
984d00b
hmm why lint cant repro locally
WweiL Jun 12, 2024
e51e1b1
Merge remote-tracking branch 'spark/master' into SPARK-48567-lastProg…
WweiL Jun 14, 2024
b2cd7c9
here you go
WweiL Jun 14, 2024
4c8f01e
minor
WweiL Jun 14, 2024
53afe51
fix test failure
WweiL Jun 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
minor
  • Loading branch information
@WweiL
WweiL committed Jun 8, 2024
commit 707d5bd2312683f08e87e792e4d8fdf265ffeae9
4 changes: 2 additions & 2 deletions python/pyspark/sql/streaming/listener.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -501,8 +501,8 @@ def fromJson(cls, j: Dict[str, Any]) -> "StreamingQueryProgress":

def __getitem__(self, key):
# Before Spark 4.0, StreamingQuery.lastProgress returns a dict, which casts id and runId
# to string. To prevent breaking change, also cast them to string when accessed with
# __getitem__.
# to string. But here they are UUID.
# To prevent breaking change, also cast them to string when accessed with __getitem__.
if key == "id" or key == "runId":
Copy link
Contributor Author

@WweiL WweiL Jun 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this is really needed. But if we delete this if, now "query.lastProgress["id"]" would return type uuid, before it was string.

Copy link
Contributor Author

@WweiL WweiL Jun 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because there would be lots of breaking changes (e.g. now the sources method also return the actual SourceProgress

spark/python/pyspark/sql/streaming/listener.py

Line 595 in ef7a116

def sources(self) -> List["SourceProgress"]:

let me also make these subclass of dict...

return str(getattr(self, key))
else:
Expand Down