[SPARK-31895][PYTHON][SQL] Support DataFrame.explain(extended: str) case to be consistent with Scala side #28711
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
|
Test build #123454 has finished for PR 28711 at commit
|
maropu
approved these changes
Jun 3, 2020
Member
maropu
left a comment
maropu
left a comment
There was a problem hiding this comment.
Thanks for the update! Looks reasonable.
Member
|
btw, typo in the description? |
Member
Author
|
Oh, I meant this is also consistent with |
Member
Author
|
Thanks @maropu. Merged to master and branch-3.0! |
HyukjinKwon
added a commit
that referenced
this pull request
Jun 3, 2020
…ase to be consistent with Scala side
### What changes were proposed in this pull request?
Scala:
```scala
scala> spark.range(10).explain("cost")
```
```
== Optimized Logical Plan ==
Range (0, 10, step=1, splits=Some(12)), Statistics(sizeInBytes=80.0 B)
== Physical Plan ==
*(1) Range (0, 10, step=1, splits=12)
```
PySpark:
```python
>>> spark.range(10).explain("cost")
```
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../spark/python/pyspark/sql/dataframe.py", line 333, in explain
raise TypeError(err_msg)
TypeError: extended (optional) should be provided as bool, got <class 'str'>
```
In addition, it is consistent with other codes too, for example, `DataFrame.sample` also can support `DataFrame.sample(1.0)` and `DataFrame.sample(False)`.
### Why are the changes needed?
To provide the consistent API support across APIs.
### Does this PR introduce _any_ user-facing change?
Nope, it's only changes in unreleased branches.
If this lands to master only, yes, users will be able to set `mode` as `df.explain("...")` in Spark 3.1.
After this PR:
```python
>>> spark.range(10).explain("cost")
```
```
== Optimized Logical Plan ==
Range (0, 10, step=1, splits=Some(12)), Statistics(sizeInBytes=80.0 B)
== Physical Plan ==
*(1) Range (0, 10, step=1, splits=12)
```
### How was this patch tested?
Unittest was added and manually tested as well to make sure:
```python
spark.range(10).explain(True)
spark.range(10).explain(False)
spark.range(10).explain("cost")
spark.range(10).explain(extended="cost")
spark.range(10).explain(mode="cost")
spark.range(10).explain()
spark.range(10).explain(True, "cost")
spark.range(10).explain(1.0)
```
Closes #28711 from HyukjinKwon/SPARK-31895.
Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
(cherry picked from commit e1d5201)
Signed-off-by: HyukjinKwon <[email protected]>
Member
|
hahaha, I see. |
HyukjinKwon
pushed a commit
to databricks/koalas
that referenced
this pull request
Jun 4, 2020
Since Spark 3.0 will support `DataFrame.explain(extended: str)` case (apache/spark#28711), we can follow it. ```py >>> df.spark.explain("extended") # doctest: +ELLIPSIS == Parsed Logical Plan == ... == Analyzed Logical Plan == ... == Optimized Logical Plan == ... == Physical Plan == ... ```
rising-star92
added a commit
to rising-star92/databricks-koalas
that referenced
this pull request
Jan 27, 2023
Since Spark 3.0 will support `DataFrame.explain(extended: str)` case (apache/spark#28711), we can follow it. ```py >>> df.spark.explain("extended") # doctest: +ELLIPSIS == Parsed Logical Plan == ... == Analyzed Logical Plan == ... == Optimized Logical Plan == ... == Physical Plan == ... ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Scala:
PySpark:
In addition, it is consistent with other codes too, for example,
DataFrame.samplealso can supportDataFrame.sample(1.0)andDataFrame.sample(False).Why are the changes needed?
To provide the consistent API support across APIs.
Does this PR introduce any user-facing change?
Nope, it's only changes in unreleased branches.
If this lands to master only, yes, users will be able to set
modeasdf.explain("...")in Spark 3.1.After this PR:
How was this patch tested?
Unittest was added and manually tested as well to make sure: