[SPARK-40587][CONNECT][FOLLOW-UP] Make sure python client support select * #38218
Conversation
There was a problem hiding this comment.
The best is to support dataframe.schema then check the schema. However the StructType support in proto is under development: #38200, and there could be other blocking issues there.
Once schema is supported in proto, I will follow up to use .schema on python side whenever it makes sense.
There was a problem hiding this comment.
so it can not support selecting struct.* now?
There was a problem hiding this comment.
No not yet. That is on TODO list. We need proto change accordingly to support it.
Right now we don't even have a complete Struct support yet :)
python/pyspark/sql/connect/plan.py
Outdated
There was a problem hiding this comment.
how did this not break before in the mypy checks?
There was a problem hiding this comment.
I don't know. This breaks when code is running.
I should ask you probably as I remember you enabled mypy :-)
There was a problem hiding this comment.
so it can not support selecting struct.* now?
|
Can one of the admins verify this patch? |
amaliujia
changed the title
amaliujia
force-pushed
the
select_start_in_python
branch
from
704b953 to
1af9f98
Compare
amaliujia
force-pushed
the
select_start_in_python
branch
2 times, most recently
from
33b16a6 to
fefc9cb
Compare
amaliujia
force-pushed
the
select_start_in_python
branch
from
fefc9cb to
b9b9bb3
Compare
| for c in self._raw_columns: | ||
| if isinstance(c, Expression): | ||
| proj_exprs.append(c.to_plan(session)) | ||
| elif c == "*": |
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
LGTM as this is a followup, but we need to think more about how to support star completely in spark connect.
|
thanks, merging to master! |
…ect * ### What changes were proposed in this pull request? 1. Sync newest proto to python client. 2. Update Aggregate to match proto change. 3. Change `select` to have it accept both `column` and `str` 4. Make sure `*` pass through the entire path which has been implemented on the server side apache#38023 ### Why are the changes needed? Update python client side to match the change in connect proto. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UT Closes apache#38218 from amaliujia/select_start_in_python. Authored-by: Rui Wang <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
6 participants
What changes were proposed in this pull request?
selectto have it accept bothcolumnandstr*pass through the entire path which has been implemented on the server side [SPARK-40587][CONNECT] Support SELECT * in an explicit way in connect proto #38023Why are the changes needed?
Update python client side to match the change in connect proto.
Does this PR introduce any user-facing change?
No
How was this patch tested?
UT