Skip to content

[SPARK-23699][PYTHON][SQL] Raise same type of error caught with Arrow enabled #20839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BryanCutler wants to merge 8 commits into from
Closed

[SPARK-23699][PYTHON][SQL] Raise same type of error caught with Arrow enabled #20839

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5caf63c
raise same type of error when not falling back
BryanCutler Mar 15, 2018
dc5ed47
create new exception with modified msg
BryanCutler Mar 16, 2018
39c3473
try printing warning msg before raise
BryanCutler Mar 20, 2018
17dd605
format msg
BryanCutler Mar 21, 2018
55209b0
made a small fix in warning msg
BryanCutler Mar 21, 2018
e9a3e2c
Revert "format msg"
BryanCutler Mar 23, 2018
1a6be1d
reword warnings
BryanCutler Mar 23, 2018
5a43edf
made a small fix in warning msg (again)
BryanCutler Mar 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
create new exception with modified msg
  • Loading branch information
@BryanCutler
BryanCutler committed Mar 16, 2018
commit dc5ed47ec36433850d6c58890d19b4173eda9a06
10 changes: 6 additions & 4 deletions python/pyspark/sql/dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2013,13 +2013,14 @@ def toPandas(self):
warnings.warn(msg)
use_arrow = False
else:
e.message = (
clazz = type(e)
msg = (
"toPandas attempted Arrow optimization because "
"'spark.sql.execution.arrow.enabled' is set to true; however, "
"failed by the reason below:\n %s\n"
"For fallback to non-optimization automatically, please set true to "
"'spark.sql.execution.arrow.fallback.enabled'." % _exception_message(e))
raise
raise clazz(msg)

# Try to use Arrow optimization when the schema is supported and the required version
# of PyArrow is found, if 'spark.sql.execution.arrow.enabled' is enabled.
Expand All @@ -2040,14 +2041,15 @@ def toPandas(self):
except Exception as e:
# We might have to allow fallback here as well but multiple Spark jobs can
# be executed. So, simply fail in this case for now.
e.message = (
clazz = type(e)
msg = (
"toPandas attempted Arrow optimization because "
"'spark.sql.execution.arrow.enabled' is set to true; however, "
"failed unexpectedly:\n %s\n"
"Note that 'spark.sql.execution.arrow.fallback.enabled' does "
"not have an effect in such failure in the middle of "
"computation." % _exception_message(e))
raise
raise clazz(msg)

# Below is toPandas without Arrow optimization.
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
Expand Down
5 changes: 3 additions & 2 deletions python/pyspark/sql/session.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -679,13 +679,14 @@ def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=Tr
"true." % _exception_message(e))
warnings.warn(msg)
else:
e.message = (
clazz = type(e)
msg = (
"createDataFrame attempted Arrow optimization because "
"'spark.sql.execution.arrow.enabled' is set to true; however, "
"failed by the reason below:\n %s\n"
"For fallback to non-optimization automatically, please set true to "
"'spark.sql.execution.arrow.fallback.enabled'." % _exception_message(e))
raise
raise clazz(msg)
data = self._convert_from_pandas(data, schema, timezone)

if isinstance(schema, StructType):
Expand Down