Skip to content

[SPARK-26103][SQL] Limit the length of debug strings for query plans #23169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
DaveDeCaprio wants to merge 39 commits into from
Closed

[SPARK-26103][SQL] Limit the length of debug strings for query plans #23169

Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
22bd4bd
Merge pull request #1 from apache/master
DaveDeCaprio Nov 27, 2018
b7f964d
Added a configurable limit on the maximum length of a plan debug string.
DaveDeCaprio Nov 28, 2018
3f4fe1f
Removed unneeded imports
DaveDeCaprio Nov 28, 2018
30e4348
Moved withSizeLimitedWriter to treeString function that uses StringWr…
DaveDeCaprio Nov 28, 2018
5528ca1
Fixed formatting
DaveDeCaprio Nov 28, 2018
3171cf3
Coverted to javadoc style multiline comment
DaveDeCaprio Nov 28, 2018
3ffdc6a
Fixed scalastyle formatting of imports
DaveDeCaprio Nov 28, 2018
45a60fc
Have size limit cut off right at the correct number of characters.
DaveDeCaprio Nov 29, 2018
9678799
Changed name to remove the debug from the config parameter name
DaveDeCaprio Nov 29, 2018
a5af842
Fixed formatting error
DaveDeCaprio Nov 29, 2018
a4be985
Changed length default to Long.MaxValue to turn off behavior unless i…
DaveDeCaprio Dec 3, 2018
f0f75c2
Merge branch 'master' of https://github.com/apache/spark into text-pl…
DaveDeCaprio Dec 3, 2018
1b692a0
Added test case for not limiting plan length and tested with a defaul…
DaveDeCaprio Dec 4, 2018
22fe117
Correctly added test case missed in the previous commit
DaveDeCaprio Dec 4, 2018
be3f265
Added more documentation of the plan length parameter.
DaveDeCaprio Dec 8, 2018
f6d0efc
Merge branch 'master' of https://github.com/apache/spark into text-pl…
DaveDeCaprio Dec 8, 2018
855f540
Removed tab
DaveDeCaprio Dec 8, 2018
e83f5f2
Merge branch 'master' of https://github.com/apache/spark into text-pl…
DaveDeCaprio Jan 5, 2019
db663b7
Merge branch 'master' of https://github.com/apache/spark into text-pl…
DaveDeCaprio Jan 20, 2019
2eecbfa
Added plan size limits to StringConcat
DaveDeCaprio Jan 21, 2019
4082aa3
Scalastyle
DaveDeCaprio Jan 21, 2019
f9085e7
Incorporated changes from code review
DaveDeCaprio Jan 21, 2019
b3d43b7
Missed one error
DaveDeCaprio Jan 21, 2019
e470ab2
Cleaned up append function and added tracking of the full plan length.
DaveDeCaprio Jan 21, 2019
35bc1d5
Got rid of unneeded "availableLength" flag.
DaveDeCaprio Jan 21, 2019
5ec58c8
Fixed errors missed
DaveDeCaprio Jan 21, 2019
bdfaf28
Updated to handle nulls again.
DaveDeCaprio Jan 22, 2019
0cfcb4e
Tabs to spaces
DaveDeCaprio Jan 22, 2019
eb69888
Remove useless test.
DaveDeCaprio Jan 22, 2019
daf02f2
Addressed code review comments.
DaveDeCaprio Mar 8, 2019
7d89388
Missed some Boolean->Unit updates
DaveDeCaprio Mar 8, 2019
4f56e48
Missed some Boolean->Unit updates
DaveDeCaprio Mar 8, 2019
a090fbb
Code formatting
DaveDeCaprio Mar 8, 2019
dcb4eb0
Fixed Fatal Warning.
DaveDeCaprio Mar 8, 2019
b4cb7bf
Fixed failing unit tests
DaveDeCaprio Mar 9, 2019
4fec590
Style/formatting issues from code review
DaveDeCaprio Mar 11, 2019
db0db18
Style/formatting issues from code review
DaveDeCaprio Mar 11, 2019
b5b30f3
Changed plan string length to bytesConf and gave a better message for…
DaveDeCaprio Mar 13, 2019
e4afa26
Tabs to spaces
DaveDeCaprio Mar 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' of https://github.com/apache/spark into text-pl… 
…an-size

# Conflicts:
#	sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
  • Loading branch information
@DaveDeCaprio
DaveDeCaprio committed Dec 3, 2018
commit f0f75c25b95010d63ecdf83bb9f280687361d154
10 changes: 10 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1616,6 +1616,14 @@ object SQLConf {
"longer, it will end with a ... and further output will be truncated.")
.longConf
.createWithDefault(Long.MaxValue)

val SET_COMMAND_REJECTS_SPARK_CONFS =
buildConf("spark.sql.legacy.execution.setCommandRejectsSparkConfs")
.internal()
.doc("If it is set to true, SET command will fail when the key is registered as " +
"a SparkConf entry.")
.booleanConf
.createWithDefault(true)
}

/**
Expand Down Expand Up @@ -2038,6 +2046,8 @@ class SQLConf extends Serializable with Logging {

def maxPlanStringLength: Long = getConf(SQLConf.MAX_PLAN_STRING_LENGTH)

def setCommandRejectsSparkConfs: Boolean = getConf(SQLConf.SET_COMMAND_REJECTS_SPARK_CONFS)

/** ********************** SQLConf functionality methods ************ */

/** Set Spark SQL configuration properties. */
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.