Skip to content

[SPARK-23653][SQL] Show sql statement in spark SQL UI #20803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LantaoJin wants to merge 6 commits into from
Closed

[SPARK-23653][SQL] Show sql statement in spark SQL UI #20803

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9d2098d
[SPARK-23653][SQL] Show sql statement in spark SQL UI
LantaoJin Mar 12, 2018
4712379
fix the variable substitution problem like `${var}`, `${env:var}`
LantaoJin Mar 14, 2018
6f8bc0d
change sql text to DF and fix the problems mentioned in review
LantaoJin Mar 15, 2018
92293c6
simplify the code about sql text in DF
LantaoJin Mar 15, 2018
89e8e74
workaround for the Scala bug
LantaoJin Mar 16, 2018
df98d83
count() shouldn't show sql text
LantaoJin Mar 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
count() shouldn't show sql text
  • Loading branch information
@LantaoJin
LantaoJin committed Mar 19, 2018
commit df98d83a2ebc194864716f08636163905e2fd114
10 changes: 7 additions & 3 deletions sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2780,7 +2780,7 @@ class Dataset[T] private[sql](
* @group action
* @since 1.6.0
*/
def count(): Long = withAction("count", groupBy().count().queryExecution) { plan =>
def count(): Long = withAction("count", groupBy().count().queryExecution, true) { plan =>
plan.executeCollect().head.getLong(0)
}

Expand Down Expand Up @@ -3257,13 +3257,17 @@ class Dataset[T] private[sql](
* Wrap a Dataset action to track the QueryExecution and time cost, then report to the
* user-registered callback functions.
*/
private def withAction[U](name: String, qe: QueryExecution)(action: SparkPlan => U) = {
private def withAction[U](
name: String,
qe: QueryExecution,
hideSqlText: Boolean = false)(action: SparkPlan => U) = {
try {
qe.executedPlan.foreach { plan =>
plan.resetMetrics()
}
val start = System.nanoTime()
val result = SQLExecution.withNewExecutionId(sparkSession, qe, sqlText) {
val result = SQLExecution.withNewExecutionId(sparkSession, qe,
if (hideSqlText) "" else sqlText) {
action(qe.executedPlan)
}
val end = System.nanoTime()
Expand Down