Skip to content

[SPARK-24250][SQL] support accessing SQLConf inside tasks #21376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cloud-fan wants to merge 4 commits into from
Closed

[SPARK-24250][SQL] support accessing SQLConf inside tasks #21376

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ba46703
support accessing SQLConf at executor side
cloud-fan May 19, 2018
a1519d4
fix json schema inference
cloud-fan May 20, 2018
d25e846
address comments
cloud-fan May 20, 2018
f7a6299
address comments
cloud-fan May 21, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
address comments
  • Loading branch information
@cloud-fan
cloud-fan committed May 21, 2018
commit f7a629906f1ba15b663eb8ab1c6b49daa48c34d2
6 changes: 5 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,9 @@ object SQLConf {

/**
* Returns the active config object within the current scope. If there is an active SparkSession,
* the proper SQLConf associated with the thread's session is used.
* the proper SQLConf associated with the thread's active session is used. If it's called from
* tasks in the executor side, a SQLConf will be created from job local properties, which are set
* and propagated from the driver side.
*
* The way this works is a little bit convoluted, due to the fact that config was added initially
* only for physical plans (and as a result not in sql/catalyst module).
Expand All @@ -112,6 +114,8 @@ object SQLConf {
new ReadOnlySQLConf(TaskContext.get())
} else {
if (Utils.isTesting && SparkContext.getActive.isDefined) {
Copy link
Member

@kiszk kiszk May 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good check!!

// DAGScheduler event loop thread does not have an active SparkSession, the `confGetter`
// will return `fallbackConf` which is unexpected. Here we prevent it from happening.
val schedulerEventLoopThread =
SparkContext.getActive.get.dagScheduler.eventProcessLoop.eventThread
if (schedulerEventLoopThread.getId == Thread.currentThread().getId) {
Expand Down