Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[SQL] Is a config modifiable at runtime
In the PR, I propose to extend `RuntimeConfig` by new method `isModifiable()` which returns `true` if a config parameter can be modified at runtime (in notebooks). For static SQL and core parameters, the method returns `false`.
  • Loading branch information
MaxGekk committed Jul 8, 2018
commit c7819e60ff674e929d47b4c7247dea0f85349ac5
6 changes: 6 additions & 0 deletions python/pyspark/sql/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,12 @@ def _checkType(self, obj, identifier):
raise TypeError("expected %s '%s' to be a string (was '%s')" %
(identifier, obj, type(obj).__name__))

@ignore_unicode_prefix
@since(2.4)
def isModifiable(self, key):
"""Is the configuration property modifiable or not."""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not make all doc comments across languages identical?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From which language would you prefer the comment? or maybe the comment should contain more information? Please, let me know if something is not clear, and we should explain that in the comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am going to change to:

Indicates that the configuration property with the given key modifiable in the current session or not.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor language tweak suggestion:

Indicates whether the configuration property with the given key is modifiable in the current session.

return self._jconf.isModifiable(key)


def _test():
import os
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1892,4 +1892,8 @@ class SQLConf extends Serializable with Logging {
}
cloned
}

def isModifiable(key: String): Boolean = {
sqlConfEntries.containsKey(key) && !staticConfKeys.contains(key)
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,13 @@ class RuntimeConfig private[sql](sqlConf: SQLConf = new SQLConf) {
sqlConf.unsetConf(key)
}

/**
* Can the configuration property be modified at runtime.
*
* @since 2.4.0
*/
def isModifiable(key: String): Boolean = sqlConf.isModifiable(key)

/**
* Returns whether a particular key is set.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,4 +54,15 @@ class RuntimeConfigSuite extends SparkFunSuite {
conf.get("k1")
}
}

test("is a config parameter modifiable") {
val conf = newConf()

// SQL configs
assert(!conf.isModifiable("spark.sql.sources.schemaStringLengthThreshold"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add a case for nonexistent sql conf key?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a couple tests

assert(conf.isModifiable("spark.sql.streaming.checkpointLocation"))
// Core configs
assert(!conf.isModifiable("spark.task.cpus"))
assert(!conf.isModifiable("spark.executor.cores"))
}
}