-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-529] [core] [yarn] Add type-safe config keys to SparkConf. #10205
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
8b00d76
[RFC] Add type-safe config keys to SparkConf.
85ab5f1
Silence mima.
bda7e2b
Fix HiveContext.
93736c2
Add entry for spark.executor.instances.
c858fa8
Move code around, create object for core config keys.
c1de25f
Implement Andrew's suggestion (`import foo.config._`).
c07293f
Merge branch 'master' into conf-opts
6b6eb27
Reorder / regroup YARN configs.
86296fd
Move mima excludes to new, correct place.
d378ced
Merge branch 'master' into conf-opts
e91c91a
Fix typo.
db4a8ae
Merge branch 'master' into conf-opts
9878c39
Merge branch 'master' into conf-opts
c3d1b4a
Merge branch 'master' into conf-opts
d8ca3e1
Allow negative values in bytesConf.
e7f775a
Fix scalastyle.
b3a3b18
Fix broken merge.
3fd9c5e
Merge branch 'master' into conf-opts
d96d92d
Merge branch 'master' into conf-opts
6f7a6d7
Revert unneeded changes.
e5604d3
Merge branch 'master' into conf-opts
e2ec60e
Feedback.
0bbee02
Use a builder to construct config options.
69142ab
Merge branch 'master' into conf-opts
0d96e3a
Fix borked merge.
03dc641
Allow transformation of config data.
0f0147c
Fix a NPE.
885d4a6
Change the way seq configs are defined.
34fed1c
Merge branch 'master' into conf-opts
3a3968d
Merge branch 'master' into conf-opts
3d15e54
Use orNull.
b6928ea
Merge branch 'master' into conf-opts
fc48cf0
Revert sql changes (to be submitted separately).
be1daed
Move new classes to an "internal" package.
dbfed7d
Merge branch 'master' into conf-opts
31156aa
Add some more javadocs.
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Feedback.
- Loading branch information
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -33,7 +33,7 @@ import org.apache.spark.network.util.{ByteUnit, JavaUtils} | |
| * string value. It's usually `toString`. But sometimes, a custom converter | ||
| * is necessary. E.g., if T is List[String], `a, b, c` is better than | ||
| * `List(a, b, c)`. | ||
| * @param doc the document for the configuration | ||
| * @param doc the documentation for the configuration | ||
| * @param isPublic if this configuration is public to the user. If it's `false`, this | ||
| * configuration is only used internally and we should not expose it to the user. | ||
| * @tparam T the value type | ||
|
|
@@ -67,21 +67,27 @@ private[spark] class ConfigEntry[T] ( | |
| } | ||
| } | ||
|
|
||
| /** | ||
| * A config entry that does not have a default value. See the `optional` method in ConfigEntry. | ||
| */ | ||
| private[spark] class OptionalConfigEntry[T]( | ||
|
||
| override val key: String, | ||
| val _valueConverter: String => T, | ||
| val _stringConverter: T => String, | ||
| _valueConverter: String => T, | ||
| val rawStringConverter: T => String, | ||
| override val doc: String, | ||
| override val isPublic: Boolean) | ||
| extends ConfigEntry[Option[T]](key, None, s => Some(_valueConverter(s)), | ||
| null, doc, isPublic) { | ||
| extends ConfigEntry[Option[T]](key, None, s => Option(_valueConverter(s)), | ||
| v => v.map(rawStringConverter).orNull, doc, isPublic) { | ||
|
|
||
| override def readFrom(conf: SparkConf): Option[T] = { | ||
| conf.getOption(key).map(_valueConverter) | ||
| } | ||
|
|
||
| } | ||
|
|
||
| /** | ||
| * A config entry whose default value is defined by another config entry. | ||
| */ | ||
| private class FallbackConfigEntry[T] ( | ||
| override val key: String, | ||
| override val doc: String, | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you add java doc, mention something like: