Skip to content

[SPARK-24807][CORE] Adding files/jars twice: output a warning and add a note #21771

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 2 commits into from
Closed

[SPARK-24807][CORE] Adding files/jars twice: output a warning and add a note #21771

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0942fd2
Output warning for second call of addFile/addJar
MaxGekk Jul 14, 2018
4a583c6
Improve comments
MaxGekk Jul 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Improve comments
  • Loading branch information
@MaxGekk
MaxGekk committed Jul 14, 2018
commit 4a583c60e6fdf0f98b0d7b5afa4e81af64cb5ebb
2 changes: 1 addition & 1 deletion R/pkg/R/context.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,7 @@ setCheckpointDirSC <- function(sc, dirName) {
#' Currently directories are only supported for Hadoop-supported filesystems.
#' Refer Hadoop-supported filesystems at \url{https://wiki.apache.org/hadoop/HCFS}.
#'
#' Note: A path can be added only once. Second addition of the same path is ignored.
#' Note: A path can be added only once. Subsequent additions of the same path are ignored.
#'
#' @rdname spark.addFile
#' @param path The path of the file to be added
Expand Down
6 changes: 3 additions & 3 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1497,7 +1497,7 @@ class SparkContext(config: SparkConf) extends Logging {
* filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
* use `SparkFiles.get(fileName)` to find its download location.
*
* @note A path can be added only once. Second addition of the same path is ignored.
* @note A path can be added only once. Subsequent additions of the same path are ignored.
*/
def addFile(path: String): Unit = {
addFile(path, false)
Expand All @@ -1519,7 +1519,7 @@ class SparkContext(config: SparkConf) extends Logging {
* @param recursive if true, a directory can be given in `path`. Currently directories are
* only supported for Hadoop-supported filesystems.
*
* @note A path can be added only once. Second addition of the same path is ignored.
* @note A path can be added only once. Subsequent additions of the same path are ignored.
*/
def addFile(path: String, recursive: Boolean): Unit = {
val uri = new Path(path).toUri
Expand Down Expand Up @@ -1811,7 +1811,7 @@ class SparkContext(config: SparkConf) extends Logging {
* @param path can be either a local file, a file in HDFS (or other Hadoop-supported filesystems),
* an HTTP, HTTPS or FTP URI, or local:/path for a file on every worker node.
*
* @note A path can be added only once. Second addition of the same path is ignored.
* @note A path can be added only once. Subsequent additions of the same path are ignored.
*/
def addJar(path: String) {
def addJarFile(file: File): String = {
Expand Down
6 changes: 3 additions & 3 deletions core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -669,7 +669,7 @@ class JavaSparkContext(val sc: SparkContext)
* filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
* use `SparkFiles.get(fileName)` to find its download location.
*
* @note A path can be added only once. Second addition of the same path is ignored.
* @note A path can be added only once. Subsequent additions of the same path are ignored.
*/
def addFile(path: String) {
sc.addFile(path)
Expand All @@ -684,7 +684,7 @@ class JavaSparkContext(val sc: SparkContext)
* A directory can be given if the recursive option is set to true. Currently directories are only
* supported for Hadoop-supported filesystems.
*
* @note A path can be added only once. Second addition of the same path is ignored.
* @note A path can be added only once. Subsequent additions of the same path are ignored.
*/
def addFile(path: String, recursive: Boolean): Unit = {
sc.addFile(path, recursive)
Expand All @@ -695,7 +695,7 @@ class JavaSparkContext(val sc: SparkContext)
* The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported
* filesystems), or an HTTP, HTTPS or FTP URI.
*
* @note A path can be added only once. Second addition of the same path is ignored.
* @note A path can be added only once. Subsequent additions of the same path are ignored.
*/
def addJar(path: String) {
sc.addJar(path)
Expand Down
4 changes: 2 additions & 2 deletions python/pyspark/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -847,7 +847,7 @@ def addFile(self, path, recursive=False):
A directory can be given if the recursive option is set to True.
Currently directories are only supported for Hadoop-supported filesystems.

.. note:: A path can be added only once. Second addition of the same path is ignored.
.. note:: A path can be added only once. Subsequent additions of the same path are ignored.

>>> from pyspark import SparkFiles
>>> path = os.path.join(tempdir, "test.txt")
Expand All @@ -870,7 +870,7 @@ def addPyFile(self, path):
file, a file in HDFS (or other Hadoop-supported filesystems), or an
HTTP, HTTPS or FTP URI.

.. note:: A path can be added only once. Second addition of the same path is ignored.
.. note:: A path can be added only once. Subsequent additions of the same path are ignored.
"""
self.addFile(path)
(dirname, filename) = os.path.split(path) # dirname may be directory or HDFS/S3 prefix
Expand Down