Skip to content

[SPARK-17153][SQL] Should read partition data when reading new files in filestream without globbing #14803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 10 commits into from
Closed

[SPARK-17153][SQL] Should read partition data when reading new files in filestream without globbing #14803

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
2771d71
Pass path as basePath for partitionSpec creation if path is not globb…
viirya Aug 25, 2016
0d841e2
Merge remote-tracking branch 'upstream/master' into filestreamsource-…
viirya Aug 29, 2016
6adf2e2
Address comment.
viirya Aug 30, 2016
f7c9067
Merge remote-tracking branch 'upstream/master' into filestreamsource-…
viirya Sep 19, 2016
04b61c7
Fix a test.
viirya Sep 19, 2016
23ba9a2
Address reviewer comments.
viirya Sep 20, 2016
5b101ab
Fix wrong comment, slightly tailor the new test.
viirya Sep 20, 2016
541dfdc
Update programming guide.
viirya Sep 20, 2016
9d16631
Merge remote-tracking branch 'upstream/master' into filestreamsource-…
viirya Sep 22, 2016
e21536e
Update doc and enhance the added test.
viirya Sep 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge remote-tracking branch 'upstream/master' into filestreamsource-… 
…option
  • Loading branch information
@viirya
viirya committed Sep 22, 2016
commit 9d16631e87789d5eca4a5737a3d96e15f1623c4b
7 changes: 4 additions & 3 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,8 @@ class FileStreamSource(
Map()
}}

private val metadataLog = new HDFSMetadataLog[Array[FileEntry]](sparkSession, metadataPath)

private val metadataLog =
new FileStreamSourceLog(FileStreamSourceLog.VERSION, sparkSession, metadataPath)
private var maxBatchId = metadataLog.getLatest().map(_._1).getOrElse(-1L)

/** Maximum number of new files to be considered in each batch */
Expand Down Expand Up @@ -140,7 +140,8 @@ class FileStreamSource(
userSpecifiedSchema = Some(schema),
className = fileFormatClassName,
options = optionsWithPartitionBasePath)
Dataset.ofRows(sparkSession, LogicalRelation(newDataSource.resolveRelation()))
Dataset.ofRows(sparkSession, LogicalRelation(newDataSource.resolveRelation(
checkFilesExist = false)))
}

/**
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.