Skip to content

[SPARK-21289][SQL][ML] Supports custom line separator for all text-based datasources #18581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 9 commits into from
Closed

[SPARK-21289][SQL][ML] Supports custom line separator for all text-based datasources #18581

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
fb589a3
Adds lineSep option for all text-based datasources
HyukjinKwon Jul 9, 2017
c4282bf
Remove perentheses in the tests
HyukjinKwon Aug 23, 2017
e7983d6
Make tests as functions so that other lineSep cna be tested easily.
HyukjinKwon Aug 24, 2017
16cc0c2
Minor cleanup
HyukjinKwon Aug 24, 2017
b006d65
`\n` by default, covering `\r`, `\r\n` and `\n`.
HyukjinKwon Aug 27, 2017
5ce9895
Fix libsvm and text too
HyukjinKwon Aug 27, 2017
bc65e6b
Add parameter descriptions in HadoopFileLinesReader
HyukjinKwon Aug 28, 2017
a240973
Minor cleanup
HyukjinKwon Oct 28, 2017
265dd48
Address comments
HyukjinKwon Dec 6, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix libsvm and text too
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Oct 28, 2017
commit 5ce9895d24a372fda8b18778b915c0c522ee21e1
6 changes: 2 additions & 4 deletions mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,10 @@ import org.apache.spark.util.SerializableConfiguration
private[libsvm] class LibSVMOutputWriter(
path: String,
dataSchema: StructType,
lineSeparator: Option[String],
lineSeparator: String,
context: TaskAttemptContext)
extends OutputWriter {

private val lineSep = lineSeparator.getOrElse("\n")

private val writer = CodecStreams.createOutputStreamWriter(context, new Path(path))

// This `asInstanceOf` is safe because it's guaranteed by `LibSVMFileFormat.verifySchema`
Expand All @@ -60,7 +58,7 @@ private[libsvm] class LibSVMOutputWriter(
writer.write(s" ${i + 1}:$v")
}

writer.write(lineSep)
writer.write(lineSeparator)
}

override def close(): Unit = {
Expand Down
6 changes: 2 additions & 4 deletions mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,10 +107,8 @@ object MLUtils extends Logging {
private[spark] def parseLibSVMFile(
sparkSession: SparkSession,
paths: Seq[String],
lineSep: Option[String]): RDD[(Double, Array[Int], Array[Double])] = {
val textOptions = lineSep
.map(sep => Map(TextOptions.LINE_SEPARATOR -> sep))
.getOrElse(Map.empty[String, String])
lineSeparator: String): RDD[(Double, Array[Int], Array[Double])] = {
val textOptions = Map(TextOptions.LINE_SEPARATOR -> lineSeparator)

val lines = sparkSession.baseRelationToDataFrame(
DataSource.apply(
Expand Down
2 changes: 1 addition & 1 deletion sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextRelation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ class SimpleTextSource extends TextBasedFileFormat with DataSourceRegister {
val projection = new InterpretedProjection(outputAttributes, inputAttributes)

val unsafeRowIterator =
new HadoopFileLinesReader(file, None, broadcastedHadoopConf.value.value).map { line =>
new HadoopFileLinesReader(file, "\n", broadcastedHadoopConf.value.value).map { line =>
val record = line.toString
new GenericInternalRow(record.split(",", -1).zip(fieldTypes).map {
case (v, dataType) =>
Expand Down