Skip to content

[SPARK-12562][SQL] DataFrame.write.format(text) requires the column name to be called value #10515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
xguo27 wants to merge 3 commits into from
Closed

[SPARK-12562][SQL] DataFrame.write.format(text) requires the column name to be called value #10515

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0da6fc4
[SPARK-12562][SQL] DataFrame.write.format(text) requires the column n…
xguo27 Dec 30, 2015
5f9aaa9
[SPARK-12562] add unit test
xguo27 Dec 30, 2015
d01ecac
[SPARK-12562] modify unit test
xguo27 Dec 30, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
[SPARK-12562] modify unit test
  • Loading branch information
@xguo27
xguo27 committed Dec 30, 2015
commit d01ecac4966896cdd05698006c032d58f511c36b
15 changes: 2 additions & 13 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ class TextSuite extends QueryTest with SharedSQLContext {
verifyFrame(sqlContext.read.text(testFile))
}

test("writing") {
val df = sqlContext.read.text(testFile)
test("SPARK-12562 verify write.text() can handle column name beyond `value`") {
val df = sqlContext.read.text(testFile).withColumnRenamed("value", "adwrasdf")
Copy link
Member

@viirya viirya Dec 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be failed because the later method verifyFrame will check if the df read in has schema like new StructType().add("value", StringType). You could update verifyFrame to check if it has only one StringType column.

Copy link
Contributor Author

@xguo27 xguo27 Dec 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After write.text(), the local text file actually does not carry the schema name like JSON does. When reading back the text file and then call verifyFrame, it will always have value as the column name.

Copy link
Member

@viirya viirya Dec 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. it is right.


val tempFile = Utils.createTempDir()
tempFile.delete()
Expand All @@ -58,17 +58,6 @@ class TextSuite extends QueryTest with SharedSQLContext {
}
}

test("SPARK-12562 verify write.text() can handle column name beyond `value`") {
val df = sqlContext.read.text(testFile).withColumnRenamed("value", "col1")

val tempFile = Utils.createTempDir()
tempFile.delete()
df.write.text(tempFile.getCanonicalPath)
verifyFrame(sqlContext.read.text(tempFile.getCanonicalPath))

Utils.deleteRecursively(tempFile)
}

private def testFile: String = {
Thread.currentThread().getContextClassLoader.getResource("text-suite.txt").toString
}
Expand Down