Skip to content

[SPARK-7319][SQL] Improve the output from DataFrame.show() #5865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kaka1992 wants to merge 13 commits into from
+112 −44
Closed

[SPARK-7319][SQL] Improve the output from DataFrame.show() #5865

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
d11d5b9
[SPARK-7294] ADD BETWEEN
May 1, 2015
baf839b
[SPARK-7294] ADD BETWEEN
May 2, 2015
7d62368
[SPARK-7294] ADD BETWEEN
May 2, 2015
30ac311
[SPARK-7294] ADD BETWEEN
May 2, 2015
b6e690b
Merge remote-tracking branch 'upstream/master' into df.show
May 2, 2015
ced487a
update pep8
May 2, 2015
7394fd5
update test show
May 3, 2015
03ef434
update
May 3, 2015
159b3d5
update
May 3, 2015
84aec3e
Update python dataFrame show test and add empty df unit test.
May 4, 2015
734369c
Update python dataFrame show test and add empty df unit test.
May 4, 2015
a1338f6
Update python dataFrame show test and add empty df unit test.
May 4, 2015
c79204b
Update
May 4, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 22 additions & 6 deletions sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ package org.apache.spark.sql
import java.io.CharArrayWriter
import java.sql.DriverManager


import scala.collection.JavaConversions._
import scala.language.implicitConversions
import scala.reflect.ClassTag
Expand All @@ -28,6 +29,7 @@ import scala.util.control.NonFatal

import com.fasterxml.jackson.core.JsonFactory

import org.apache.commons.lang3.StringUtils
import org.apache.spark.annotation.{DeveloperApi, Experimental}
import org.apache.spark.api.java.JavaRDD
import org.apache.spark.api.python.SerDeUtil
Expand Down Expand Up @@ -175,6 +177,7 @@ class DataFrame private[sql](
* @param numRows Number of rows to show
*/
private[sql] def showString(numRows: Int): String = {
val sb = new StringBuilder
Copy link
Contributor

@rxin rxin May 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be great to create unit test for showString in DataFrameSuite.

val data = take(numRows)
val numCols = schema.fieldNames.length

Expand All @@ -194,12 +197,25 @@ class DataFrame private[sql](
}
}

// Pad the cells
rows.map { row =>
row.zipWithIndex.map { case (cell, i) =>
String.format(s"%-${colWidths(i)}s", cell)
}.mkString(" ")
}.mkString("\n")
// Create SeparateLine
val sep: String = colWidths.map("-" * _).addString(sb, "+", "+", "+\n").toString()

// column names
rows.head.zipWithIndex.map { case (cell, i) =>
StringUtils.leftPad(cell.toString, colWidths(i))
}.addString(sb, "|", "|", "|\n")

sb.append(sep)

// data
rows.tail.map {
_.zipWithIndex.map { case (cell, i) =>
StringUtils.leftPad(cell.toString, colWidths(i))
}.addString(sb, "|", "|", "|\n")
}

sb.append(sep)
sb.toString()
}

override def toString: String = {
Expand Down
5 changes: 5 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -598,6 +598,11 @@ class DataFrameSuite extends QueryTest {
testData.select($"*").show(1000)
}

test("SPARK-7319 showString") {
assert(testData.select($"*").showString(1).split("\n") === Seq("+---+-----+",
Copy link
Contributor

@rxin rxin May 4, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can use multi line string in Scala, especially with stripMargin.

http://alvinalexander.com/scala/scala-multiline-strings-heredoc-syntax

"|key|value|", "+---+-----+", "| 1| 1|", "+---+-----+"))
}

test("createDataFrame(RDD[Row], StructType) should convert UDTs (SPARK-6672)") {
val rowRDD = TestSQLContext.sparkContext.parallelize(Seq(Row(new ExamplePoint(1.0, 2.0))))
val schema = StructType(Array(StructField("point", new ExamplePointUDT(), false)))
Expand Down