Skip to content

[SPARK-3930] [SPARK-3933] Support fixed-precision decimal in SQL, and some optimizations #2983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mateiz wants to merge 11 commits into from
Closed

[SPARK-3930] [SPARK-3933] Support fixed-precision decimal in SQL, and some optimizations #2983

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
ec0a947
Make the result of AVG on Decimals be Decimal, not Double
mateiz Oct 2, 2014
7af0c3b
Add optional precision and scale to DecimalType, but use Unlimited fo…
mateiz Oct 3, 2014
81db9cb
Added mutable Decimal that will be more efficient for small precisions
mateiz Oct 5, 2014
2118c0d
Some test and bug fixes
mateiz Oct 28, 2014
b28933d
Support decimal precision/scale in Hive metastore
mateiz Oct 29, 2014
d1d9d68
Fix compile error and test issues after rebase
mateiz Oct 29, 2014
4dc6bae
Fix decimal support in PySpark
mateiz Oct 30, 2014
eb84820
Support reading/writing decimals as fixed-length binary in Parquet
mateiz Oct 31, 2014
31f915e
Implement Davies's suggestions in Python
mateiz Oct 31, 2014
227f24a
Review comments
mateiz Nov 1, 2014
35e6b02
Fix issues after merge
mateiz Nov 1, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Some test and bug fixes
  • Loading branch information
@mateiz
mateiz committed Nov 1, 2014
commit 2118c0dcd8d021f876a438bc87f2b30cf432a9b7
3 changes: 2 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ trait HiveTypeCoercion {
}
}


// scalastyle:off
/**
* Calculates and propagates precision for fixed-precision decimals. Hive has a number of
* rules for this based on the SQL standard and MS SQL:
Expand Down Expand Up @@ -313,6 +313,7 @@ trait HiveTypeCoercion {
* - FLOAT and DOUBLE cause fixed-length decimals to turn into DOUBLE (this is the same as Hive,
* but note that unlimited decimals are considered bigger than doubles in WidenTypes)
*/
// scalastyle:on
object DecimalPrecision extends Rule[LogicalPlan] {
import scala.math.{max, min}

Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ object Literal {
case s: Short => Literal(s, ShortType)
case s: String => Literal(s, StringType)
case b: Boolean => Literal(b, BooleanType)
case d: BigDecimal => Literal(d, DecimalType.Unlimited)
case d: BigDecimal => Literal(Decimal(d), DecimalType.Unlimited)
case d: Decimal => Literal(d, DecimalType.Unlimited)
case t: Timestamp => Literal(t, TimestampType)
case d: Date => Literal(d, DateType)
Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ object DataType {
// NOTE: Map fields must be sorted in alphabetical order to keep consistent with the Python side.
private def parseDataType(json: JValue): DataType = json match {
case JString(name) =>
PrimitiveType.nameToType(name) // TODO: Doesn't work for fixed-precision decimal
PrimitiveType.nameToType(name)

case JSortedObject(
("containsNull", JBool(n)),
Expand Down
3 changes: 2 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoin.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,8 @@ case class BroadcastHashJoin(

@transient
private val broadcastFuture = future {
val input: Array[Row] = buildPlan.executeCollect()
// Note that we use .execute().collect() because we don't want to convert data to Scala types
val input: Array[Row] = buildPlan.execute().map(_.copy()).collect()
val hashed = HashedRelation(input.iterator, buildSideKeyGenerator, input.length)
sparkContext.broadcast(hashed)
}
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -393,7 +393,7 @@ private[sql] object JsonRDD extends Logging {
case IntegerType => value.asInstanceOf[IntegerType.JvmType]
case LongType => toLong(value)
case DoubleType => toDouble(value)
case DecimalType() => toDecimal(value) // TODO: fix precision and scale
case DecimalType() => toDecimal(value)
case BooleanType => value.asInstanceOf[BooleanType.JvmType]
case NullType => null

Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/ScalaReflectionRelationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ class ScalaReflectionRelationSuite extends FunSuite {

assert(sql("SELECT * FROM reflectData").collect().head ===
Seq("a", 1, 1L, 1.toFloat, 1.toDouble, 1.toShort, 1.toByte, true,
BigDecimal(1), new Timestamp(12345), Seq(1,2,3)))
BigDecimal(1), new Date(12345), new Timestamp(12345), Seq(1,2,3)))
}

test("query case class RDD with nulls") {
Expand Down