Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
82fd38d
[SPARK-5200] Disable web UI in Hive ThriftServer tests
JoshRosen Jan 12, 2015
ef9224e
[SPARK-5102][Core]subclass of MapStatus needs to be registered with Kryo
lianhuiwang Jan 12, 2015
13e610b
SPARK-4159 [BUILD] Addendum: improve running of single test after ena…
srowen Jan 12, 2015
a3978f3
[SPARK-5078] Optionally read from SPARK_LOCAL_HOSTNAME
marmbrus Jan 12, 2015
aff49a3
SPARK-5172 [BUILD] spark-examples-***.jar shades a wrong Hadoop distr…
srowen Jan 12, 2015
3aed305
[SPARK-4999][Streaming] Change storeInBlockManager to false by default
jerryshao Jan 12, 2015
5d9fa55
[SPARK-5049][SQL] Fix ordering of partition columns in ParquetTableScan
marmbrus Jan 12, 2015
1e42e96
[SPARK-5138][SQL] Ensure schema can be inferred from a namedtuple
mulby Jan 13, 2015
f7741a9
[SPARK-5006][Deploy]spark.port.maxRetries doesn't work
WangTaoTheTonic Jan 13, 2015
9dea64e
[SPARK-4697][YARN]System properties should override environment varia…
WangTaoTheTonic Jan 13, 2015
39e333e
[SPARK-5131][Streaming][DOC]: There is a discrepancy in WAL implement…
uncleGen Jan 13, 2015
8ead999
[SPARK-5223] [MLlib] [PySpark] fix MapConverter and ListConverter in …
Jan 13, 2015
6463e0b
[SPARK-4912][SQL] Persistent tables for the Spark SQL data sources api
yhuai Jan 13, 2015
14e3f11
[SPARK-5168] Make SQLConf a field rather than mixin in SQLContext
rxin Jan 13, 2015
f996909
[SPARK-5123][SQL] Reconcile Java/Scala API for data types.
rxin Jan 14, 2015
d5eeb35
[SPARK-5167][SQL] Move Row into sql package and make it usable for Java.
rxin Jan 14, 2015
a3f7421
[SPARK-5248] [SQL] move sql.types.decimal.Decimal to sql.types.Decimal
adrian-wang Jan 14, 2015
81f72a0
[SPARK-5211][SQL]Restore HiveMetastoreTypes.toDataType
yhuai Jan 14, 2015
38bdc99
[SQL] some comments fix for GROUPING SETS
adrian-wang Jan 14, 2015
5840f54
[SPARK-2909] [MLlib] [PySpark] SparseVector in pyspark now supports i…
MechCoder Jan 14, 2015
9d4449c
[SPARK-5228][WebUI] Hide tables for "Active Jobs/Completed Jobs/Faile…
sarutak Jan 14, 2015
259936b
[SPARK-4014] Add TaskContext.attemptNumber and deprecate TaskContext.…
JoshRosen Jan 14, 2015
2fd7f72
[SPARK-5235] Make SQLConf Serializable
alexbaretta Jan 14, 2015
76389c5
[SPARK-5234][ml]examples for ml don't have sparkContext.stop
Jan 14, 2015
13d2406
[SPARK-5254][MLLIB] Update the user guide to position spark.ml better
mengxr Jan 15, 2015
cfa397c
[SPARK-5193][SQL] Tighten up SQLContext API
rxin Jan 15, 2015
6abc45e
[SPARK-5254][MLLIB] remove developers section from spark.ml guide
mengxr Jan 15, 2015
4b325c7
[SPARK-5193][SQL] Tighten up HiveContext API
rxin Jan 15, 2015
3c8650c
[SPARK-5224] [PySpark] improve performance of parallelize list/ndarray
Jan 15, 2015
1881431
[SPARK-5274][SQL] Reconcile Java and Scala UDFRegistration.
rxin Jan 16, 2015
65858ba
[Minor] Fix tiny typo in BlockManager
sarutak Jan 16, 2015
96c2c71
[SPARK-4857] [CORE] Adds Executor membership events to SparkListener
Jan 16, 2015
a79a9f9
[SPARK-4092] [CORE] Fix InputMetrics for coalesce'd Rdds
Jan 16, 2015
2be82b1
[SPARK-1507][YARN]specify # cores for ApplicationMaster
WangTaoTheTonic Jan 16, 2015
e200ac8
[SPARK-5201][CORE] deal with int overflow in the ParallelCollectionRD…
advancedxy Jan 16, 2015
f6b852a
[DOCS] Fix typo in return type of cogroup
srowen Jan 16, 2015
e8422c5
[SPARK-5231][WebUI] History Server shows wrong job submission time.
sarutak Jan 16, 2015
ecf943d
[WebUI] Fix collapse of WebUI layout
sarutak Jan 16, 2015
d05c9ee
[SPARK-4923][REPL] Add Developer API to REPL to allow re-publishing t…
Jan 16, 2015
fd3a8a1
[SPARK-733] Add documentation on use of accumulators in lazy transfor…
Jan 16, 2015
ee1c1f3
[SPARK-4937][SQL] Adding optimization to simplify the And, Or condit…
scwf Jan 16, 2015
61b427d
[SPARK-5193][SQL] Remove Spark SQL Java-specific API.
rxin Jan 17, 2015
f3bfc76
[SQL][minor] Improved Row documentation.
rxin Jan 17, 2015
c1f3c27
[SPARK-4937][SQL] Comment for the newly optimization rules in `Boolea…
scwf Jan 17, 2015
6999910
[SPARK-5096] Use sbt tasks instead of vals to get hadoop version
marmbrus Jan 18, 2015
e7884bc
[SQL][Minor] Added comments and examples to explain BooleanSimplifica…
rxin Jan 18, 2015
e12b5b6
MAINTENANCE: Automated closing of pull requests.
pwendell Jan 18, 2015
ad16da1
[HOTFIX]: Minor clean up regarding skipped artifacts in build files.
pwendell Jan 18, 2015
1727e08
[SPARK-5279][SQL] Use java.math.BigDecimal as the exposed Decimal type.
rxin Jan 18, 2015
1a200a3
[SQL][Minor] Update sql doc according to data type APIs changes
scwf Jan 18, 2015
1955645
[SQL][minor] Put DataTypes.java in java dir.
rxin Jan 19, 2015
7dbf1fd
[SQL] fix typo in class description
Jan 19, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SQL][Minor] Update sql doc according to data type APIs changes
Follow up of apache#3925
/cc rxin

Author: scwf <[email protected]>

Closes apache#4095 from scwf/sql-doc and squashes the following commits:

97e311b [scwf] update sql doc since now expose only one version of the data type APIs
  • Loading branch information
scwf authored and rxin committed Jan 18, 2015
commit 1a200a3eea9257677baf9c1c66311b5efd6c80fe
41 changes: 21 additions & 20 deletions docs/sql-programming-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -1333,9 +1333,9 @@ import org.apache.spark.sql._
<div data-lang="java" markdown="1">

All data types of Spark SQL are located in the package of
`org.apache.spark.sql.api.java`. To access or create a data type,
`org.apache.spark.sql.types`. To access or create a data type,
please use factory methods provided in
`org.apache.spark.sql.api.java.DataType`.
`org.apache.spark.sql.types.DataTypes`.

<table class="table">
<tr>
Expand All @@ -1346,109 +1346,110 @@ please use factory methods provided in
<td> <b>ByteType</b> </td>
<td> byte or Byte </td>
<td>
DataType.ByteType
DataTypes.ByteType
</td>
</tr>
<tr>
<td> <b>ShortType</b> </td>
<td> short or Short </td>
<td>
DataType.ShortType
DataTypes.ShortType
</td>
</tr>
<tr>
<td> <b>IntegerType</b> </td>
<td> int or Integer </td>
<td>
DataType.IntegerType
DataTypes.IntegerType
</td>
</tr>
<tr>
<td> <b>LongType</b> </td>
<td> long or Long </td>
<td>
DataType.LongType
DataTypes.LongType
</td>
</tr>
<tr>
<td> <b>FloatType</b> </td>
<td> float or Float </td>
<td>
DataType.FloatType
DataTypes.FloatType
</td>
</tr>
<tr>
<td> <b>DoubleType</b> </td>
<td> double or Double </td>
<td>
DataType.DoubleType
DataTypes.DoubleType
</td>
</tr>
<tr>
<td> <b>DecimalType</b> </td>
<td> java.math.BigDecimal </td>
<td>
DataType.DecimalType
DataTypes.createDecimalType()<br />
DataTypes.createDecimalType(<i>precision</i>, <i>scale</i>).
</td>
</tr>
<tr>
<td> <b>StringType</b> </td>
<td> String </td>
<td>
DataType.StringType
DataTypes.StringType
</td>
</tr>
<tr>
<td> <b>BinaryType</b> </td>
<td> byte[] </td>
<td>
DataType.BinaryType
DataTypes.BinaryType
</td>
</tr>
<tr>
<td> <b>BooleanType</b> </td>
<td> boolean or Boolean </td>
<td>
DataType.BooleanType
DataTypes.BooleanType
</td>
</tr>
<tr>
<td> <b>TimestampType</b> </td>
<td> java.sql.Timestamp </td>
<td>
DataType.TimestampType
DataTypes.TimestampType
</td>
</tr>
<tr>
<td> <b>DateType</b> </td>
<td> java.sql.Date </td>
<td>
DataType.DateType
DataTypes.DateType
</td>
</tr>
<tr>
<td> <b>ArrayType</b> </td>
<td> java.util.List </td>
<td>
DataType.createArrayType(<i>elementType</i>)<br />
DataTypes.createArrayType(<i>elementType</i>)<br />
<b>Note:</b> The value of <i>containsNull</i> will be <i>true</i><br />
DataType.createArrayType(<i>elementType</i>, <i>containsNull</i>).
DataTypes.createArrayType(<i>elementType</i>, <i>containsNull</i>).
</td>
</tr>
<tr>
<td> <b>MapType</b> </td>
<td> java.util.Map </td>
<td>
DataType.createMapType(<i>keyType</i>, <i>valueType</i>)<br />
DataTypes.createMapType(<i>keyType</i>, <i>valueType</i>)<br />
<b>Note:</b> The value of <i>valueContainsNull</i> will be <i>true</i>.<br />
DataType.createMapType(<i>keyType</i>, <i>valueType</i>, <i>valueContainsNull</i>)<br />
DataTypes.createMapType(<i>keyType</i>, <i>valueType</i>, <i>valueContainsNull</i>)<br />
</td>
</tr>
<tr>
<td> <b>StructType</b> </td>
<td> org.apache.spark.sql.api.java.Row </td>
<td>
DataType.createStructType(<i>fields</i>)<br />
DataTypes.createStructType(<i>fields</i>)<br />
<b>Note:</b> <i>fields</i> is a List or an array of StructFields.
Also, two fields with the same name are not allowed.
</td>
Expand All @@ -1458,7 +1459,7 @@ please use factory methods provided in
<td> The value type in Java of the data type of this field
(For example, int for a StructField with the data type IntegerType) </td>
<td>
DataType.createStructField(<i>name</i>, <i>dataType</i>, <i>nullable</i>)
DataTypes.createStructField(<i>name</i>, <i>dataType</i>, <i>nullable</i>)
</td>
</tr>
</table>
Expand Down