Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
5f70b1d
Add examples (extend) in each function and improve documentation with…
HyukjinKwon Oct 17, 2016
77abafa
aggregate OK
HyukjinKwon Oct 18, 2016
8cd6d80
xml OK
HyukjinKwon Oct 18, 2016
2614572
arithmetic OK
HyukjinKwon Oct 18, 2016
29d0262
bitwiseExpressions OK and double-check others
HyukjinKwon Oct 18, 2016
710c68e
CallMethodViaReflection OK
HyukjinKwon Oct 18, 2016
1d44ec3
Cast OK
HyukjinKwon Oct 18, 2016
78b1fc7
collectionOperations OK
HyukjinKwon Oct 18, 2016
e2062af
complexTypeCreator OK and double check others
HyukjinKwon Oct 19, 2016
e9672db
conditionalExpressions OK
HyukjinKwon Oct 19, 2016
9baa847
datetimeExpressions OK
HyukjinKwon Oct 19, 2016
fff85f6
generators OK
HyukjinKwon Oct 19, 2016
2462775
InputFileName OK
HyukjinKwon Oct 19, 2016
10892fb
jsonExpressions OK
HyukjinKwon Oct 19, 2016
45e7f99
mathExpressions OK, double check others and fix scala style
HyukjinKwon Oct 19, 2016
1d69e40
misc OK, double-check others and ignore a test in SQLQuerySuite for now
HyukjinKwon Oct 20, 2016
9b24e78
MonotonicallyIncreasingID OK
HyukjinKwon Oct 20, 2016
ed1c839
nullExpressions OK
HyukjinKwon Oct 20, 2016
1535186
predicates OK
HyukjinKwon Oct 20, 2016
8efee7e
randomExpressions OK
HyukjinKwon Oct 20, 2016
a111d2a
regexpExpressions OK
HyukjinKwon Oct 20, 2016
a8ddcc2
SparkPartitionID OK
HyukjinKwon Oct 20, 2016
99b5658
tringExpressions OK
HyukjinKwon Oct 20, 2016
a29472e
windowExpressions OK
HyukjinKwon Oct 20, 2016
73ccda0
conditionalExpressions OK, double-check others and fix tests
HyukjinKwon Oct 20, 2016
d927bff
double-check
HyukjinKwon Oct 20, 2016
91d2ab5
Fix tests in SQLQuerySuite and DDLSuite first
HyukjinKwon Oct 21, 2016
7841860
Take out a space after `Extended Usage:`.
HyukjinKwon Oct 21, 2016
01eecfe
Consistent spacing in Examples
HyukjinKwon Oct 21, 2016
1979d92
Remove repeated _FUNC_, consolidate usages and simplify the arguments
HyukjinKwon Oct 21, 2016
d56ac66
Fix tests DDLSuite, SQLQuerySuite and hive/SQLQuerySuite with consist…
HyukjinKwon Oct 21, 2016
8a773d4
Fix scala style.
HyukjinKwon Oct 21, 2016
f71e39e
double-check
HyukjinKwon Oct 21, 2016
caa71c8
double-check again
HyukjinKwon Oct 22, 2016
5163a87
Address Sean's comments
HyukjinKwon Oct 22, 2016
f8d11aa
another double-check
HyukjinKwon Oct 22, 2016
da40f85
Fix the tests in DDLSuite
HyukjinKwon Oct 22, 2016
feafdc2
another double-check
HyukjinKwon Oct 23, 2016
2656c62
another double-check
HyukjinKwon Oct 23, 2016
15c02ca
Address comments and another double-check
HyukjinKwon Oct 24, 2016
035baef
revert mistakes
HyukjinKwon Oct 24, 2016
c55ecb6
another double-check and remove extra change for newlines
HyukjinKwon Oct 25, 2016
ad7b71d
Address comments
HyukjinKwon Oct 26, 2016
3965d00
Address comments
HyukjinKwon Oct 27, 2016
498d69c
Missed ones
HyukjinKwon Oct 27, 2016
400cee5
add back-quotes, fix typos and specify the types more specifically
HyukjinKwon Oct 27, 2016
2b437fe
address type and more details
HyukjinKwon Oct 29, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
double-check again
  • Loading branch information
HyukjinKwon committed Oct 22, 2016
commit caa71c89d67d7168a527c953c74be7bc5c4da27f
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ import org.apache.spark.util.Utils
usage = "_FUNC_(class, method[, arg1[, arg2 ..]]) - Calls method with reflection.",
extended = """
Arguments:
class - a string literal that represents full-qualified class name.
method - a string literal that represents method name.
class - a string literal that represents a fully-qualified class name.
method - a string literal that represents a method name.
arg - a string literal that represents arguments for the method.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it always literal?

Copy link
Member Author

@HyukjinKwon HyukjinKwon Oct 23, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, it seems arg is not. Let me try to find again such mistakes here.


Examples:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,11 +114,11 @@ object Cast {

/** Cast the child expression to the target data type. */
@ExpressionDescription(
usage = "_FUNC_(expr AS type) - Casts value expr to the target data type.",
usage = "_FUNC_(expr AS type) - Casts the value expr to the target data type.",
extended = """
Arguments:
expr - an expression of any type.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spark-sql> SELECT cast(array(1) as string), cast(struct(1) as string), cast(map(1,1) as string);
[1] [1] keys: [1], values: [1]

type - data type.
type - data type to cast expr into.

Examples:
> SELECT _FUNC_('10' as int);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,15 @@ import org.apache.spark.sql.types._
controls approximation accuracy at the cost of memory. Higher value of `accuracy` yields
better accuracy, `1.0/accuracy` is the relative error of the approximation.
When percentage is an array, each value of the percentage array must be between 0.0 and 1.0.
In this case, returns the approximate percentile array of column `col` at the given
percentage array.
""",
extended = """
Arguments:
col - a numeric expression.
percentage - a numeric literal or an array literal of numeric type that defines the
percentile. For example, 0.5 means 50-percentile.
accuracy - a numeric literal.
percentile between 0.0 and 1.0. For example, 0.5 means 50-percentile.
accuracy - a numeric literal of approximation accuracy.

Examples:
> SELECT percentile_approx(10.0, array(0.5, 0.4, 0.1), 100);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import org.apache.spark.sql.types._
*/
@ExpressionDescription(
usage = """
_FUNC_(expr[, isIgnoreNull]) - Returns the first value of expr for a group of rows.
_FUNC_(expr[, isIgnoreNull]) - Returns the first value of `expr` for a group of rows.
If isIgnoreNull is true, returns only non-null values.
""",
extended = """
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import org.apache.spark.sql.types._
*/
@ExpressionDescription(
usage = """
_FUNC_(expr[, isIgnoreNull]) - Returns the last value of expr for a group of rows.
_FUNC_(expr[, isIgnoreNull]) - Returns the last value of `expr` for a group of rows.
If isIgnoreNull is true, returns only non-null values.
""",
extended = """
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,7 @@ case class CreateNamedStructUnsafe(children: Seq[Expression]) extends Expression
Arguments:
text - a string expression that represents data to convert into map.
pairDelim - a string literal that defines the delimiter to separate each pair. For example, ",".
keyValueDelim - a string literal that defines the delimiter to separate key and value in the pair. For example, ":".
keyValueDelim - a string literal that defines the delimiter to separate the key and value in the pair. For example, ":".

Examples:
> SELECT _FUNC_('a:1,b:2,c:3', ',', ':');
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,14 @@ import org.apache.spark.sql.types._
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, expr3) - If expr1 is TRUE then IF() returns expr2; otherwise it returns expr3.",
extended = """
_FUNC_(expr1, expr2, expr3)

Arguments:
expr1 - a boolean expression.
expr2 - an expression of any type that represents the return value when expr1 is TRUE.
expr3 - an expression of any type that represents the return value when expr1 is FALSE.

Examples:
> SELECT _FUNC_(1 < 2, 'a', 'b');
a
Arguments:
expr1 - a boolean expression.
expr2 - an expression of any type that represents the return value when expr1 is TRUE.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when expr1 is TRUE -> when expr1 evaluates to true.

expr3 - an expression of any type that represents the return value when expr1 is FALSE.

Examples:
> SELECT _FUNC_(1 < 2, 'a', 'b');
a
""")
// scalastyle:on line.size.limit
case class If(predicate: Expression, trueValue: Expression, falseValue: Expression)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -400,11 +400,11 @@ case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCa

// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(timestamp, fmt) - Converts expr to a value of string in the format specified by the date format fmt.",
usage = "_FUNC_(timestamp, fmt) - Converts timestamp to a value of string in the format specified by the date format fmt.",
extended = """
Arguments:
timestamp - a timestamp expression.
fmt - string type expression that represents date/timestamp format used in java.text.SimpleDateFormat.
fmt - string type expression that represents the date/timestamp format available in java.text.SimpleDateFormat.

Examples:
> SELECT _FUNC_('2016-04-08', 'y');
Expand Down Expand Up @@ -443,7 +443,7 @@ case class DateFormatClass(left: Expression, right: Expression) extends BinaryEx
extended = """
Arguments:
expr - a date, string or timestamp expression.
pattern - a string expression that represents date/timestamp format used in
pattern - a string expression that represents the date/timestamp format available in
java.text.SimpleDateFormat.

Examples:
Expand Down Expand Up @@ -476,7 +476,7 @@ case class ToUnixTimestamp(timeExp: Expression, format: Expression) extends Unix
extended = """
Arguments:
expr - a date, string or timestamp expression.
pattern - a string expression that represents date/timestamp format used in
pattern - a string expression that represents the date/timestamp format available in
java.text.SimpleDateFormat.

Examples:
Expand Down Expand Up @@ -613,7 +613,7 @@ abstract class UnixTime extends BinaryExpression with ExpectsInputTypes {
extended = """
Arguments:
unix_time - a numeric expression.
format - a string expression that represents date/timestamp format used in
format - a string expression that represents the date/timestamp format available in
java.text.SimpleDateFormat.

Examples:
Expand Down Expand Up @@ -846,7 +846,7 @@ case class TimeAdd(start: Expression, interval: Expression)
extended = """
Arguments:
timestamp - a timestamp expression.
timezone - a string expression that represents time zone (e.g., 'Asia/Seoul').
timezone - a string expression that represents time zone. For example, 'Asia/Seoul'.

Examples:
> SELECT from_utc_timestamp('2016-08-31', 'Asia/Seoul');
Expand Down Expand Up @@ -970,7 +970,7 @@ case class AddMonths(startDate: Expression, numMonths: Expression)
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(timestamp1, timestamp2) - Returns number of months between dates expr1 and expr2.",
usage = "_FUNC_(timestamp1, timestamp2) - Returns number of months between dates timestamp1 and timestamp2.",
extended = """
Arguments:
timestamp1 - a timestamp expression.
Expand Down Expand Up @@ -1014,7 +1014,7 @@ case class MonthsBetween(date1: Expression, date2: Expression)
extended = """
Arguments:
timestamp - a timestamp expression.
timezone - a string expression that represents time zone (e.g., 'Asia/Seoul').
timezone - a string expression that represents time zone. For example, 'Asia/Seoul'.

Examples:
> SELECT _FUNC_('2016-08-31', 'Asia/Seoul');
Expand Down Expand Up @@ -1070,7 +1070,7 @@ case class ToUTCTimestamp(left: Expression, right: Expression)
* Returns the date part of a timestamp or string.
*/
@ExpressionDescription(
usage = "_FUNC_(date) - Extracts the date part of the date or datetime expression expr.",
usage = "_FUNC_(date) - Extracts the date part of the date or timestamp expression.",
extended = """
Arguments:
date - a date expression.
Expand Down Expand Up @@ -1184,7 +1184,7 @@ case class TruncDate(date: Expression, format: Expression)
* Returns the number of days from startDate to endDate.
*/
@ExpressionDescription(
usage = "_FUNC_(date1, date2) - Returns the number of days between expr1 and expr2.",
usage = "_FUNC_(date1, date2) - Returns the number of days between date1 and date2.",
extended = """
Arguments:
date1 - a date expression.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ trait StringRegexExpression extends ImplicitCastInputTypes {
extended = """
Arguments:
str - a string expression.
pattern - a string expression.
pattern - a string expression that defines a regular expression pattern.
""")
case class Like(left: Expression, right: Expression)
extends BinaryExpression with StringRegexExpression {
Expand Down Expand Up @@ -130,7 +130,7 @@ case class Like(left: Expression, right: Expression)
extended = """
Arguments:
str - a string expression.
regexp - a string expression.
regexp - a string expression that defines a regular expression pattern.
""")
case class RLike(left: Expression, right: Expression)
extends BinaryExpression with StringRegexExpression {
Expand Down Expand Up @@ -189,7 +189,7 @@ case class RLike(left: Expression, right: Expression)
extended = """
Arguments:
str - a string expression.
regex - a string expression.
regex - a string expression that defines a regular expression pattern.

Examples:
> SELECT _FUNC_('oneAtwoBthreeC', '[ABC]');
Expand Down Expand Up @@ -229,8 +229,8 @@ case class StringSplit(str: Expression, pattern: Expression)
extended = """
Arguments:
str - a string expression.
regexp - a string expression.
rep - a string expression.
regexp - a string expression that defines a regular expression pattern.
rep - a string expression that defines the replacement.

Examples:
> SELECT _FUNC_('100-200', '(\d+)', 'num');
Expand Down Expand Up @@ -340,7 +340,7 @@ case class RegExpReplace(subject: Expression, regexp: Expression, rep: Expressio
extended = """
Arguments:
str - a string expression.
regexp - a string expression.
regexp - a string expression that defines a regular expression pattern.
idx - a numeric expression that defines the index of a capturing group in regexp.

Examples:
Expand Down
Loading