Skip to content

[SPARK-17963][SQL][Documentation] Add examples (extend) in each expression and improve documentation with arguments #15513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 47 commits into from
Closed

[SPARK-17963][SQL][Documentation] Add examples (extend) in each expression and improve documentation with arguments #15513

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
5f70b1d
Add examples (extend) in each function and improve documentation with…
HyukjinKwon Oct 17, 2016
77abafa
aggregate OK
HyukjinKwon Oct 18, 2016
8cd6d80
xml OK
HyukjinKwon Oct 18, 2016
2614572
arithmetic OK
HyukjinKwon Oct 18, 2016
29d0262
bitwiseExpressions OK and double-check others
HyukjinKwon Oct 18, 2016
710c68e
CallMethodViaReflection OK
HyukjinKwon Oct 18, 2016
1d44ec3
Cast OK
HyukjinKwon Oct 18, 2016
78b1fc7
collectionOperations OK
HyukjinKwon Oct 18, 2016
e2062af
complexTypeCreator OK and double check others
HyukjinKwon Oct 19, 2016
e9672db
conditionalExpressions OK
HyukjinKwon Oct 19, 2016
9baa847
datetimeExpressions OK
HyukjinKwon Oct 19, 2016
fff85f6
generators OK
HyukjinKwon Oct 19, 2016
2462775
InputFileName OK
HyukjinKwon Oct 19, 2016
10892fb
jsonExpressions OK
HyukjinKwon Oct 19, 2016
45e7f99
mathExpressions OK, double check others and fix scala style
HyukjinKwon Oct 19, 2016
1d69e40
misc OK, double-check others and ignore a test in SQLQuerySuite for now
HyukjinKwon Oct 20, 2016
9b24e78
MonotonicallyIncreasingID OK
HyukjinKwon Oct 20, 2016
ed1c839
nullExpressions OK
HyukjinKwon Oct 20, 2016
1535186
predicates OK
HyukjinKwon Oct 20, 2016
8efee7e
randomExpressions OK
HyukjinKwon Oct 20, 2016
a111d2a
regexpExpressions OK
HyukjinKwon Oct 20, 2016
a8ddcc2
SparkPartitionID OK
HyukjinKwon Oct 20, 2016
99b5658
tringExpressions OK
HyukjinKwon Oct 20, 2016
a29472e
windowExpressions OK
HyukjinKwon Oct 20, 2016
73ccda0
conditionalExpressions OK, double-check others and fix tests
HyukjinKwon Oct 20, 2016
d927bff
double-check
HyukjinKwon Oct 20, 2016
91d2ab5
Fix tests in SQLQuerySuite and DDLSuite first
HyukjinKwon Oct 21, 2016
7841860
Take out a space after `Extended Usage:`.
HyukjinKwon Oct 21, 2016
01eecfe
Consistent spacing in Examples
HyukjinKwon Oct 21, 2016
1979d92
Remove repeated _FUNC_, consolidate usages and simplify the arguments
HyukjinKwon Oct 21, 2016
d56ac66
Fix tests DDLSuite, SQLQuerySuite and hive/SQLQuerySuite with consist…
HyukjinKwon Oct 21, 2016
8a773d4
Fix scala style.
HyukjinKwon Oct 21, 2016
f71e39e
double-check
HyukjinKwon Oct 21, 2016
caa71c8
double-check again
HyukjinKwon Oct 22, 2016
5163a87
Address Sean's comments
HyukjinKwon Oct 22, 2016
f8d11aa
another double-check
HyukjinKwon Oct 22, 2016
da40f85
Fix the tests in DDLSuite
HyukjinKwon Oct 22, 2016
feafdc2
another double-check
HyukjinKwon Oct 23, 2016
2656c62
another double-check
HyukjinKwon Oct 23, 2016
15c02ca
Address comments and another double-check
HyukjinKwon Oct 24, 2016
035baef
revert mistakes
HyukjinKwon Oct 24, 2016
c55ecb6
another double-check and remove extra change for newlines
HyukjinKwon Oct 25, 2016
ad7b71d
Address comments
HyukjinKwon Oct 26, 2016
3965d00
Address comments
HyukjinKwon Oct 27, 2016
498d69c
Missed ones
HyukjinKwon Oct 27, 2016
400cee5
add back-quotes, fix typos and specify the types more specifically
HyukjinKwon Oct 27, 2016
2b437fe
address type and more details
HyukjinKwon Oct 29, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix tests DDLSuite, SQLQuerySuite and hive/SQLQuerySuite with consist… 
…ent names
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Oct 21, 2016
commit d56ac66c6549aa15d628498348dde358a7396ff3
72 changes: 36 additions & 36 deletions ...talyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,10 @@ case class DateSub(startDate: Expression, days: Expression)
}

@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the hour component of the string/timestamp/interval.",
usage = "_FUNC_(timestamp) - Returns the hour component of the string/timestamp/interval.",
extended = """
Arguments:
expr - a timestamp expression.
timestamp - a timestamp expression.

Examples:
> SELECT _FUNC_('2009-07-30 12:58:59');
Expand All @@ -171,10 +171,10 @@ case class Hour(child: Expression) extends UnaryExpression with ImplicitCastInpu
}

@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the minute component of the string/timestamp/interval.",
usage = "_FUNC_(timestamp) - Returns the minute component of the string/timestamp/interval.",
extended = """
Arguments:
expr - a timestamp expression.
timestamp - a timestamp expression.

Examples:
> SELECT _FUNC_('2009-07-30 12:58:59');
Expand All @@ -197,10 +197,10 @@ case class Minute(child: Expression) extends UnaryExpression with ImplicitCastIn
}

@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the second component of the string/timestamp/interval.",
usage = "_FUNC_(timestamp) - Returns the second component of the string/timestamp/interval.",
extended = """
Arguments:
expr - a timestamp expression.
timestamp - a timestamp expression.

Examples:
> SELECT _FUNC_('2009-07-30 12:58:59');
Expand All @@ -223,10 +223,10 @@ case class Second(child: Expression) extends UnaryExpression with ImplicitCastIn
}

@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the day of year of date/timestamp.",
usage = "_FUNC_(date) - Returns the day of year of date/timestamp.",
extended = """
Arguments:
expr - a date expression.
date - a date expression.

Examples:
> SELECT _FUNC_('2016-04-09');
Expand All @@ -249,10 +249,10 @@ case class DayOfYear(child: Expression) extends UnaryExpression with ImplicitCas
}

@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the year component of the date/timestamp/interval.",
usage = "_FUNC_(date) - Returns the year component of the date/timestamp/interval.",
extended = """
Arguments:
expr - a date expression.
date - a date expression.

Examples:
> SELECT _FUNC_('2016-07-30');
Expand All @@ -275,10 +275,10 @@ case class Year(child: Expression) extends UnaryExpression with ImplicitCastInpu
}

@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the quarter of the year for date, in the range 1 to 4.",
usage = "_FUNC_(date) - Returns the quarter of the year for date, in the range 1 to 4.",
extended = """
Arguments:
expr - a date expression.
date - a date expression.

Examples:
> SELECT _FUNC_('2016-08-31');
Expand All @@ -301,10 +301,10 @@ case class Quarter(child: Expression) extends UnaryExpression with ImplicitCastI
}

@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the month component of the date/timestamp/interval.",
usage = "_FUNC_(date) - Returns the month component of the date/timestamp/interval.",
extended = """
Arguments:
expr - a date expression.
date - a date expression.

Examples:
> SELECT _FUNC_('2016-07-30');
Expand All @@ -327,10 +327,10 @@ case class Month(child: Expression) extends UnaryExpression with ImplicitCastInp
}

@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the day of month of date/timestamp, or the day of interval.",
usage = "_FUNC_(date) - Returns the day of month of date/timestamp, or the day of interval.",
extended = """
Arguments:
expr - a date expression.
date - a date expression.

Examples:
> SELECT _FUNC_('2009-07-30');
Expand All @@ -353,10 +353,10 @@ case class DayOfMonth(child: Expression) extends UnaryExpression with ImplicitCa
}

@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the week of the year of the given date.",
usage = "_FUNC_(date) - Returns the week of the year of the given date.",
extended = """
Arguments:
expr - a date expression.
date - a date expression.

Examples:
> SELECT _FUNC_('2008-02-20');
Expand Down Expand Up @@ -400,10 +400,10 @@ case class WeekOfYear(child: Expression) extends UnaryExpression with ImplicitCa

// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr, fmt) - Converts expr to a value of string in the format specified by the date format fmt.",
usage = "_FUNC_(timestamp, fmt) - Converts expr to a value of string in the format specified by the date format fmt.",
extended = """
Arguments:
expr - a timestamp expression.
timestamp - a timestamp expression.
fmt - string type expression that represents date/timestamp format used in java.text.SimpleDateFormat.

Examples:
Expand Down Expand Up @@ -705,10 +705,10 @@ case class FromUnixTime(sec: Expression, format: Expression)
* Returns the last day of the month which the date belongs to.
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the last day of the month which the date belongs to.",
usage = "_FUNC_(date) - Returns the last day of the month which the date belongs to.",
extended = """
Arguments:
expr - a date expression.
date - a date expression.

Examples:
> SELECT _FUNC_('2009-01-12');
Expand Down Expand Up @@ -842,10 +842,10 @@ case class TimeAdd(start: Expression, interval: Expression)
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr, timezone) - Assumes given timestamp is UTC and converts to given timezone.",
usage = "_FUNC_(timestamp, timezone) - Assumes given timestamp is UTC and converts to given timezone.",
Copy link
Member

@srowen srowen Oct 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recently ran into this function in https://issues.apache.org/jira/browse/SPARK-17971?focusedCommentId=15583676&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15583676 and found its behavior pretty confusing. If you're willing, maybe a couple sentences here and in "to_unix_timestamp" could help clarify, because, timestamps actually have no timezone.

I think I'd describe this as:

Given a timestamp, which corresponds to a certain time of day in UTC, returns another timestamp that corresponds to the same time of day in the given timezone.

Copy link
Member Author

@HyukjinKwon HyukjinKwon Oct 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the details about this.

extended = """
Arguments:
expr - a timestamp expression.
timestamp - a timestamp expression.
timezone - a string expression that represents time zone (e.g., 'Asia/Seoul').

Examples:
Expand Down Expand Up @@ -969,11 +969,11 @@ case class AddMonths(startDate: Expression, numMonths: Expression)
* Returns number of months between dates date1 and date2.
*/
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Returns number of months between dates expr1 and expr2.",
usage = "_FUNC_(timestamp1, timestamp2) - Returns number of months between dates expr1 and expr2.",
extended = """
Arguments:
expr1 - a timestamp expression.
expr2 - a timestamp expression.
timestamp1 - a timestamp expression.
timestamp2 - a timestamp expression.

Examples:
> SELECT _FUNC_('1997-02-28 10:30:00', '1996-10-30');
Expand Down Expand Up @@ -1008,10 +1008,10 @@ case class MonthsBetween(date1: Expression, date2: Expression)
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr, timezone) - Assumes given timestamp is in given timezone and converts to UTC.",
usage = "_FUNC_(timestamp, timezone) - Assumes given timestamp is in given timezone and converts to UTC.",
Copy link
Member

@srowen srowen Oct 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise I think this is more accurately described as

Given a timestamp, which corresponds to a certain time of day in the given timezone, returns another timestamp that corresponds to the same time of day in UTC.

extended = """
Arguments:
expr - a timestamp expression.
timestamp - a timestamp expression.
timezone - a string expression that represents time zone (e.g., 'Asia/Seoul').

Examples:
Expand Down Expand Up @@ -1068,10 +1068,10 @@ case class ToUTCTimestamp(left: Expression, right: Expression)
* Returns the date part of a timestamp or string.
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Extracts the date part of the date or datetime expression expr.",
usage = "_FUNC_(date) - Extracts the date part of the date or datetime expression expr.",
extended = """
Arguments:
expr - a date expression.
date - a date expression.

Examples:
> SELECT _FUNC_('2009-07-30 04:17:52');
Expand Down Expand Up @@ -1099,10 +1099,10 @@ case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastIn
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr, fmt) - Returns returns date with the time portion of the day truncated to the unit specified by the format model fmt.",
usage = "_FUNC_(date, fmt) - Returns returns date with the time portion of the day truncated to the unit specified by the format model fmt.",
extended = """
Arguments:
expr - a date expression.
date - a date expression.
fmt - a string expression that represents the truncate level. Available values are "YEAR", "YYYY", "YY", "MON", "MONTH" and "MM".

Examples:
Expand Down Expand Up @@ -1182,11 +1182,11 @@ case class TruncDate(date: Expression, format: Expression)
* Returns the number of days from startDate to endDate.
*/
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Returns the number of days between expr1 and expr2.",
usage = "_FUNC_(date1, date2) - Returns the number of days between expr1 and expr2.",
extended = """
Arguments:
expr1 - a date expression.
expr2 - a date expression.
date1 - a date expression.
date2 - a date expression.

Examples:
> SELECT _FUNC_('2009-07-30', '2009-07-31');
Expand Down
36 changes: 18 additions & 18 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -872,11 +872,11 @@ case class Pow(left: Expression, right: Expression)
* @param right number of bits to left shift.
*/
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Bitwise left shift.",
usage = "_FUNC_(base, expr) - Bitwise left shift.",
extended = """
Arguments:
expr1 - a numeric expression.
expr2 - a numeric expression.
base - a numeric expression that defines the base number to shift.
expr - a numeric expression.

Examples:
> SELECT _FUNC_(2, 1);
Expand Down Expand Up @@ -909,11 +909,11 @@ case class ShiftLeft(left: Expression, right: Expression)
* @param right number of bits to right shift.
*/
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Bitwise right shift.",
usage = "_FUNC_(base, expr) - Bitwise right shift.",
Copy link
Member

@srowen srowen Oct 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might specify it's the "Bitwise (signed) right shift"

extended = """
Arguments:
expr1 - a numeric expression.
expr2 - a numeric expression.
base - a numeric expression that defines the base number to shift.
expr - a numeric expression.

Examples:
> SELECT _FUNC_(4, 1);
Expand Down Expand Up @@ -946,11 +946,11 @@ case class ShiftRight(left: Expression, right: Expression)
* @param right the number of bits to right shift.
*/
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Bitwise unsigned right shift.",
usage = "_FUNC_(base, expr) - Bitwise unsigned right shift.",
extended = """
Arguments:
expr1 - a numeric expression.
expr2 - a numeric expression.
base - a numeric expression that defines the base number to shift.
expr - a numeric expression.

Examples:
> SELECT _FUNC_(4, 1);
Expand Down Expand Up @@ -998,11 +998,11 @@ case class Hypot(left: Expression, right: Expression)
* @param right the number to compute the logarithm of.
*/
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Returns the logarithm of expr1 with base expr2.",
usage = "_FUNC_(base, expr) - Returns the logarithm of expr with base.",
extended = """
Arguments:
expr1 - a numeric expression.
expr2 - a numeric expression.
base - a numeric expression that represents the logarithm base.
expr - a numeric expression.

Examples:
> SELECT _FUNC_(10, 100);
Expand Down Expand Up @@ -1232,11 +1232,11 @@ abstract class RoundBase(child: Expression, scale: Expression,
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Round expr1 to expr2 decimal places using HALF_UP rounding mode.",
usage = "_FUNC_(expr, d) - Round expr to d decimal places using HALF_UP rounding mode.",
extended = """
Arguments:
expr1 - a numeric expression.
expr2 - a numeric expression.
expr - a numeric expression.
d - a numeric expression that defines tge decimal places using HALF_UP rounding mode.

Examples:
> SELECT _FUNC_(2.5, 0);
Expand All @@ -1256,11 +1256,11 @@ case class Round(child: Expression, scale: Expression)
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Round expr1 to expr2 decimal places using HALF_EVEN rounding mode.",
usage = "_FUNC_(expr, d) - Round expr to d decimal places using HALF_EVEN rounding mode.",
Copy link
Member

@gatorsmile gatorsmile Oct 23, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add Returns at the beginning.

extended = """
Arguments:
expr1 - a numeric expression.
expr2 - a numeric expression.
expr - a numeric expression.
d - a numeric expression that defines decimal places using HALF_EVEN rounding mode.

Examples:
> SELECT _FUNC_(2.5, 0);
Expand Down
4 changes: 1 addition & 3 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,8 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
"Class: org.apache.spark.sql.catalyst.expressions.Upper",
"Usage: upper(str) - Returns str with all characters changed to uppercase",
"Extended Usage:",
"upper(str)",
"Arguments:",
"str - string type or any type expression that can be implicitly converted",
"to string type.",
"str - a string expression.",
"Examples:",
"> SELECT upper('SparkSql');",
"SPARKSQL")
Expand Down
16 changes: 7 additions & 9 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1455,7 +1455,7 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
sql("DESCRIBE FUNCTION log"),
Row("Class: org.apache.spark.sql.catalyst.expressions.Logarithm") ::
Row("Function: log") ::
Row("Usage: log(expr1, expr2) - Returns the logarithm of expr1 with base expr2.") :: Nil
Row("Usage: log(base, expr) - Returns the logarithm of expr with base.") :: Nil
)
// predicate operator
checkAnswer(
Expand Down Expand Up @@ -1498,15 +1498,13 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
Row("Class: org.apache.spark.sql.catalyst.expressions.BitwiseXor") ::
Row(
"""Extended Usage:
| expr1 ^ expr2
| Arguments:
| expr1 - a numeric expression.
| expr2 - a numeric expression.
|
| Arguments:
| expr1 - any intergal numeric type expression.
| expr2 - any intergal numeric type expression.
|
| Examples:
| > SELECT 3 ^ 5;
| 2
| Examples:
| > SELECT 3 ^ 5;
| 2
| """.stripMargin) ::
Row("Function: ^") ::
Row("Usage: expr1 ^ expr2 - Bitwise exclusive OR.") :: Nil
Expand Down
4 changes: 1 addition & 3 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,10 +254,8 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
"Class: org.apache.spark.sql.catalyst.expressions.Upper",
"Usage: upper(str) - Returns str with all characters changed to uppercase",
"Extended Usage:",
"upper(str)",
"Arguments:",
"str - string type or any type expression that can be implicitly converted",
"to string type.",
"str - a string expression.",
"Examples:",
"> SELECT upper('SparkSql');",
"SPARKSQL")
Expand Down