[SPARK-28793][DOC][SQL] Document CREATE FUNCTION in SQL Reference #25894

dilipbiswal · 2019-09-22T22:05:36Z

What changes were proposed in this pull request?

Document CREATE FUNCTION statement in SQL Reference Guide.

Why are the changes needed?

Currently Spark lacks documentation on the supported SQL constructs causing
confusion among users who sometimes have to look at the code to understand the
usage. This is aimed at addressing this issue.

Does this PR introduce any user-facing change?

Yes.

Before:
There was no documentation for this.

After.

How was this patch tested?

Tested using jykyll build --serve

SparkQA · 2019-09-22T22:18:38Z

Test build #111168 has finished for PR 25894 at commit cddbebb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
-- public class SimpleUdf extends UDF
-- public class SimpleUdfR extends UDF

gatorsmile · 2019-09-22T22:48:00Z

docs/sql-ref-syntax-ddl-create-function.md

+  <dt><code><em>TEMPORARY</em></code></dt>
+  <dd>
+    Indicates the scope of function being created. When TEMPORARY is specified, the
+    created function is valid in the current session. No persistent entry is made 


valid -> valid and visible

@gatorsmile thanks.. fixed.

gatorsmile · 2019-09-22T22:50:40Z

docs/sql-ref-syntax-ddl-create-function.md

+
+### Examples
+{% highlight sql %}
+-- 1. Create a simple UDF `SimpleUdf` that adds the supplied integet value by 10.


integet -> integral

@gatorsmile fixed.

gatorsmile · 2019-09-22T22:53:06Z

docs/sql-ref-syntax-ddl-create-function.md

+  <dt><code><em>class_name</em></code></dt>
+  <dd>
+    Specifies the name of the class that provides the implementation for function to be created.
+    The implementing class should extend from one of the base classes in `Hive` as follows:


UserDefinedAggregateFunction. We also support this.

Also we can extend org.apache.spark.sql.api.java.UDF0,... UDF22. see https://github.com/apache/spark/blob/master/sql/core/src/test/java/test/org/apache/spark/sql/JavaStringLength.java

@gatorsmile I have created a place holder link for custom scalar functions. There is already a place holder for aggregate functions and a lot of content is already in place.

gatorsmile · 2019-09-22T23:00:06Z

docs/sql-ref-syntax-ddl-create-function.md

+in Spark. Temporary functions are scoped at a session level where as permanent
+functions are created in the persistent catalog and are made available to
+all sessions. The resources specified in the `USING` clause are made available
+to all executors when they are executed for the first time.


We also need to explain we can create/register temporary SQL functions via Python/Scala/Java APIs.

@gatorsmile I have created a place holder link for custom scalar functions. There is already a place holder for aggregate functions and a lot of content is already in place.

SparkQA · 2019-09-23T05:27:57Z

Test build #111187 has finished for PR 25894 at commit 8c734c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-08T01:04:00Z

Test build #111860 has finished for PR 25894 at commit 3d0389f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-10-14T05:39:28Z

docs/sql-ref-syntax-ddl-create-function.md

+### Related statements
+- [SHOW FUNCTIONS](sql-ref-syntax-aux-show-functions.html)
+- [DESCRIBE FUNCTION](sql-ref-syntax-aux-describe-function.html)
+- [DESCRIBE FUNCTION](sql-ref-syntax-aux-describe-function.html)


@gatorsmile oops.. thanks for catching it. Will fix.

gatorsmile · 2019-10-14T05:42:11Z

docs/sql-getting-started.md

 </div>

+## Scalar Functions
+(to be filled soon)


Create a JIRA?

@gatorsmile created here

gatorsmile

LGTM except two comments.

SparkQA · 2019-10-14T06:11:54Z

Test build #112005 has finished for PR 25894 at commit 0cb396b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2019-10-17T16:26:59Z

cc @gatorsmile

srowen · 2019-10-21T17:09:57Z

docs/sql-ref-syntax-ddl-create-function.md

+  <dt><code><em>IF NOT EXISTS</em></code></dt>
+  <dd>
+    If specified, creates the function only when it does not exist. The creation
+    of function succeeds (no error is thrown), if the specified function already


Nit: remove comma

srowen · 2019-10-21T17:10:09Z

docs/sql-ref-syntax-ddl-create-function.md

+  </dd>
+  <dt><code><em>TEMPORARY</em></code></dt>
+  <dd>
+    Indicates the scope of function being created. When TEMPORARY is specified, the


Nit: code-format TEMPORARY?

srowen · 2019-10-21T17:10:15Z

docs/sql-ref-syntax-ddl-create-function.md

+<dl>
+  <dt><code><em>OR REPLACE</em></code></dt>
+  <dd>
+    If specified, the resources for function are reloaded. This is mainly useful


for the function

srowen · 2019-10-21T17:10:38Z

docs/sql-ref-syntax-ddl-create-function.md

+  <dt><code><em>class_name</em></code></dt>
+  <dd>
+    Specifies the name of the class that provides the implementation for function to be created.
+    The implementing class should extend from one of the base classes as follows:


remove 'from'

srowen · 2019-10-21T17:11:07Z

docs/sql-ref-syntax-ddl-create-function.md

+
+### Examples
+{% highlight sql %}
+-- 1. Create a simple UDF `SimpleUdf` that adds the supplied integral value by 10.


"increments" rather than "adds"?

srowen · 2019-10-21T17:11:18Z

docs/sql-ref-syntax-ddl-create-function.md

+--    import org.apache.hadoop.hive.ql.exec.UDF;
+--    public class SimpleUdf extends UDF {
+--      public int evaluate(int value) {
+--      return value + 10;


Nit: indent this more

srowen · 2019-10-21T17:11:32Z

docs/sql-ref-syntax-ddl-create-function.md

+--      return value + 10;
+--      }
+--    }
+-- 2. Compile and place it in a jar file called `SimpleUdf.jar` in /tmp.


Nit: jar -> JAR

srowen · 2019-10-21T17:11:45Z

docs/sql-ref-syntax-ddl-create-function.md

+  |   simple_temp_udf|
+  +------------------+
+
+-- 1. Mofify `SimpleUdf`'s implementation to add supplied integral value by 20.


Mofify -> Modify

srowen · 2019-10-21T17:11:52Z

docs/sql-ref-syntax-ddl-create-function.md

+
+--    public class SimpleUdfR extends UDF {
+--      public int evaluate(int value) {
+--      return value + 20;


Indent here

SparkQA · 2019-10-21T20:15:30Z

Test build #112408 has finished for PR 25894 at commit 0a964c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-10-22T13:56:52Z

Merged to master

dilipbiswal · 2019-10-23T04:54:31Z

Thanks a lot @srowen @gatorsmile

[SPARK-28793][DOC] Document CREATE FUNCTION in SQL Reference

cddbebb

gatorsmile reviewed Sep 22, 2019

View reviewed changes

Code review

8c734c9

dongjoon-hyun added DOCUMENTATION SQL labels Sep 23, 2019

Code review

3d0389f

gatorsmile reviewed Oct 14, 2019

View reviewed changes

gatorsmile approved these changes Oct 14, 2019

View reviewed changes

Code review

0cb396b

srowen requested changes Oct 21, 2019

View reviewed changes

Code review

0a964c5

srowen approved these changes Oct 21, 2019

View reviewed changes

srowen closed this in c1c6485 Oct 22, 2019

[SPARK-28793][DOC][SQL] Document CREATE FUNCTION in SQL Reference #25894

[SPARK-28793][DOC][SQL] Document CREATE FUNCTION in SQL Reference #25894

Uh oh!

Conversation

dilipbiswal commented Sep 22, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Sep 22, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 23, 2019

Uh oh!

SparkQA commented Oct 8, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 14, 2019

Uh oh!

dilipbiswal commented Oct 17, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 21, 2019

Uh oh!

srowen commented Oct 22, 2019

Uh oh!

dilipbiswal commented Oct 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!