[SPARK-32088][PYTHON][FOLLOWUP] Replace collect() by show() in the example for timestamp_seconds

[MaxGekk]

What changes were proposed in this pull request?

Modify the example for timestamp_seconds and replace collect() by show().

Why are the changes needed?

The SQL config spark.sql.session.timeZone doesn't influence on the collect in the example. The code below demonstrates that:

$ export TZ="UTC"

>>> from pyspark.sql.functions import timestamp_seconds
>>> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
>>> time_df = spark.createDataFrame([(1230219000,)], ['unix_time'])
>>> time_df.select(timestamp_seconds(time_df.unix_time).alias('ts')).collect()
[Row(ts=datetime.datetime(2008, 12, 25, 15, 30))]

The expected time is 07:30 but we get 15:30.

Does this PR introduce any user-facing change?

No

How was this patch tested?

By running the modified example via:

$ ./python/run-tests --modules=pyspark-sql

[jiangxb1987]

LGTM pending jenkins test

[SparkQA]

Test build #124688 has finished for PR 28959 at commit aa6a0e1.

• This patch fails to generate documentation.
• This patch merges cleanly.
• This patch adds no public classes.

[MaxGekk]

It seems the build failure is not related to the changes:

[error] /home/jenkins/workspace/SparkPullRequestBuilder@6/core/src/main/scala/org/apache/spark/SSLOptions.scala:71: type Server is not a member of object org.eclipse.jetty.util.ssl.SslContextFactory
[error]       val sslContextFactory = new SslContextFactory.Server()
[error]                                                     ^
[info] No documentation generated with unsuccessful compiler run
[error] one error found

[dongjoon-hyun]

Hi, @MaxGekk . Are you sure that is due to #28863 ? Is it build error or test error? I need your confirmation.

[dongjoon-hyun]

Hi, @MaxGekk and @jiangxb1987 . Could you give me more hints?

[dongjoon-hyun]

Until now, I'm observing the following. I'll monitor more, but it would be great if you can give me more evidence so that I can help you, @MaxGekk and @jiangxb1987 .

1. GitHub Action is healthy all profile combination.
2. Jenkins job is running on the latest master branch.

• https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/907/

[HyukjinKwon]

retest this please

[HyukjinKwon]

Sure, let's block before the test results.

[SparkQA]

Test build #124718 has finished for PR 28959 at commit aa6a0e1.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[cloud-fan]

do we set the timezone in pyspark doctest, like the scala side test?

[cloud-fan]

I'm wondering if this is the common approach we do for all timestamp related pyspark doc tests.

[HyukjinKwon]

Yeah, we can do that too. Let's set it when we face another similar issue. It's not difficult to do it.

[SparkQA]

Test build #124729 has finished for PR 28959 at commit aa6a0e1.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #124735 has finished for PR 28959 at commit aa6a0e1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master.

[HyukjinKwon]

FYI,

[error] /home/jenkins/workspace/SparkPullRequestBuilder@6/core/src/main/scala/org/apache/spark/SSLOptions.scala:71: type Server is not a member of object org.eclipse.jetty.util.ssl.SslContextFactory
[error]       val sslContextFactory = new SslContextFactory.Server()
[error]                                                     ^
[info] No documentation generated with unsuccessful compiler run
[error] one error found

This is from unidoc when you generate the documentation. It sorts of compiles Scala files back to Java files, and extract the documentations from there. It's kind of buggy time to time.