[SPARK-31756][WEBUI] Add real headless browser support for UI test #28627

sarutak · 2020-05-24T04:46:02Z

What changes were proposed in this pull request?

This PR mainly adds two things.

Real headless browser support for UI test
A test suite using headless Chrome as one instance of those browsers.

Also, for environment where Chrome and Chrome driver is not installed, ChromeUITest tag is added to filter out the test suite.
By default, test suites with ChromeUITest is disabled.

Why are the changes needed?

In the current master, there are two problems for UI test.

Lots of tests especially JavaScript related ones are done manually.
Appearance is better to be confirmed by our eyes but logic should be tested by test cases ideally.
Compared to the real web browsers, HtmlUnit doesn't seem to support JavaScript enough.
I added a JavaScript related test before for SPARK-31534 using HtmlUnit which is simple library based headless browser for test.
The test I added works somehow but some JavaScript related error is shown in unit-tests.log.

======= EXCEPTION START ========
Exception class=[net.sourceforge.htmlunit.corejs.javascript.JavaScriptException]
com.gargoylesoftware.htmlunit.ScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:904)
        at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
        at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:835)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:807)
        at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
        at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptFunctionJob.runJavaScript(JavaScriptFunctionJob.java:52)
        at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptExecutionJob.run(JavaScriptExecutionJob.java:102)
        at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptJobManagerImpl.runSingleJob(JavaScriptJobManagerImpl.java:426)
        at com.gargoylesoftware.htmlunit.javascript.background.DefaultJavaScriptExecutor.run(DefaultJavaScriptExecutor.java:157)
        at java.lang.Thread.run(Thread.java:748)
Caused by: net.sourceforge.htmlunit.corejs.javascript.JavaScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2)
        at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1009)
        at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
        at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
        at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413)
        at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:252)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$4.doRun(JavaScriptEngine.java:828)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:889)
        ... 10 more
JavaScriptException value = Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)".
== CALLING JAVASCRIPT ==
  function () {
      throw e;
  }
======= EXCEPTION END ========

I tried to upgrade HtmlUnit to 2.40.0 but what is worse, the test become not working even though it works on real browsers like Chrome, Safari and Firefox without error.

[info] UISeleniumSuite:
[info] - SPARK-31534: text for tooltip should be escaped *** FAILED *** (17 seconds, 745 milliseconds)
[info]   The code passed to eventually never returned normally. Attempted 2 times over 12.910785232 seconds. Last failure message: com.gargoylesoftware.htmlunit.ScriptException: ReferenceError: Assignment to undefined "regeneratorRuntime" in strict mode (http://192.168.1.209:62132/static/vis-timeline-graph2d.min.js#52(Function)#1)

To resolve those problems, it's better to support headless browser for UI test.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I tested with following patterns. Both Chrome and Chrome driver should be installed to test.

sbt / with default excluded tags (ChromeUISeleniumSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed)
`build/sbt -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite org.apache.spark.sql.SQLQueryTestSuite"
sbt / overwrite default excluded tags as empty string (Both suites are expected to succeed)
`build/sbt -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite org.apache.spark.sql.SQLQueryTestSuite"
sbt / set test.exclude.tags to org.apache.spark.tags.ExtendedSQLTest (Both suites are expected to be skipped)
`build/sbt -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite org.apache.spark.sql.SQLQueryTestSuite"
Maven / with default excluded tags (ChromeUISeleniumSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed)
build/mvn -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite,org.apache.spark.sql.SQLQueryTestSuite test
Maven / overwrite default excluded tags as empty string (Both suites are expected to succeed)
build/mvn -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite,org.apache.spark.sql.SQLQueryTestSuite test
Maven / set test.exclude.tags to org.apache.spark.tags.ExtendedSQLTest (Both suites are expected to be skipped)
build/mvn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite,org.apache.spark.sql.SQLQueryTestSuite test

…tests.py

…er-support-take2

SparkQA · 2020-05-24T07:05:01Z

Test build #123042 has finished for PR 28627 at commit 6b47fe8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-24T07:11:02Z

retest this please.

SparkQA · 2020-05-24T13:54:53Z

Test build #123052 has finished for PR 28627 at commit 6b47fe8.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-05-24T14:08:55Z

Jenkins retest this please

dongjoon-hyun · 2020-05-24T20:12:48Z

dev/run-tests.py

 from sparktestsupport.toposort import toposort_flatten
 import sparktestsupport.modules as modules

+always_excluded_tags = [


always_excluded_tags might be misleading because this should be tested in the proper environments.

dongjoon-hyun · 2020-05-24T20:15:20Z

pom.xml


    <test.java.home>${java.home}</test.java.home>
-    <test.exclude.tags></test.exclude.tags>
+    <test.exclude.tags>org.apache.spark.tags.ChromeUITest</test.exclude.tags>


Could you add some comment before this line?

dongjoon-hyun · 2020-05-24T20:18:42Z

project/SparkBuild.scala

-        tags.split(",").flatMap { tag => Seq("-l", tag) }.toSeq
-      }.getOrElse(Nil): _*),
+      sys.props.get("test.exclude.tags").map(tag => tag.split(",").toSeq)
+        .getOrElse(defaultExcludedTagsForScalaTest).filter(!_.trim.isEmpty)


This looks not robust enough.
If a developer set an irrelevant tag (ExtendedSQLTest or ExtendedHiveTest), ChromeUITest suddenly will cause a failure in the irrelevant Core module. Did I understand correctly?

It's what I intended that if developers set test.excluded.tags explicitly, all the tags which are intended to be excluded should be added explicitly.
But as you concerned, this change seems to break the build compatibility.
I'll submit another idea.

dongjoon-hyun

Hi, @sarutak .
I left a few comments. Especially, could you take a look at #28627 (comment) ?

As a downstream user, I'm using the other tags ExtendedHiveTest or ExtendedSQLTest a lot in my test environment. This had better be orthogonal to the other tags.

SparkQA · 2020-05-24T20:53:37Z

Test build #123060 has finished for PR 28627 at commit 6b47fe8.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-25T03:25:34Z

Test patterns are updated in the description.

SparkQA · 2020-05-25T07:05:01Z

Test build #123071 has finished for PR 28627 at commit 70d07a3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-25T07:12:46Z

retest this please.

SparkQA · 2020-05-25T13:58:21Z

Test build #123076 has finished for PR 28627 at commit 70d07a3.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-25T13:59:50Z

retest this please.

SparkQA · 2020-05-25T20:43:11Z

Test build #123087 has finished for PR 28627 at commit 70d07a3.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-25T20:56:05Z

retest this please.

SparkQA · 2020-05-26T03:43:39Z

Test build #123099 has finished for PR 28627 at commit 70d07a3.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-26T23:32:26Z

retest this please.

sarutak · 2020-05-26T23:47:20Z

@dongjoon-hyun Is it possible to finish tests within 400m with Maven?
I see #28640 also fails tests with Maven because of time out.
Can we extend the time for tests with Maven somehow?

SparkQA · 2020-05-27T06:17:05Z

Test build #123151 has finished for PR 28627 at commit 70d07a3.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-27T22:32:37Z

retest this please.

SparkQA · 2020-05-28T05:18:46Z

Test build #123202 has finished for PR 28627 at commit 70d07a3.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-28T05:55:19Z

@HyukjinKwon @shaneknapp
The time limit for CI seems to be too short for Maven.
Tests for this PR has failed 9 times due to the limit.
I see another PR fails with the same reason.
I understand we should avoid expanding the limit as much as possible but can we expand the limit for Maven or do you have better idea?

HyukjinKwon · 2020-05-28T08:07:34Z

retest this please

HyukjinKwon · 2020-05-28T08:09:58Z

The problem is that it's getting too long to finish each PR build .. so last time it was pretty arguable when it increased the build time IIRC. Maybe it's time to profile and reduce the testing time again .. adding @cloud-fan too FYI. Shall we increase a bit more for now since it blocks Maven testing as can be seen above?

SparkQA · 2020-05-28T11:30:43Z

Test build #123224 has finished for PR 28627 at commit 70d07a3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…adless-browser-support-take2

SparkQA · 2020-05-29T05:57:01Z

Test build #123247 has finished for PR 28627 at commit 2e805ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-29T06:00:02Z

Couldn't finish within 500m...

sarutak · 2020-05-29T06:00:13Z

retest this please.

SparkQA · 2020-05-29T07:05:02Z

Test build #123265 has finished for PR 28627 at commit 2e805ec.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-29T07:14:07Z

retest this please.

SparkQA · 2020-05-29T08:41:06Z

Test build #123268 has finished for PR 28627 at commit 2e805ec.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-05-29T09:16:30Z

retest this please

dongjoon-hyun · 2020-05-29T10:20:07Z

It seems that we are hitting time-out still in some cases.

BTW, @sarutak . Is the following a successful run?

[SPARK-31756][WEBUI] Add real headless browser support for UI test #28627 (comment)

sarutak · 2020-05-29T11:38:44Z

BTW, @sarutak . Is the following a successful run?

Actually, it's not.
According to the console log, it's almost finished but aborted...
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123247/console

dongjoon-hyun · 2020-05-29T12:26:07Z

Oh. Got it, @sarutak .

SparkQA · 2020-05-29T17:03:50Z

Test build #123282 has finished for PR 28627 at commit 2e805ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. Thank you for updating the PR by introducing test.default.exclude.tags.
Merged to master.

sarutak · 2020-05-30T02:46:27Z

Thanks all for the review and expanding the timeout!

### What changes were proposed in this pull request? This PR mainly adds two things. 1. Real headless browser support for UI test 2. A test suite using headless Chrome as one instance of those browsers. Also, for environment where Chrome and Chrome driver is not installed, `ChromeUITest` tag is added to filter out the test suite. By default, test suites with `ChromeUITest` is disabled. ### Why are the changes needed? In the current master, there are two problems for UI test. 1. Lots of tests especially JavaScript related ones are done manually. Appearance is better to be confirmed by our eyes but logic should be tested by test cases ideally. 2. Compared to the real web browsers, HtmlUnit doesn't seem to support JavaScript enough. I added a JavaScript related test before for SPARK-31534 using HtmlUnit which is simple library based headless browser for test. The test I added works somehow but some JavaScript related error is shown in unit-tests.log. ``` ======= EXCEPTION START ======== Exception class=[net.sourceforge.htmlunit.corejs.javascript.JavaScriptException] com.gargoylesoftware.htmlunit.ScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:904) at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628) at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:835) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:807) at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216) at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptFunctionJob.runJavaScript(JavaScriptFunctionJob.java:52) at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptExecutionJob.run(JavaScriptExecutionJob.java:102) at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptJobManagerImpl.runSingleJob(JavaScriptJobManagerImpl.java:426) at com.gargoylesoftware.htmlunit.javascript.background.DefaultJavaScriptExecutor.run(DefaultJavaScriptExecutor.java:157) at java.lang.Thread.run(Thread.java:748) Caused by: net.sourceforge.htmlunit.corejs.javascript.JavaScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1009) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800) at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105) at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413) at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:252) at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$4.doRun(JavaScriptEngine.java:828) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:889) ... 10 more JavaScriptException value = Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". == CALLING JAVASCRIPT == function () { throw e; } ======= EXCEPTION END ======== ``` I tried to upgrade HtmlUnit to 2.40.0 but what is worse, the test become not working even though it works on real browsers like Chrome, Safari and Firefox without error. ``` [info] UISeleniumSuite: [info] - SPARK-31534: text for tooltip should be escaped *** FAILED *** (17 seconds, 745 milliseconds) [info] The code passed to eventually never returned normally. Attempted 2 times over 12.910785232 seconds. Last failure message: com.gargoylesoftware.htmlunit.ScriptException: ReferenceError: Assignment to undefined "regeneratorRuntime" in strict mode (http://192.168.1.209:62132/static/vis-timeline-graph2d.min.js#52(Function)#1) ``` To resolve those problems, it's better to support headless browser for UI test. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I tested with following patterns. Both Chrome and Chrome driver should be installed to test. 1. sbt / with default excluded tags (ChromeUISeleniumSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed) `build/sbt -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite org.apache.spark.sql.SQLQueryTestSuite" 2. sbt / overwrite default excluded tags as empty string (Both suites are expected to succeed) `build/sbt -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite org.apache.spark.sql.SQLQueryTestSuite" 3. sbt / set `test.exclude.tags` to `org.apache.spark.tags.ExtendedSQLTest` (Both suites are expected to be skipped) `build/sbt -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite org.apache.spark.sql.SQLQueryTestSuite" 4. Maven / with default excluded tags (ChromeUISeleniumSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed) `build/mvn -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite,org.apache.spark.sql.SQLQueryTestSuite test` 5. Maven / overwrite default excluded tags as empty string (Both suites are expected to succeed) `build/mvn -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite,org.apache.spark.sql.SQLQueryTestSuite test` 6. Maven / set `test.exclude.tags` to `org.apache.spark.tags.ExtendedSQLTest` (Both suites are expected to be skipped) `build/mvn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite,org.apache.spark.sql.SQLQueryTestSuite test` Closes apache#28627 from sarutak/real-headless-browser-support-take2. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This PR backports the change of #28627 . Some fixes for UI like #28690 should be merged to `branch-3.0` including testcases so #28627 is need to be backported. This PR mainly adds two things. 1. Real headless browser support for UI test 2. A test suite using headless Chrome as one instance of those browsers. Also, for environment where Chrome and Chrome driver is not installed, `ChromeUITest` tag is added to filter out the test suite. By default, test suites with `ChromeUITest` is disabled. ### Why are the changes needed? In `branch-3.0`, there are two problems for UI test. 1. Lots of tests especially JavaScript related ones are done manually. Appearance is better to be confirmed by our eyes but logic should be tested by test cases ideally. 2. Compared to the real web browsers, HtmlUnit doesn't seem to support JavaScript enough. I added a JavaScript related test before for SPARK-31534 using HtmlUnit which is simple library based headless browser for test. The test I added works somehow but some JavaScript related error is shown in unit-tests.log. ``` ======= EXCEPTION START ======== Exception class=[net.sourceforge.htmlunit.corejs.javascript.JavaScriptException] com.gargoylesoftware.htmlunit.ScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:904) at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628) at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:835) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:807) at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216) at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptFunctionJob.runJavaScript(JavaScriptFunctionJob.java:52) at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptExecutionJob.run(JavaScriptExecutionJob.java:102) at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptJobManagerImpl.runSingleJob(JavaScriptJobManagerImpl.java:426) at com.gargoylesoftware.htmlunit.javascript.background.DefaultJavaScriptExecutor.run(DefaultJavaScriptExecutor.java:157) at java.lang.Thread.run(Thread.java:748) Caused by: net.sourceforge.htmlunit.corejs.javascript.JavaScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1009) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800) at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105) at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413) at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:252) at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$4.doRun(JavaScriptEngine.java:828) at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:889) ... 10 more JavaScriptException value = Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". == CALLING JAVASCRIPT == function () { throw e; } ======= EXCEPTION END ======== ``` I tried to upgrade HtmlUnit to 2.40.0 but what is worse, the test become not working even though it works on real browsers like Chrome, Safari and Firefox without error. ``` [info] UISeleniumSuite: [info] - SPARK-31534: text for tooltip should be escaped *** FAILED *** (17 seconds, 745 milliseconds) [info] The code passed to eventually never returned normally. Attempted 2 times over 12.910785232 seconds. Last failure message: com.gargoylesoftware.htmlunit.ScriptException: ReferenceError: Assignment to undefined "regeneratorRuntime" in strict mode (http://192.168.1.209:62132/static/vis-timeline-graph2d.min.js#52(Function)#1) ``` To resolve those problems, it's better to support headless browser for UI test. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I tested with following patterns. Both Chrome and Chrome driver should be installed to test. 1. sbt / with default excluded tags (ChromeUISeleniumSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed) `build/sbt -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite org.apache.spark.sql.SQLQueryTestSuite"` 2. sbt / overwrite default excluded tags as empty string (Both suites are expected to succeed) `build/sbt -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite org.apache.spark.sql.SQLQueryTestSuite"` 3. sbt / set `test.exclude.tags` to `org.apache.spark.tags.ExtendedSQLTest` (Both suites are expected to be skipped) `build/sbt -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite org.apache.spark.sql.SQLQueryTestSuite"` 4. Maven / with default excluded tags (ChromeUISeleniumSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed) `build/mvn -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite,org.apache.spark.sql.SQLQueryTestSuite test` 5. Maven / overwrite default excluded tags as empty string (Both suites are expected to succeed) `build/mvn -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite,org.apache.spark.sql.SQLQueryTestSuite test` 6. Maven / set `test.exclude.tags` to `org.apache.spark.tags.ExtendedSQLTest` (Both suites are expected to be skipped) `build/mvn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite,org.apache.spark.sql.SQLQueryTestSuite test` Closes #28702 from sarutak/real-headless-browser-support-3.0. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Gengliang Wang <[email protected]>

shaneknapp · 2020-06-10T00:40:12Z

sorry, i don't watch github for @mentions... if this is still a problem, file a build/infra jira and assign it to me. :)

…

On Thu, May 28, 2020 at 11:00 PM Kousuke Saruta ***@***.***> wrote: Couldn't finish within 500m... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28627 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMIHLDX32JFW75S7HT5DQ3RT5FPBANCNFSM4NIXL3ZA> .

sarutak added 8 commits May 19, 2020 14:11

Add RealBrowserUISeleniumSuite.scala

424092a

Add ChromeUITest.java

4fe3e80

Add ChromeUISeeniumSuite.scala

760eb0a

Remove JavaScript related tests from UISeleniumSuite.scala

c2f6a6d

Disable tests taged as 'ChromeUITest' when tests are run through run-…

e701dc9

…tests.py

Fixed RealBrowserUISeleniumSuite.

6ccbc39

Merge branch 'real-headless-browser-support' into real-headless-brows…

7e49348

…er-support-take2

Disabled newly added suite by default.

6b47fe8

probot-autolabeler bot added BUILD CORE WEB UI labels May 24, 2020

sarutak mentioned this pull request May 24, 2020

[SPARK-31804][WEBUI] Add real headless browser support for HistoryServer tests #28622

Closed

dongjoon-hyun reviewed May 24, 2020

View reviewed changes

dongjoon-hyun requested changes May 24, 2020

View reviewed changes

Introduced test.default.exclude.tags to set excluded tests by default.

70d07a3

Merge branch 'master' of https://github.com/apache/spark into real-he…

2e805ec

…adless-browser-support-take2

dongjoon-hyun approved these changes May 29, 2020

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-31756][WEBUI][test-maven] Add real headless browser support for UI test~~ [SPARK-31756][WEBUI] Add real headless browser support for UI test May 29, 2020

dongjoon-hyun closed this in d3eba5b May 29, 2020

sarutak mentioned this pull request Jun 2, 2020

[SPARK-31882][WEBUI] DAG-viz is not rendered correctly with pagination #28690

Closed

sarutak mentioned this pull request Jun 2, 2020

[SPARK-31756][WEBUI][3.0] Add real headless browser support for UI test #28702

Closed

[SPARK-31756][WEBUI] Add real headless browser support for UI test #28627

[SPARK-31756][WEBUI] Add real headless browser support for UI test #28627

Uh oh!

Conversation

sarutak commented May 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented May 24, 2020

Uh oh!

sarutak commented May 24, 2020

Uh oh!

SparkQA commented May 24, 2020

Uh oh!

srowen commented May 24, 2020

Uh oh!

dongjoon-hyun May 24, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun May 24, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun May 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarutak May 25, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 24, 2020

Uh oh!

sarutak commented May 25, 2020

Uh oh!

SparkQA commented May 25, 2020

Uh oh!

sarutak commented May 25, 2020

Uh oh!

SparkQA commented May 25, 2020

Uh oh!

sarutak commented May 25, 2020

Uh oh!

SparkQA commented May 25, 2020

Uh oh!

sarutak commented May 25, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

sarutak commented May 26, 2020

Uh oh!

sarutak commented May 26, 2020

Uh oh!

SparkQA commented May 27, 2020

Uh oh!

sarutak commented May 27, 2020

Uh oh!

SparkQA commented May 28, 2020

Uh oh!

sarutak commented May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented May 28, 2020

Uh oh!

HyukjinKwon commented May 28, 2020

Uh oh!

SparkQA commented May 28, 2020

Uh oh!

SparkQA commented May 29, 2020

Uh oh!

sarutak commented May 29, 2020

Uh oh!

sarutak commented May 29, 2020

Uh oh!

SparkQA commented May 29, 2020

Uh oh!

sarutak commented May 24, 2020 •

edited

Loading

dongjoon-hyun May 24, 2020 •

edited

Loading

sarutak commented May 28, 2020 •

edited

Loading

sarutak commented May 29, 2020 •

edited

Loading