[SPARK-8675] Executors created by LocalBackend won't get the same classpath as other executor backends

[coderplay]

AFAIK, some spark application always use LocalBackend to do some local initiatives, spark sql is an example. Starting a LocalPoint won't add user classpath into executor.

  override def start() {
    localEndpoint = SparkEnv.get.rpcEnv.setupEndpoint(
      "LocalBackendEndpoint", new LocalEndpoint(SparkEnv.get.rpcEnv, scheduler, this, totalCores))
  }

Thus will cause local executor fail with these scenarios, loading hadoop built-in native libraries, loading other user defined native libraries, loading user jars, reading s3 config from a site.xml file, etc

[JoshRosen]

Jenkins, this is ok to test.

[SparkQA]

Test build #36024 has finished for PR 7091 at commit 45bf62c.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Sha1(child: Expression) extends UnaryExpression with ExpectsInputTypes

[andrewor14]

Looks fine. @coderplay can you fix the scalastyle?

[coderplay]

@andrewor14 Sure, will do. Any existing formatter for eclipse or idea?

[andrewor14]

There are some for intellij but Spark has different style requirements than typical Scala applications. Here you can just click into the last test failure and look at the bottom of the test output and fix those manually.

[SparkQA]

Test build #36963 has finished for PR 7091 at commit d215b7f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Looks like it's failing tests legitimately. You might need to add the file: scheme to the URL, e.g.
....map { p => new URL("file:/" + p.stripPrefix("/")) }

[coderplay]

it's a solution, but what if user specify a hdfs://xx/yy.jar ?

[coderplay]

From https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala#L202 , seems spark doesn't support non-local classpath

[andrewor14]

if you're running in local mode it's unlikely your classpath will be from hdfs

[SparkQA]

Test build #36978 has finished for PR 7091 at commit 365838f.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[coderplay]

======================================================================
FAIL: test_time_with_timezone (__main__.SQLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/tests.py", line 716, in test_time_with_timezone
    self.assertEqual(now, now1)
AssertionError: datetime.datetime(2015, 7, 9, 18, 12, 37, 910045) != datetime.datetime(2015, 7, 9, 18, 12, 37, 910044)

Seems it's caused by commit c9e2ef5

[andrewor14]

retest this please

[SparkQA]

Test build #36996 has finished for PR 7091 at commit 365838f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #1034 has finished for PR 7091 at commit 365838f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Merging into master, thanks!

[andrewor14]

@coderplay can you close this now? I already merged it in master. Thanks for your changes.