Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
034ae30
[SPARK-26033][PYTHON][TESTS] Break large ml/tests.py file into smalle…
BryanCutler Nov 18, 2018
bbbdaa8
[SPARK-26105][PYTHON] Clean unittest2 imports up that were added for …
HyukjinKwon Nov 19, 2018
630e25e
[SPARK-26026][BUILD] Published Scaladoc jars missing from Maven Central
srowen Nov 19, 2018
ce2cdc3
[SPARK-26043][CORE] Make SparkHadoopUtil private to Spark
srowen Nov 19, 2018
b58b1fd
[SPARK-26068][CORE] ChunkedByteBufferInputStream should handle empty …
LinhongLiu Nov 19, 2018
48ea64b
[SPARK-26112][SQL] Update since versions of new built-in functions.
ueshin Nov 19, 2018
35c5516
[SPARK-26024][SQL] Update documentation for repartitionByRange
JulienPeloton Nov 19, 2018
219b037
[SPARK-26071][SQL] disallow map as map key
cloud-fan Nov 19, 2018
32365f8
[SPARK-26090][CORE][SQL][ML] Resolve most miscellaneous deprecation a…
srowen Nov 19, 2018
86cc907
This is a dummy commit to trigger ASF git sync
srowen Nov 19, 2018
a09d5ba
[SPARK-26107][SQL] Extend ReplaceNullWithFalseInPredicate to support …
rednaxelafx Nov 20, 2018
a00aaf6
[MINOR][YARN] Make memLimitExceededLogMessage more clean
wangyum Nov 20, 2018
c34c422
[SPARK-26076][BUILD][MINOR] Revise ambiguous error message from load-…
gengliangwang Nov 20, 2018
ab61ddb
[SPARK-26118][WEB UI] Introducing spark.ui.requestHeaderSize for sett…
attilapiros Nov 20, 2018
db136d3
[SPARK-26084][SQL] Fixes unresolved AggregateExpression.references ex…
ssimeonov Nov 20, 2018
42c4838
[BUILD] refactor dev/lint-python in to something readable
shaneknapp Nov 20, 2018
23bcd6c
[SPARK-26043][HOTFIX] Hotfix a change to SparkHadoopUtil that doesn't…
srowen Nov 21, 2018
4785105
[SPARK-26124][BUILD] Update plugins to latest versions
srowen Nov 21, 2018
2df34db
[SPARK-26122][SQL] Support encoding for multiLine in CSV datasource
MaxGekk Nov 21, 2018
4b7f7ef
[SPARK-26120][TESTS][SS][SPARKR] Fix a streaming query leak in Struct…
zsxwing Nov 21, 2018
a480a62
[SPARK-25954][SS] Upgrade to Kafka 2.1.0
dongjoon-hyun Nov 21, 2018
540afc2
[SPARK-26109][WEBUI] Duration in the task summary metrics table and t…
shahidki31 Nov 21, 2018
6bbdf34
[SPARK-8288][SQL] ScalaReflection can use companion object constructor
drewrobb Nov 21, 2018
07a700b
[SPARK-26129][SQL] Instrumentation for per-query planning time
rxin Nov 21, 2018
81550b3
[SPARK-26066][SQL] Move truncatedString to sql/catalyst and add spark…
MaxGekk Nov 21, 2018
4aa9ccb
[SPARK-26127][ML] Remove deprecated setters from tree regression and …
mgaido91 Nov 21, 2018
9b48107
[SPARK-25957][K8S] Make building alternate language binding docker im…
ramaddepally Nov 21, 2018
ce7b57c
[SPARK-26106][PYTHON] Prioritizes ML unittests over the doctests in P…
HyukjinKwon Nov 22, 2018
38628dd
[SPARK-25935][SQL] Prevent null rows from JSON parser
MaxGekk Nov 22, 2018
ab2eafb
[SPARK-26085][SQL] Key attribute of non-struct type under typed aggre…
viirya Nov 22, 2018
8d54bf7
[SPARK-26099][SQL] Verification of the corrupt column in from_csv/fro…
MaxGekk Nov 22, 2018
15c0384
[SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix java.version pro…
tasanuma Nov 22, 2018
ab00533
[SPARK-26129][SQL] edge behavior for QueryPlanningTracker.topRulesByT…
rxin Nov 22, 2018
aeda76e
[GRAPHX] Remove unused variables left over by previous refactoring.
huonw Nov 22, 2018
dd8c179
[SPARK-25867][ML] Remove KMeans computeCost
mgaido91 Nov 22, 2018
d81d95a
[SPARK-19368][MLLIB] BlockMatrix.toIndexedRowMatrix() optimization fo…
uzadude Nov 22, 2018
1d766f0
[SPARK-26144][BUILD] `build/mvn` should detect `scala.version` based …
dongjoon-hyun Nov 22, 2018
76aae7f
[SPARK-24553][UI][FOLLOWUP] Fix unnecessary UI redirect
jerryshao Nov 22, 2018
0ec7b99
[SPARK-26021][SQL] replace minus zero with zero in Platform.putDouble…
Nov 23, 2018
1d3dd58
[SPARK-25954][SS][FOLLOWUP][TEST-MAVEN] Add Zookeeper 3.4.7 test depe…
dongjoon-hyun Nov 23, 2018
92fc0a8
[SPARK-26069][TESTS][FOLLOWUP] Add another possible error message
zsxwing Nov 23, 2018
466d011
[SPARK-26117][CORE][SQL] use SparkOutOfMemoryError instead of OutOfMe…
heary-cao Nov 23, 2018
8e8d117
[SPARK-26108][SQL] Support custom lineSep in CSV datasource
MaxGekk Nov 23, 2018
ecb785f
[SPARK-26038] Decimal toScalaBigInt/toJavaBigInteger for decimals not…
juliuszsompolski Nov 23, 2018
de84899
[SPARK-26140] Enable custom metrics implementation in shuffle reader
rxin Nov 23, 2018
7f5f7a9
[SPARK-25786][CORE] If the ByteBuffer.hasArray is false , it will thr…
10110346 Nov 24, 2018
0f56977
[SPARK-26156][WEBUI] Revise summary section of stage page
gengliangwang Nov 24, 2018
eea4a03
[MINOR][K8S] Invalid property "spark.driver.pod.name" is referenced i…
Leemoonsoo Nov 25, 2018
41d5aae
[SPARK-26148][PYTHON][TESTS] Increases default parallelism in PySpark…
HyukjinKwon Nov 25, 2018
c5daccb
[MINOR] Update all DOI links to preferred resolver
katrinleinweber Nov 25, 2018
9414578
[SPARK-25908][SQL][FOLLOW-UP] Add back unionAll
gatorsmile Nov 25, 2018
6339c8c
[SPARK-24762][SQL] Enable Option of Product encoders
viirya Nov 26, 2018
6ab8485
[SPARK-26169] Create DataFrameSetOperationsSuite
gatorsmile Nov 26, 2018
6bb60b3
[SPARK-26168][SQL] Update the code comments in Expression and Aggregate
gatorsmile Nov 26, 2018
1bb60ab
[SPARK-26153][ML] GBT & RandomForest avoid unnecessary `first` job to…
zhengruifeng Nov 26, 2018
2512a1d
[SPARK-26121][STRUCTURED STREAMING] Allow users to define prefix of K…
Nov 26, 2018
3df307a
[SPARK-25960][K8S] Support subpath mounting with Kubernetes
NiharS Nov 26, 2018
76ef02e
[SPARK-21809] Change Stage Page to use datatables to support sorting …
Nov 26, 2018
fbf62b7
[SPARK-25451][SPARK-26100][CORE] Aggregated metrics table doesn't sho…
shahidki31 Nov 26, 2018
6f1a1c1
[SPARK-25451][HOTFIX] Call stage.attemptNumber instead of attemptId.
Nov 26, 2018
9deaa72
[INFRA] Close stale PR.
Nov 26, 2018
c995e07
[SPARK-26140] followup: rename ShuffleMetricsReporter
rxin Nov 27, 2018
1c487f7
[SPARK-24762][SQL][FOLLOWUP] Enable Option of Product encoders
viirya Nov 27, 2018
85383d2
[SPARK-25860][SPARK-26107][FOLLOW-UP] Rule ReplaceNullWithFalseInPred…
gatorsmile Nov 27, 2018
6a064ba
[SPARK-26141] Enable custom metrics implementation in shuffle write
rxin Nov 27, 2018
65244b1
[SPARK-23356][SQL][TEST] add new test cases for a + 1,a + b and Rand …
heary-cao Nov 27, 2018
2d89d10
[SPARK-26025][K8S] Speed up docker image build on dev repo.
Nov 27, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-26144][BUILD] build/mvn should detect scala.version based …
…on `scala.binary.version`

## What changes were proposed in this pull request?

Currently, `build/mvn` downloads and uses **Scala 2.12.7** in `Scala-2.11` Jenkins job. The root cause is `build/mvn` got the first match from `pom.xml` blindly.

- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.11/6/consoleFull
```
exec: curl -s -L https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz
exec: curl -s -L https://downloads.lightbend.com/scala/2.12.7/scala-2.12.7.tgz
exec: curl -s -L https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz
```

## How was this patch tested?

Manual.
```
$ build/mvn clean
exec: curl --progress-bar -L https://downloads.lightbend.com/scala/2.12.7/scala-2.12.7.tgz
...
$ git clean -fdx
$ dev/change-scala-version.sh 2.11
$ build/mvn clean
exec: curl --progress-bar -L https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz
```

Closes apache#23118 from dongjoon-hyun/SPARK-26144.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Nov 22, 2018
commit 1d766f0e222c24e8e8cad68e664e83f4f71f7541
3 changes: 2 additions & 1 deletion build/mvn
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,8 @@ install_zinc() {
# the build/ folder
install_scala() {
# determine the Scala version used in Spark
local scala_version=`grep "scala.version" "${_DIR}/../pom.xml" | head -n1 | awk -F '[<>]' '{print $3}'`
local scala_binary_version=`grep "scala.binary.version" "${_DIR}/../pom.xml" | head -n1 | awk -F '[<>]' '{print $3}'`
local scala_version=`grep "scala.version" "${_DIR}/../pom.xml" | grep ${scala_binary_version} | head -n1 | awk -F '[<>]' '{print $3}'`
local scala_bin="${_DIR}/scala-${scala_version}/bin/scala"
local TYPESAFE_MIRROR=${TYPESAFE_MIRROR:-https://downloads.lightbend.com}

Expand Down