Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
42988ef
Implement a Docker image generator gradle plugin for Spark applications.
mccheah Jun 8, 2018
4b32aa9
Fix circle
mccheah Jun 9, 2018
6bded05
Fix more circle
mccheah Jun 9, 2018
2bb1702
No need to setup docker for compilation only
mccheah Jun 9, 2018
cad9d41
Fix gradle
mccheah Jun 9, 2018
92139c0
Add license headers
mccheah Jun 14, 2018
5a847bb
Merge remote-tracking branch 'palantir/master' into add-docker-image-…
mccheah Jun 15, 2018
46f23c9
Address comments.
mccheah Jun 25, 2018
71b835a
Merge remote-tracking branch 'palantir/master' into add-docker-image-…
mccheah Jun 25, 2018
4149dfd
Remove extra task
mccheah Jun 25, 2018
814978f
Remove 2.11
mccheah Jun 25, 2018
e123548
Remove extra script
mccheah Jun 25, 2018
6bf957a
Remove some imports
mccheah Jun 25, 2018
3194cd1
Add more tests
mccheah Jun 26, 2018
53c0b1a
Don't bundle resources in tgz, just include individual files in resou…
mccheah Jul 18, 2018
d046ab7
Merge remote-tracking branch 'palantir/master' into add-docker-image-…
mccheah Jul 18, 2018
186b3a3
Fix license placement.
mccheah Jul 18, 2018
0f6be98
Use InputFile and not Input
mccheah Jul 18, 2018
6352a26
Fix build
mccheah Jul 18, 2018
1fa699c
Add back K8s integration tests
mccheah Jul 19, 2018
de403f0
Revert changes to SparkBuild
mccheah Jul 19, 2018
cad1bb1
Remove hive version 2.0.2 from test suite
mccheah Jul 19, 2018
1ae4f61
Use shared Spark session for unsafe row suite
mccheah Jul 19, 2018
9ea102a
Merge remote-tracking branch 'palantir/master' into add-docker-image-…
mccheah Jul 23, 2018
94c6c55
Revert "Use shared Spark session for unsafe row suite"
mccheah Jul 23, 2018
999b5f8
Address comments.
mccheah Aug 17, 2018
6704dd5
Fix build script
mccheah Aug 17, 2018
e2c862d
Fix build again
mccheah Aug 17, 2018
3017341
Fix licenses and ignore build dir licenses
mccheah Aug 17, 2018
fb57b7e
Remove extraneous scripts
mccheah Aug 31, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add license headers
  • Loading branch information
mccheah committed Jun 14, 2018
commit 92139c0c7d50485b563fdf44fe649057b7e84b90
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,7 @@ jobs:
- *checkout-code
- setup_remote_docker
- run: |
dev/run-spark-docker-gradle-plugin-tests.py | tee /tmp/run-spark-docker-gradle-plugin-tests.log
dev/run-spark-docker-gradle-plugin-tests.sh | tee /tmp/run-spark-docker-gradle-plugin-tests.log

run-backcompat-tests:
# depends on build-sbt
Expand Down
18 changes: 18 additions & 0 deletions dev/build-spark-docker-gradle-plugin.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,22 @@
#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

set -euo pipefail
ROOT=$(git rev-parse --show-toplevel)
cd "$ROOT/resource-managers/kubernetes/docker"
Expand Down
6 changes: 0 additions & 6 deletions dev/run-spark-docker-gradle-plugin-tests.py

This file was deleted.

23 changes: 23 additions & 0 deletions dev/run-spark-docker-gradle-plugin-tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
set -euo pipefail
ROOT=$(git rev-parse --show-toplevel)
cd "$ROOT/resource-managers/kubernetes/docker"
./gradlew --info test

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do these scripts just assume they'll be executing in kubernetes/docker? I'd rather be explicit about this in circle, something like ./build/gradlew -p resource-managers/kubernetes/docker --info test

cd -
17 changes: 17 additions & 0 deletions resource-managers/kubernetes/docker/build.gradle
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

buildscript {
repositories {
jcenter()
Expand Down
17 changes: 17 additions & 0 deletions resource-managers/kubernetes/docker/gradle/publish.gradle
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

apply plugin: 'com.jfrog.bintray'
apply plugin: 'nebula.maven-base-publish'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use nebula.maven-publish and remove line below as well as bintrayupload dependencies and publications. Just make publish depend on bintrayupload (see baseline for reference)

apply plugin: 'nebula.maven-resolved-dependencies'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-4.7-bin.zip

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

latest

Expand Down
22 changes: 22 additions & 0 deletions resource-managers/kubernetes/docker/gradlew

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

buildscript {
repositories {
jcenter()
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.spark.test;

import java.util.List;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,22 @@
#!/bin/bash
#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

set -eux
SPARK_REPO_ROOT=$(git rev-parse --show-toplevel)
SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
Expand Down
17 changes: 17 additions & 0 deletions resource-managers/kubernetes/docker/settings.gradle
Original file line number Diff line number Diff line change
@@ -1 +1,18 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

rootProject.name = 'spark-docker-image-generator_2.11'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't add 2.11 - this isn't a scala project?

17 changes: 17 additions & 0 deletions resource-managers/kubernetes/docker/versions.props
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

org.apache.commons:commons-compress = 1.8.1
commons-io:commons-io = 2.4
com.fasterxml.jackson:* = 2.8.0
Expand Down