Skip to content

Commit d8fbad8

Browse files
wangyumGitHub Enterprise
authored andcommitted
[HADP-57203] Add gluten bundle jar to $SPARK_HOME/jars (apache#734)
``` -rw-r--r--@ 1 yumwang staff 105761739 Mar 20 10:30 gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0-SNAPSHOT.jar yumwang@G9L07H60PK test % ll spark-3.5.0-ebay.0-SNAPSHOT-bin-ebay/jars | grep "gluten" -rw-r--r--@ 1 yumwang staff 105761739 Mar 20 10:30 gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0-SNAPSHOT.jar ```
1 parent 47cc44d commit d8fbad8

File tree

1 file changed

+11
-1
lines changed

1 file changed

+11
-1
lines changed

build/adlc/release.sh

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,10 +56,20 @@ cp -r $DISTDIR $TARDIR
5656
cp -f $SPARK_HOME/conf/log4j2.properties $TARDIR/conf/log4j2.properties
5757
cp -f $SPARK_HOME/conf/spark-env.sh $TARDIR/conf/spark-env.sh
5858

59-
# 2. delete zookeeper & htrace jar
59+
# 2. delete zookeeper & htrace jar and add gluten jar
6060
rm -f $TARDIR/jars/zookeeper-*.jar
6161
rm -f $TARDIR/jars/htrace-*.jar
6262

63+
# Please add NUOBJECT_ACCESS_KEY and NUOBJECT_SECRET_KEY to Jenkins's Global properties
64+
GLUTEN_JAR_PREFIX="gluten-velox-bundle-spark3.5_${SCALA_BINARY_VERSION}"
65+
GLUTEN_JAR_PATH=$(s3cmd --access_key=$NUOBJECT_ACCESS_KEY --secret_key=$NUOBJECT_SECRET_KEY ls s3://spark3/ | grep "${GLUTEN_JAR_PREFIX}" | awk '{print $4}')
66+
if [[ -z $GLUTEN_JAR_PATH ]]; then
67+
echo "gluten jar not found in s3://spark3/"
68+
else
69+
s3cmd --access_key=$NUOBJECT_ACCESS_KEY --secret_key=$NUOBJECT_SECRET_KEY get ${GLUTEN_JAR_PATH}
70+
mv ${GLUTEN_JAR_PREFIX}* $TARDIR/jars/
71+
fi
72+
6373
# 3. prepare spark archive
6474
cd $TARDIR/jars/ && zip $ARCHIVE_NAME *.jar && cd -
6575
mv $TARDIR/jars/$ARCHIVE_NAME .

0 commit comments

Comments
 (0)