[SPARK-13474][PROJECT INFRA] Update packaging scripts to push artifacts to home.apache.org #11350
Conversation
JoshRosen
changed the title
|
Test build #51897 has finished for PR 11350 at commit
|
|
Test build #51900 has finished for PR 11350 at commit
|
|
Argh, looks like I need to fix some LFTP problems. Hold off on merge or review for now. |
|
Test build #2580 has finished for PR 11350 at commit
|
|
Test build #51914 has finished for PR 11350 at commit
|
|
+1 |
dev/create-release/release-build.sh
Outdated
|
|
||
| if [ -n "$REMOTE_PARENT_MAX_LENGTH" ]; then | ||
| old_dirs=$($SSH $USER_HOST ls -t $REMOTE_PARENT_DIR | tail -n +$REMOTE_PARENT_MAX_LENGTH) | ||
| old_dirs=$(LFTP nlist $REMOTE_PARENT_DIR | sort | head -n +$REMOTE_PARENT_MAX_LENGTH) |
There was a problem hiding this comment.
It turns out that nlist will return ./ and ../, so we need to filter those out here.
dev/create-release/release-build.sh
Outdated
| $SSH $USER_HOST rm -f "$REMOTE_PARENT_DIR/latest" | ||
| $SSH $USER_HOST ln -s $dest_dir "$REMOTE_PARENT_DIR/latest" | ||
| LFTP rm -f "$REMOTE_PARENT_DIR/latest" | ||
| LFTP ln -s $dest_dir "$REMOTE_PARENT_DIR/latest" |
There was a problem hiding this comment.
The FTP protocol itself does not support symlinks and the older versions of lftp don't have the ln command. I'm going to work around this by uploading the binaries twice, once to the latest folder and once to the other one. With the symlink, at least the change to /latest appeared relatively atomic so you wouldn't see half-uploaded files in there. I'll try to work around this by uploading to dest_dir, moving, then uploading again. This isn't perfect but seems like a decent hack.
|
Test build #51983 has finished for PR 11350 at commit
|
dev/create-release/release-build.sh
Outdated
| LFTP mv $dest_dir "$REMOTE_PARENT_DIR/latest" | ||
| # Re-upload a second time and leave the files in the timestamped upload directory: | ||
| LFTP mkdir -p $dest_dir | ||
| LFTP mput -O $dest_dir spark-* |
There was a problem hiding this comment.
I might need to quote the spark-* so that the parameter expansion is done by lftp rather than the shell.
|
Test build #51986 has finished for PR 11350 at commit
|
|
Test build #51988 has finished for PR 11350 at commit
|
|
Test build #51995 has finished for PR 11350 at commit
|
|
This is now working for package snapshots; will test for doc snapshots soon. See https://home.apache.org/~pwendell/spark-nightly/spark-master-bin/ for what the results look like. |
|
Test build #51998 has finished for PR 11350 at commit
|
|
Test build #52004 has finished for PR 11350 at commit
|
|
Test build #52020 has finished for PR 11350 at commit
|
|
Woohoo! Doc publishing seems to be working now (albeit slowly), so this should be ready for a final sign-off and merge. |
|
Test build #52024 has finished for PR 11350 at commit
|
|
Ping @marmbrus for sign-off, since this impacts your 1.6.1 RC. I'll backport this change to branch-1.6 so you can cut that branch. |
|
LGTM. Thanks for doing this! |
|
I'm going to merge this into master and branch-1.6 so that I can begin packaging a 1.6.1 release candidate. |
…ts to home.apache.org Due to the people.apache.org -> home.apache.org migration, we need to update our packaging scripts to publish artifacts to the new server. Because the new server only supports sftp instead of ssh, we need to update the scripts to use lftp instead of ssh + rsync. Author: Josh Rosen <[email protected]> Closes #11350 from JoshRosen/update-release-scripts-for-apache-home. (cherry picked from commit f77dc4e) Signed-off-by: Josh Rosen <[email protected]>
4 participants
Due to the people.apache.org -> home.apache.org migration, we need to update our packaging scripts to publish artifacts to the new server. Because the new server only supports sftp instead of ssh, we need to update the scripts to use lftp instead of ssh + rsync.