add new aarch64 linux nodes

[futurejones]

Part 3 of restructuring the aarch64 server resources.

Add new build nodes for -

• Amazon Linux 2
• CentOS 8
• Ubuntu 18.04
• Ubuntu 20.04

I have setup a docker hub repo with pre built images for these OS's to simplify the setup.
https://hub.docker.com/u/swiftarm
Details of Dockerfiles used are here -
https://github.com/futurejones/swift-arm64/tree/master/swift-ci-docker

Sample of Jenkins pipeline I have been using to test docker images and server setup -
https://github.com/futurejones/swift-arm64/blob/master/swift-ci-docker/Jenkinsfiles/Jenkinsfile

@shahmishal The server is setup and should be all ready to go except for your user keys and docker permissions. Will these be the same as before?

[shahmishal]

@futurejones Email sent for additional information

[shahmishal]

Node attached, working on setting up new jobs.

[shahmishal]

@Benkovsky20000 Thanks for reviewing the changes, can you explain the reason for last two comments?

N

docker pull swiftarm/ubuntu_xenial

[futurejones]

@shahmishal @tkremenek - Who is "Benkovsky20000"? - I thought only Mishal could approve PR's for the community CI.

[shahmishal]

Community members can provide feedback, however it does not count toward mergeability.

I have already reported spam comments to GitHub on multiple PRs.

[futurejones]

@tkremenek @shahmishal Is there a reason why this taking so long? It has been over 1 month since I submitted this PR.
If you need help I am more than happy to assist with the Jenkins CI setup.

[shahmishal]

Sorry for taking long time get this completed. This is the first pull request which is going to support Docker/Jenkins Pipeline on ci-external.swift.org, I am working on making sure everything is secure with Docker/Jenkins Pipeline config.

[futurejones]

ok thanks.

[shahmishal]

master branch bots created:

• Amazon Linux 2 - https://ci-external.swift.org/job/oss-swift-RA-amazon-linux-2-aarch64/
• CentOS 8 - https://ci-external.swift.org/job/oss-swift-RA-centos8-aarch64/
• Ubuntu 18.04 - https://ci-external.swift.org/job/oss-swift-RA-ubuntu-18.04-aarch64/
• Ubuntu 20.04 - https://ci-external.swift.org/job/oss-swift-RA-ubuntu-20.04-aarch64/

Node: https://ci-external.swift.org/computer/ubuntu-18.04-aarch64-02/

[shahmishal]

Jenkinsfile:

env.DOCKER_IMAGE, env.OS, env.OS_VERSION are different for each platform.

node('ubuntu-18.04-aarch64-02'){
    
    deleteDir()
    
    def DATE = sh(script: "echo `date +%Y-%m-%d`", returnStdout: true).trim()
    env.SWIFT_BRANCH = 'master'
    env.SWIFT_SCHEME = 'master'
    env.SWIFT_VERSION = '5.3'
    env.RELEASE = 'dev' // change to 'release' for RELEASE versions
    env.DOCKER_IMAGE = 'swiftarm/amzn_2:latest'
    env.OS = 'amazonlinux'
    env.OS_VERSION = '2'
    env.WORK_DIR = '/home/build-user'
    stage('Checkout'){
        checkout([$class: 'GitSCM', branches: [[name: "*/${SWIFT_BRANCH}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'swift']], submoduleCfg: [], userRemoteConfigs: [[url: 'https://github.com/apple/swift.git']]])
        echo "scheme=${SWIFT_SCHEME}: cloning supporting repos"
        sh "./swift/utils/update-checkout --clone --scheme ${SWIFT_SCHEME}"
    }
    stage('Pull Docker Image'){
        sh "docker pull ${DOCKER_IMAGE}"
    }
    stage('Build Swift'){
        try {
            echo 'Building toolchain'
            sh "docker rm -f ${JOB_NAME} || true"
            sh "docker volume rm ${JOB_NAME} || true"
            sh """
                docker run \
                  --cap-add=SYS_PTRACE \
                  --security-opt seccomp=unconfined \
                  --name ${JOB_NAME} \
                   -w ${WORK_DIR} \
                  -v ${WORKSPACE}:/source \
                  -v ${JOB_NAME}:${WORK_DIR} \
                  ${DOCKER_IMAGE} \
                  /bin/bash -lc \
                  'cp -r /source/* ${WORK_DIR}; \
                  ./swift/utils/build-script --jobs=24\
                  --preset buildbot_linux,no_test \
                  install_destdir=${WORK_DIR}/swift-install \
                  installable_package=${WORK_DIR}/output/${JOB_NAME}-${RELEASE}-aarch64-${BUILD_ID}-${DATE}.tar.gz'
                 """
        }finally{
            sh """
                docker rm -f ${env.JOB_NAME}
                docker volume rm --force ${env.JOB_NAME} || true
               """
        }
    }
}

[shahmishal]

Sorry it took a while to get it configured, please let me know if you see any issues. Thanks!

[futurejones]

@shahmishal, thanks for getting this done.

There is one small problem with the Jenkinsfile in the installation package path.
installable_package=${WORK_DIR}/output/${JOB_NAME}-${RELEASE}-aarch64-${BUILD_ID}-${DATE}.tar.gz
The output directory needs to be created before the work directory is copied into the docker volume.
or just remove the output directory from the path.
installable_package=${WORK_DIR}/${JOB_NAME}-${RELEASE}-aarch64-${BUILD_ID}-${DATE}.tar.gz

This is causing the builds to fail with the following error the following error -

~/swift-install ~
+ tar -c -z -f /home/build-user/output/oss-swift-RA-amazon-linux-2-aarch64-dev-aarch64-7-2020-07-15.tar.gz --owner=0 --group=0 usr/
tar (child): /home/build-user/output/oss-swift-RA-amazon-linux-2-aarch64-dev-aarch64-7-2020-07-15.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: /home/build-user/output/oss-swift-RA-amazon-linux-2-aarch64-dev-aarch64-7-2020-07-15.tar.gz: Cannot write: Broken pipe
tar: Error is not recoverable: exiting now

[futurejones]

Also the other Swift - Ubuntu 18.04 Linux aarch64 (master) smoketest job is saying This project is currently disabled.
Is this a naming conflict? Should we change this job name to Swift - Ubuntu 18.04 Linux aarch64 (master/smoketest) or something similar?
https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-18.04-aarch64/

[shahmishal]

@futurejones Updated following jobs by removing output from the path.

• Amazon Linux 2 - https://ci-external.swift.org/job/oss-swift-RA-amazon-linux-2-aarch64/
• CentOS 8 - https://ci-external.swift.org/job/oss-swift-RA-centos8-aarch64/
• Ubuntu 18.04 - https://ci-external.swift.org/job/oss-swift-RA-ubuntu-18.04-aarch64/
• Ubuntu 20.04 - https://ci-external.swift.org/job/oss-swift-RA-ubuntu-20.04-aarch64/

Re-enabled Ubuntu 18.04 job with smoke test in the title.

• https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-18.04-aarch64/