Skip to content

Build OpenSearch in CD workflow in order to build security plugin #1364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vrozov merged 3 commits into from
Aug 7, 2021
Merged

Build OpenSearch in CD workflow in order to build security plugin #1364

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bfc1d2b
Build OpenSearch in CD workflow in order to build security plugin
cliu123 Jul 30, 2021
e0d8e5a
Remove the unnecessary configurations that were for uploading artifac…
cliu123 Aug 2, 2021
03a261d
Move Build OpenSearch step after Cache
cliu123 Aug 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Build OpenSearch in CD workflow in order to build security plugin
  • Loading branch information
@cliu123
cliu123 committed Jul 30, 2021
commit bfc1d2bdbc741172a35f24f3336931967800ce8c
88 changes: 50 additions & 38 deletions .github/workflows/cd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,25 @@ jobs:

steps:

- name: Set up JDK 11
- name: Set up JDK 14
uses: actions/setup-java@v1
with:
java-version: 11.0.x
java-version: 14.0.x
server-id: sonatype-nexus-staging
server-username: SONATYPE_USER
server-password: SONATYPE_PASSWORD

- name: Checkout OpenSearch
uses: actions/checkout@v2
with:
repository: 'opensearch-project/OpenSearch'
path: OpenSearch
ref: '1.0'

- name: Build OpenSearch
Copy link
Contributor

@vrozov vrozov Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this step be after Cache Maven packages step?

Copy link
Member Author

@cliu123 cliu123 Aug 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't OpenSearch artifact built be cached?
If moving Build OpenSearch step and putting it after Cache Maven packages step, both of Build OpenSearch and Build steps will be after Cache Maven packages step. Shouldn't the build outputs be cached?

Copy link
Contributor

@vrozov vrozov Aug 3, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the point of caching OpenSearch maven packages when they are rebuilt anyway?

Copy link
Member Author

@cliu123 cliu123 Aug 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. In this case, only downloaded dependencies should be cached. There is no point of caching build outputs.
Moved. Thanks!

working-directory: ./OpenSearch
run: ./gradlew publishToMavenLocal -Dbuild.snapshot=false

- name: Checkout security
uses: actions/checkout@v2

Expand All @@ -41,42 +52,43 @@ jobs:
zip -r artifacts.zip artifacts
echo "TAG_VERSION=${GITHUB_REF/refs\/tags\//}" >> $GITHUB_ENV

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_STAGING_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_STAGING_SECRET_ACCESS_KEY }}
aws-region: us-west-2

- name: Upload Artifacts to S3
run: |
zip=`ls artifacts/*.zip`
rpm=`ls artifacts/*.rpm`
deb=`ls artifacts/*.deb`

# Inject the build number before the suffix
zip_outfile=`basename ${zip%.zip}-build-${GITHUB_RUN_NUMBER}.zip`
rpm_outfile=`basename ${rpm%.rpm}-build-${GITHUB_RUN_NUMBER}.rpm`
deb_outfile=`basename ${deb%.deb}-build-${GITHUB_RUN_NUMBER}.deb`

s3_prefix="s3://staging.artifacts.opendistroforelasticsearch.amazon.com/snapshots/elasticsearch-plugins/security/"

echo "Copying ${zip} to ${s3_prefix}${zip_outfile}"
aws s3 cp --quiet $zip ${s3_prefix}${zip_outfile}

echo "Copying ${rpm} to ${s3_prefix}${rpm_outfile}"
aws s3 cp --quiet $rpm ${s3_prefix}${rpm_outfile}

echo "Copying ${deb} to ${s3_prefix}${deb_outfile}"
aws s3 cp --quiet $deb ${s3_prefix}${deb_outfile}

- name: Upload Artifacts to Maven Central
env:
SONATYPE_USER: ${{ secrets.SONATYPE_USER }}
SONATYPE_PASSWORD: ${{ secrets.SONATYPE_PASSWORD }}
run: |
gpg --batch --import --no-tty <(echo -e "${{ secrets.PGP_PRIVATE_KEY }}")
mvn -B deploy -Padvanced -Prelease -DskipTests -Dgpg.passphrase=${{ secrets.PGP_PASSPHRASE }}
# TODO: Update the following configuration commented out once the new CI/CD process of OpenSearch is confirmed.
# - name: Configure AWS Credentials
# uses: aws-actions/configure-aws-credentials@v1
# with:
# aws-access-key-id: ${{ secrets.AWS_STAGING_ACCESS_KEY_ID }}
# aws-secret-access-key: ${{ secrets.AWS_STAGING_SECRET_ACCESS_KEY }}
# aws-region: us-west-2
#
# - name: Upload Artifacts to S3
# run: |
# zip=`ls artifacts/*.zip`
# rpm=`ls artifacts/*.rpm`
# deb=`ls artifacts/*.deb`
#
# # Inject the build number before the suffix
# zip_outfile=`basename ${zip%.zip}-build-${GITHUB_RUN_NUMBER}.zip`
# rpm_outfile=`basename ${rpm%.rpm}-build-${GITHUB_RUN_NUMBER}.rpm`
# deb_outfile=`basename ${deb%.deb}-build-${GITHUB_RUN_NUMBER}.deb`
#
# s3_prefix="s3://staging.artifacts.opendistroforelasticsearch.amazon.com/snapshots/elasticsearch-plugins/security/"
#
# echo "Copying ${zip} to ${s3_prefix}${zip_outfile}"
# aws s3 cp --quiet $zip ${s3_prefix}${zip_outfile}
#
# echo "Copying ${rpm} to ${s3_prefix}${rpm_outfile}"
# aws s3 cp --quiet $rpm ${s3_prefix}${rpm_outfile}
#
# echo "Copying ${deb} to ${s3_prefix}${deb_outfile}"
# aws s3 cp --quiet $deb ${s3_prefix}${deb_outfile}
#
# - name: Upload Artifacts to Maven Central
# env:
# SONATYPE_USER: ${{ secrets.SONATYPE_USER }}
# SONATYPE_PASSWORD: ${{ secrets.SONATYPE_PASSWORD }}
# run: |
# gpg --batch --import --no-tty <(echo -e "${{ secrets.PGP_PRIVATE_KEY }}")
# mvn -B deploy -Padvanced -Prelease -DskipTests -Dgpg.passphrase=${{ secrets.PGP_PASSPHRASE }}

- name: Create Github Draft Release
id: create_release
Expand Down