From deaab242ceb8587783722826714b31aea146e800 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Wed, 13 Nov 2019 13:57:01 -0500 Subject: [PATCH 01/26] update default GeoLambda base version in Python layer --- python/Dockerfile | 2 +- python/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/python/Dockerfile b/python/Dockerfile index 6d947b7..63681a2 100644 --- a/python/Dockerfile +++ b/python/Dockerfile @@ -1,4 +1,4 @@ -ARG VERSION=1.2.0 +ARG VERSION=2.0.0 FROM developmentseed/geolambda:${VERSION} LABEL maintainer="Development Seed " diff --git a/python/README.md b/python/README.md index b199342..edc5a29 100644 --- a/python/README.md +++ b/python/README.md @@ -19,7 +19,7 @@ An example Lambda handler is located at [lambda/lambda_function.py](lambda/lambd Now, use the [Dockerfile](Dockerfile) can be used to create a new Docker image based on any version of GeoLambda with any version of Python by providing the versions as build arguments to `docker run`. This will install the specified version of Python along with any Python packages provided in [requirements.txt](requirements.txt). ``` -$ VERSION=1.2.0 +$ VERSION=2.0.0 $ docker build . --build-arg VERSION=${VERSION} --build-arg PYVERSION=3.7.4 -t :latest ``` From ffb88bf7facc07917dc807a90a72843b829d254e Mon Sep 17 00:00:00 2001 From: Scott Eilerman Date: Wed, 13 May 2020 16:27:59 -0600 Subject: [PATCH 02/26] add --symlinks flag to zip command --- bin/package.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/package.sh b/bin/package.sh index ba2dc0e..d72911a 100755 --- a/bin/package.sh +++ b/bin/package.sh @@ -21,4 +21,4 @@ rsync -ax $PREFIX/share/proj $DEPLOY_DIR/share/ # zip up deploy package cd $DEPLOY_DIR -zip -ruq ../lambda-deploy.zip ./ +zip -ruq --symlinks ../lambda-deploy.zip ./ From fbec8ec862d5881f699d0189cf407a3bfdf0b69d Mon Sep 17 00:00:00 2001 From: Blue Tyson Date: Wed, 19 Aug 2020 05:31:40 +0930 Subject: [PATCH 03/26] Update README.md Base --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 70a18fe..1b32b7b 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ The GeoLambda project provides public Docker images and AWS Lambda Layers containing common geospatial native libraries. GeoLambda contains the libraries PROJ.5, GEOS, GeoTIFF, HDF4/5, SZIP, NetCDF, OpenJPEG, WEBP, ZSTD, and GDAL. For some applications you may wish to minimize the size of the libraries by excluding unused libraries, or you may wish to add other libraries. In this case this repository can be used as a template to create your own Docker image or Lambda Layer following the instructions in this README. -This repository also contains additional images and layers for specific runtimes. Using them as a Layer assumes the use of the basee GeoLambda layer. +This repository also contains additional images and layers for specific runtimes. Using them as a Layer assumes the use of the base GeoLambda layer. - [Python](python/README.md) From 70ec14daa6c31c5d809105d0692c3309a5310de4 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Fri, 16 Oct 2020 15:07:00 -0400 Subject: [PATCH 04/26] CHANGELOG updates --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 484033b..4fb93a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ## [Unreleased] +### Fixed +- README updates regarding versions + +### Changed +- Keep symlinks when zipping, resulting in smaller deploy package + ## [v2.0.0] - 2019-10-25 From 44b5e5bfba73fa18ac3a6716db1acbb412118f1c Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Fri, 16 Oct 2020 19:11:14 -0400 Subject: [PATCH 05/26] add rsync to dependencies --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 61497da..9426681 100644 --- a/Dockerfile +++ b/Dockerfile @@ -6,7 +6,7 @@ LABEL authors="Matthew Hanson " # install system libraries RUN \ yum makecache fast; \ - yum install -y wget libpng-devel nasm; \ + yum install -y wget libpng-devel nasm rsync; \ yum install -y bash-completion --enablerepo=epel; \ yum clean all; \ yum autoremove From 60f3406b4dd9be2c861da19d06ba76efb95fdcb6 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Fri, 16 Oct 2020 19:11:30 -0400 Subject: [PATCH 06/26] set envvars when running test docker --- build-and-test.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/build-and-test.sh b/build-and-test.sh index bc7309a..2bb67d0 100755 --- a/build-and-test.sh +++ b/build-and-test.sh @@ -9,4 +9,5 @@ cd python docker build . --build-arg VERSION=${VERSION} -t developmentseed/geolambda:${VERSION}-python docker run -v ${PWD}:/home/geolambda -t developmentseed/geolambda:${VERSION}-python package-python.sh -docker run --rm -v ${PWD}/lambda:/var/task -v ${PWD}/../lambda:/opt lambci/lambda:python3.7 lambda_function.lambda_handler '{}' +docker run -e GDAL_DATA=/opt/share/gdal -e PROJ_LIB=/opt/share/proj \ + --rm -v ${PWD}/lambda:/var/task -v ${PWD}/../lambda:/opt lambci/lambda:python3.7 lambda_function.lambda_handler '{}' From 93606962ed351ddeeb584fc6ecc13ebb10a6dcd9 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Fri, 16 Oct 2020 19:21:09 -0400 Subject: [PATCH 07/26] deploy to additional regions on release, resolves #83 and #86 --- .circleci/config.yml | 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 00a0f11..a3012a4 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -50,7 +50,7 @@ jobs: cd .. # deploy public Lambda layers pip install awscli - for region in us-east-1 us-west-2 eu-central-1 + for region in us-east-1 us-west-2 eu-central-1 eu-west-2 eu-north-1 do LVERSION="$(aws lambda publish-layer-version --region ${region} \ --layer-name geolambda --license-info 'MIT' \ diff --git a/README.md b/README.md index 1b32b7b..3d082ad 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ When using GeoLambda some environment variables need to be set. These are set in ### Lambda Layers -If you just wish to use the publicly available Lambda layers you will need the ARN for the layer in the same region as your Lambda function. Currently, GeoLambda layers are available in `us-east-1`, `us-west-2`, and `eu-central-1`. If you want to use it in another region please file an issue or you can also create your own layer using this repository (see instructions below on 'Create a new version'). +If you just wish to use the publicly available Lambda layers you will need the ARN for the layer in the same region as your Lambda function. Currently, the latest GeoLambda layers are deployed in `us-east-1`, `us-west-2`, `eu-central-1`, `eu-west-2`, and `eu-north-1`. If you want to use it in another region please file an issue or you can also create your own layer using this repository (see instructions below on 'Create a new version'). #### v2.0.0 From bf674e7fd2e3cefdec1aef1505dc20914217ad33 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Fri, 16 Oct 2020 19:21:30 -0400 Subject: [PATCH 08/26] fix README, resolves #87 --- python/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/python/README.md b/python/README.md index edc5a29..30a1da6 100644 --- a/python/README.md +++ b/python/README.md @@ -39,7 +39,7 @@ This will create geolambda base layer (`lambda-deploy.zip`) file and `/geolambda $ docker run --rm -v $PWD:/home/geolambda -it developmentseed/geolambda:${VERSION} package.sh ``` -This will copy site-packages in /geolambda/python/lambda directory needed for development. Also, it will create a `lambda-layer-deploy.zip` file in the current directory. +This will copy site-packages in /geolambda/python/lambda directory needed for development. Also, it will create a `lambda-deploy.zip` file in the current directory. Contents of the zip file are following AWS conventions: https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path ``` @@ -68,7 +68,7 @@ Deploy the Lambda layer by using AWS CLI. $ aws lambda publish-layer-version \ --layer-name geolambda-python \ --description "Python bindings for GDAL" \ - --zip-file fileb://lambda-layer-deploy.zip + --zip-file fileb://lambda-deploy.zip ``` ### Pre-built images From 8a7aee46ea07cb0b8d1ab46328a64a945dca2132 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Sat, 17 Oct 2020 00:29:40 -0400 Subject: [PATCH 09/26] include base geolambda files in python geolambda --- bin/package.sh | 4 +--- python/bin/package-python.sh | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/bin/package.sh b/bin/package.sh index d72911a..d737f0e 100755 --- a/bin/package.sh +++ b/bin/package.sh @@ -3,8 +3,6 @@ # directory used for deployment export DEPLOY_DIR=lambda -echo Creating deploy package - # make deployment directory and add lambda handler mkdir -p $DEPLOY_DIR/lib @@ -21,4 +19,4 @@ rsync -ax $PREFIX/share/proj $DEPLOY_DIR/share/ # zip up deploy package cd $DEPLOY_DIR -zip -ruq --symlinks ../lambda-deploy.zip ./ +zip -ruq ../lambda-deploy.zip ./ diff --git a/python/bin/package-python.sh b/python/bin/package-python.sh index fc6f7ce..ca4a656 100755 --- a/python/bin/package-python.sh +++ b/python/bin/package-python.sh @@ -3,6 +3,20 @@ # directory used for development export DEPLOY_DIR=lambda +# make deployment directory and add lambda handler +mkdir -p $DEPLOY_DIR/lib + +# copy libs +cp -P ${PREFIX}/lib/*.so* $DEPLOY_DIR/lib/ +cp -P ${PREFIX}/lib64/libjpeg*.so* $DEPLOY_DIR/lib/ + +strip $DEPLOY_DIR/lib/* || true + +# copy GDAL_DATA files over +mkdir -p $DEPLOY_DIR/share +rsync -ax $PREFIX/share/gdal $DEPLOY_DIR/share/ +rsync -ax $PREFIX/share/proj $DEPLOY_DIR/share/ + # Get Python version PYVERSION=$(cat /root/.pyenv/version) MAJOR=${PYVERSION%%.*} From 1919e1ed551b9117705a08eec50484e7775f815f Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Sat, 17 Oct 2020 00:29:48 -0400 Subject: [PATCH 10/26] update CHANGELOG --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fb93a5..0f65834 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,11 +6,15 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ## [Unreleased] +### +- Added deployments to `eu-west-2` and `eu-north-1` + ### Fixed - README updates regarding versions ### Changed - Keep symlinks when zipping, resulting in smaller deploy package +- Python Lambda Layers no longer need the base GeoLambda layer in addition to the python layer ## [v2.0.0] - 2019-10-25 From 962a3db738b05ff2f714a0942bdc4e56e29508dc Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Mon, 19 Oct 2020 10:19:50 -0400 Subject: [PATCH 11/26] bump python requirements --- python/requirements.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/python/requirements.txt b/python/requirements.txt index 4f9177b..26c3563 100644 --- a/python/requirements.txt +++ b/python/requirements.txt @@ -1,4 +1,4 @@ -GDAL==3.0.1 -rasterio==1.1.0 --no-binary rasterio -shapely==1.6.4.post2 -pyproj==2.4.0 \ No newline at end of file +GDAL==3.1.3 +rasterio==1.1.7 --no-binary rasterio +shapely==1.7.1 +pyproj==2.6.1 From 64f1cc3234ee0674ad4b78d1fc346117980fad2f Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Mon, 19 Oct 2020 10:19:56 -0400 Subject: [PATCH 12/26] bump version --- VERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION b/VERSION index 227cea2..7ec1d6d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.0.0 +2.1.0 From d4b8fb039a27320af062d642600a8b8012553449 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Mon, 19 Oct 2020 10:20:10 -0400 Subject: [PATCH 13/26] bump versions of most packages --- Dockerfile | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/Dockerfile b/Dockerfile index 9426681..673aab7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -13,22 +13,22 @@ RUN \ # versions of packages ENV \ - GDAL_VERSION=3.0.1 \ - PROJ_VERSION=6.2.0 \ - GEOS_VERSION=3.8.0 \ + GDAL_VERSION=3.1.3 \ + PROJ_VERSION=7.1.1 \ + GEOS_VERSION=3.8.1 \ GEOTIFF_VERSION=1.5.1 \ - HDF4_VERSION=4.2.14 \ - HDF5_VERSION=1.10.5 \ - NETCDF_VERSION=4.7.1 \ - NGHTTP2_VERSION=1.39.2 \ + HDF4_VERSION=4.2.15 \ + HDF5_VERSION=1.10.7 \ + NETCDF_VERSION=4.7.4 \ + NGHTTP2_VERSION=1.41.0 \ OPENJPEG_VERSION=2.3.1 \ - CURL_VERSION=7.66.0 \ - LIBJPEG_TURBO_VERSION=2.0.3 \ + LIBJPEG_TURBO_VERSION=2.0.5 \ + CURL_VERSION=7.73.0 \ PKGCONFIG_VERSION=0.29.2 \ SZIP_VERSION=2.1.1 \ - WEBP_VERSION=1.0.3 \ - ZSTD_VERSION=1.4.3 \ - OPENSSL_VERSION=1.0.2 + WEBP_VERSION=1.1.0 \ + ZSTD_VERSION=1.4.5 \ + OPENSSL_VERSION=1.1.1 # Paths to things ENV \ @@ -53,14 +53,25 @@ RUN \ make -j ${NPROC} install; \ cd ../; rm -rf pkg-config +# sqlite3 (required by proj) +RUN \ + mkdir sqlite3; \ + wget -qO- https://www.sqlite.org/2020/sqlite-autoconf-3330000.tar.gz \ + | tar xvz -C sqlite3 --strip-components=1; cd sqlite3; \ + ./configure --prefix=$PREFIX; \ + make; make install; \ + cd ../; rm -rf sqlite3; + # proj RUN \ mkdir proj; \ - wget -qO- http://download.osgeo.org/proj/proj-$PROJ_VERSION.tar.gz | tar xvz -C proj --strip-components=1; cd proj; \ - ./configure --prefix=$PREFIX; \ + wget -qO- http://download.osgeo.org/proj/proj-$PROJ_VERSION.tar.gz \ + | tar xvz -C proj --strip-components=1; cd proj; \ + SQLITE3_LIBS="=L$PREFIX/lib -lsqlite3" SQLITE3_INCLUDE_DIRS=$PREFIX/include/proj ./configure --prefix=$PREFIX; \ make -j ${NPROC} install; \ cd ..; rm -rf proj + # nghttp2 RUN \ mkdir nghttp2; \ From a2df40982ac3b0e6739101a87e5fd7759477f7de Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Mon, 19 Oct 2020 10:20:30 -0400 Subject: [PATCH 14/26] set to beta version --- VERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION b/VERSION index 7ec1d6d..ac83fe0 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.1.0 +2.1.0-b1 From b78b43a9682c84d61ff31ed7d7ce00fb8702e755 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Thu, 14 Jan 2021 18:50:17 -0500 Subject: [PATCH 15/26] bump python to 3.7.9 --- python/Dockerfile | 2 +- python/PYVERSION | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 python/PYVERSION diff --git a/python/Dockerfile b/python/Dockerfile index 63681a2..bea1ed1 100644 --- a/python/Dockerfile +++ b/python/Dockerfile @@ -4,7 +4,7 @@ FROM developmentseed/geolambda:${VERSION} LABEL maintainer="Development Seed " LABEL authors="Matthew Hanson " -ARG PYVERSION=3.7.4 +ARG PYVERSION=3.7.9 # install Python ENV \ diff --git a/python/PYVERSION b/python/PYVERSION new file mode 100644 index 0000000..c77a7de --- /dev/null +++ b/python/PYVERSION @@ -0,0 +1 @@ +3.7.9 From 22b5f4ce910c5101fedc32bcfb897eaff455dcd2 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Thu, 14 Jan 2021 18:50:38 -0500 Subject: [PATCH 16/26] update packaging --- bin/package.sh | 2 +- build-and-test.sh | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/bin/package.sh b/bin/package.sh index d737f0e..19640ed 100755 --- a/bin/package.sh +++ b/bin/package.sh @@ -19,4 +19,4 @@ rsync -ax $PREFIX/share/proj $DEPLOY_DIR/share/ # zip up deploy package cd $DEPLOY_DIR -zip -ruq ../lambda-deploy.zip ./ +zip --symlinks -ruq ../lambda-deploy.zip ./ diff --git a/build-and-test.sh b/build-and-test.sh index 2bb67d0..f99772e 100755 --- a/build-and-test.sh +++ b/build-and-test.sh @@ -1,6 +1,7 @@ #!/bin/bash VERSION=$(cat VERSION) +PYVERSION=$(cat python/PYVERSION) docker build . -t developmentseed/geolambda:${VERSION} docker run --rm -v $PWD:/home/geolambda -it developmentseed/geolambda:${VERSION} package.sh @@ -9,5 +10,5 @@ cd python docker build . --build-arg VERSION=${VERSION} -t developmentseed/geolambda:${VERSION}-python docker run -v ${PWD}:/home/geolambda -t developmentseed/geolambda:${VERSION}-python package-python.sh -docker run -e GDAL_DATA=/opt/share/gdal -e PROJ_LIB=/opt/share/proj \ +docker run -e --build-arg PYVERSION=${PYVERSION} GDAL_DATA=/opt/share/gdal -e PROJ_LIB=/opt/share/proj \ --rm -v ${PWD}/lambda:/var/task -v ${PWD}/../lambda:/opt lambci/lambda:python3.7 lambda_function.lambda_handler '{}' From 5118038a99bed7c674a19aeb8e27176a991f8b67 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Fri, 15 Jan 2021 00:12:59 -0500 Subject: [PATCH 17/26] bump version --- VERSION | 2 +- python/Dockerfile | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/VERSION b/VERSION index ac83fe0..7ec1d6d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.1.0-b1 +2.1.0 diff --git a/python/Dockerfile b/python/Dockerfile index bea1ed1..1e63278 100644 --- a/python/Dockerfile +++ b/python/Dockerfile @@ -1,4 +1,4 @@ -ARG VERSION=2.0.0 +ARG VERSION=2.1.0 FROM developmentseed/geolambda:${VERSION} LABEL maintainer="Development Seed " From 41a1b9758ca48bc077bd2f08edbdfcb394ea9f9a Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Fri, 15 Jan 2021 00:13:25 -0500 Subject: [PATCH 18/26] bump versions of GDAL, Proj, libgeotiff, libopenjpeg, libjpegturbo --- Dockerfile | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Dockerfile b/Dockerfile index 673aab7..2fbb825 100644 --- a/Dockerfile +++ b/Dockerfile @@ -13,16 +13,16 @@ RUN \ # versions of packages ENV \ - GDAL_VERSION=3.1.3 \ - PROJ_VERSION=7.1.1 \ + GDAL_VERSION=3.2.1 \ + PROJ_VERSION=7.2.1 \ GEOS_VERSION=3.8.1 \ - GEOTIFF_VERSION=1.5.1 \ + GEOTIFF_VERSION=1.6.0 \ HDF4_VERSION=4.2.15 \ HDF5_VERSION=1.10.7 \ NETCDF_VERSION=4.7.4 \ NGHTTP2_VERSION=1.41.0 \ - OPENJPEG_VERSION=2.3.1 \ - LIBJPEG_TURBO_VERSION=2.0.5 \ + OPENJPEG_VERSION=2.4.0 \ + LIBJPEG_TURBO_VERSION=2.0.6 \ CURL_VERSION=7.73.0 \ PKGCONFIG_VERSION=0.29.2 \ SZIP_VERSION=2.1.1 \ From b87e66fab3411738cdd9e7a9be514cf413232ca0 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Wed, 3 Feb 2021 23:01:51 -0500 Subject: [PATCH 19/26] use built sqlite3 and disable elastic driver (causing gdal problem) --- Dockerfile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Dockerfile b/Dockerfile index 2fbb825..7811326 100644 --- a/Dockerfile +++ b/Dockerfile @@ -205,9 +205,11 @@ RUN \ --with-zstd=${PREFIX} \ --with-jpeg=${PREFIX} \ --with-threads=yes \ + --with-sqlite3=$PREFIX \ --with-curl=${PREFIX}/bin/curl-config \ --without-python \ --without-libtool \ + --disable-driver-elastic \ --with-geos=$PREFIX/bin/geos-config \ --with-hide-internal-symbols=yes \ CFLAGS="-O2 -Os" CXXFLAGS="-O2 -Os" \ From f1c21f1fc2fc230bcca65cdfe647558b46c10e20 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Thu, 4 Feb 2021 00:19:58 -0500 Subject: [PATCH 20/26] update python requirements --- python/requirements.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/python/requirements.txt b/python/requirements.txt index 26c3563..67ea0fc 100644 --- a/python/requirements.txt +++ b/python/requirements.txt @@ -1,4 +1,4 @@ -GDAL==3.1.3 -rasterio==1.1.7 --no-binary rasterio +GDAL==3.2.1 +rasterio==1.2.0 --no-binary rasterio shapely==1.7.1 -pyproj==2.6.1 +pyproj==3.0.0.post1 From 316745206244058af5a6b007f6a6966b94235661 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Thu, 4 Feb 2021 00:20:06 -0500 Subject: [PATCH 21/26] updated build and test script --- build-and-test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build-and-test.sh b/build-and-test.sh index f99772e..bed52a6 100755 --- a/build-and-test.sh +++ b/build-and-test.sh @@ -10,5 +10,5 @@ cd python docker build . --build-arg VERSION=${VERSION} -t developmentseed/geolambda:${VERSION}-python docker run -v ${PWD}:/home/geolambda -t developmentseed/geolambda:${VERSION}-python package-python.sh -docker run -e --build-arg PYVERSION=${PYVERSION} GDAL_DATA=/opt/share/gdal -e PROJ_LIB=/opt/share/proj \ +docker run -e GDAL_DATA=/opt/share/gdal -e PROJ_LIB=/opt/share/proj \ --rm -v ${PWD}/lambda:/var/task -v ${PWD}/../lambda:/opt lambci/lambda:python3.7 lambda_function.lambda_handler '{}' From 39c9c03e1da5266c6ca25e906dcdb4c77108a834 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Thu, 4 Feb 2021 00:57:29 -0500 Subject: [PATCH 22/26] include geolambda base layer with python package --- build-and-test.sh | 2 +- python/bin/package-python.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/build-and-test.sh b/build-and-test.sh index bed52a6..26ac040 100755 --- a/build-and-test.sh +++ b/build-and-test.sh @@ -11,4 +11,4 @@ docker build . --build-arg VERSION=${VERSION} -t developmentseed/geolambda:${VER docker run -v ${PWD}:/home/geolambda -t developmentseed/geolambda:${VERSION}-python package-python.sh docker run -e GDAL_DATA=/opt/share/gdal -e PROJ_LIB=/opt/share/proj \ - --rm -v ${PWD}/lambda:/var/task -v ${PWD}/../lambda:/opt lambci/lambda:python3.7 lambda_function.lambda_handler '{}' + --rm -v ${PWD}/lambda:/var/task lambci/lambda:python3.7 lambda_function.lambda_handler '{}' diff --git a/python/bin/package-python.sh b/python/bin/package-python.sh index ca4a656..6dd86fc 100755 --- a/python/bin/package-python.sh +++ b/python/bin/package-python.sh @@ -41,7 +41,7 @@ cp -r $DEPLOY_DIR ./python rm ./python/lambda_function.py # zip up deploy package -zip -ruq lambda-deploy.zip ./python +zip --symlinks -ruq lambda-deploy.zip ./python # cleanup rm -rf ./python \ No newline at end of file From e390b4f2f960e618d14051353c0ed5d09a82ba91 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Thu, 4 Feb 2021 01:14:00 -0500 Subject: [PATCH 23/26] fix envvars in docker image --- Dockerfile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index 7811326..17508f2 100644 --- a/Dockerfile +++ b/Dockerfile @@ -37,9 +37,9 @@ ENV \ PREFIX=/usr/local \ GDAL_CONFIG=/usr/local/bin/gdal-config \ LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64 \ - PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig:/usr/lib64/pkgconfig \ - GDAL_DATA=${PREFIX}/share/gdal \ - PROJ_LIB=${PREFIX}/share/proj + PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/lib64/pkgconfig \ + GDAL_DATA=/usr/local/share/gdal \ + PROJ_LIB=/usr/local/share/proj # switch to a build directory WORKDIR /build From dc427a68ea0b6c8641c610a86983d2df550dba65 Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Thu, 4 Feb 2021 01:14:12 -0500 Subject: [PATCH 24/26] update README --- README.md | 24 ++++++++++++++++++++++++ python/README.md | 9 +++++---- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 3d082ad..3c15d06 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ While GeoLambda was initially intended for AWS Lambda it is also useful as a bas | 1.1.0 | 2.4.1 | | | 1.2.0 | 2.4.2 | Separate Python (3.7.4) image and Lambda Layer added | | 2.0.0 | 3.0.1 | libgeotiff 1.5.1, proj 6.2.0 | +| 2.1.0 | 3.2.1 | libgeotiff 1.6.0, proj 7.2.1, openjpeg 2.4.0 | #### Environment variables @@ -28,6 +29,29 @@ When using GeoLambda some environment variables need to be set. These are set in If you just wish to use the publicly available Lambda layers you will need the ARN for the layer in the same region as your Lambda function. Currently, the latest GeoLambda layers are deployed in `us-east-1`, `us-west-2`, `eu-central-1`, `eu-west-2`, and `eu-north-1`. If you want to use it in another region please file an issue or you can also create your own layer using this repository (see instructions below on 'Create a new version'). +#### v2.1.0 + +| Region | ARN | +| ------ | --- | +| us-east-1 | arn:aws:lambda:us-east-1:552188055668:layer:geolambda:4 | +| us-west-2 | arn:aws:lambda:us-west-2:552188055668:layer:geolambda:4 | +| eu-central-1 | arn:aws:lambda:eu-central-1:552188055668:layer:geolambda:4 | +| eu-west-2 | | +| eu-north-1 | | + +#### v2.1.0-python + +See the [GeoLambda Python README](python/README.md). The Python Lambda Layer includes the libraries `numpy`, `rasterio`, `GDAL`, `pyproj`, and `shapely`, plus everything in the standard GeoLambda layer. Note this is a change from v2.0.0 where both Layers needed to be included in a Lambda. + +| Region | ARN | +| ------ | --- | +| us-east-1 | arn:aws:lambda:us-east-1:552188055668:layer:geolambda-python:3 | +| us-west-2 | arn:aws:lambda:us-west-2:552188055668:layer:geolambda-python:3 | +| eu-central-1 | arn:aws:lambda:eu-central-1:552188055668:layer:geolambda-python:3 | +| eu-west-2 | | +| eu-north-1 | | + + #### v2.0.0 | Region | ARN | diff --git a/python/README.md b/python/README.md index 30a1da6..f7028b6 100644 --- a/python/README.md +++ b/python/README.md @@ -19,11 +19,11 @@ An example Lambda handler is located at [lambda/lambda_function.py](lambda/lambd Now, use the [Dockerfile](Dockerfile) can be used to create a new Docker image based on any version of GeoLambda with any version of Python by providing the versions as build arguments to `docker run`. This will install the specified version of Python along with any Python packages provided in [requirements.txt](requirements.txt). ``` -$ VERSION=2.0.0 -$ docker build . --build-arg VERSION=${VERSION} --build-arg PYVERSION=3.7.4 -t :latest +$ VERSION=2.1.0 +$ docker build . --build-arg VERSION=${VERSION} --build-arg PYVERSION=3.7.9 -t :latest ``` -If not provided, `VERSION` (the version of GeoLambda to use) will default to `latest` and `PYVERSION` (Python version) will default to `3.7.4`. +If not provided, `VERSION` (the version of GeoLambda to use) will default to `latest` and `PYVERSION` (Python version) will default to `3.7.9`. **4. Set up development environment and lambda layer zip file** @@ -55,7 +55,8 @@ You can use the [LambCI Docker images](https://github.com/lambci/docker-lambda) ``` # current dir: geolambda/python -$ docker run --rm -v ${PWD}/lambda:/var/task -v ${PWD}/../lambda:/opt lambci/lambda:python3.7 lambda_function.lambda_handler '{}' +$ docker run -e GDAL_DATA=/opt/share/gdal -e PROJ_LIB=/opt/share/proj \ + --rm -v ${PWD}/lambda:/var/task lambci/lambda:python3.7 lambda_function.lambda_handler '{}' ``` The last argument is a JSON string that will be passed as the event payload to the handler function. From 696c1b18640637d7193c99a078d271796aa6462c Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Thu, 4 Feb 2021 01:14:34 -0500 Subject: [PATCH 25/26] update CHANGELOG --- CHANGELOG.md | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f65834..2f437fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ## [Unreleased] -### +### Added - Added deployments to `eu-west-2` and `eu-north-1` ### Fixed @@ -15,6 +15,29 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ### Changed - Keep symlinks when zipping, resulting in smaller deploy package - Python Lambda Layers no longer need the base GeoLambda layer in addition to the python layer +- Manually compile sqlite3 used in Proj, GDAL, nghttp2 +- GDAL 3.2.1 +- Proj 7.2.1 +- GEOS 3.8.1 +- GeoTIFF 1.5.1 +- HDF4 4.2.15 +- HDF5 1.10.7 +- NetCDF 4.7.4 +- Nghttp2 1.41.0 +- OpenJPEG 2.4.0 +- libJPEG Turbo 2.4.0 +- Curl 7.73.0 +- Webp 1.1.0 +- Zstd 1.4.5 +- OpenSSL 1.1.1 +- In python layer: + - Python 3.7.4 -> 3.7.9 + - rasterio 1.1.0 -> 1.2.0 + - shapely 1.6.4.post2 -> 1.7.1 + - pyproj 2.4.0 -> 3.0.0.post1 + +### Removed +- GDAL Elasticsearch driver ## [v2.0.0] - 2019-10-25 @@ -95,6 +118,7 @@ Package Versions - GDAL_VERSION=2.3.1 [Unreleased]: https://github.com/sat-utils/sat-stac/compare/master...develop +[v2.1.0]: https://github.com/developmentseed/geolambda/compare/2.0.0...2.1.0 [v2.0.0]: https://github.com/developmentseed/geolambda/compare/1.2.0...2.0.0 [v1.2.0]: https://github.com/developmentseed/geolambda/compare/1.1.0...1.2.0 [v1.1.0]: https://github.com/developmentseed/geolambda/compare/1.0.0...1.1.0 From a5a5102568ea26f2cb5067b406d684fccce8273d Mon Sep 17 00:00:00 2001 From: Matthew Hanson Date: Thu, 4 Feb 2021 02:02:18 -0500 Subject: [PATCH 26/26] README updates --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3c15d06..965f0fb 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # GeoLambda: geospatial AWS Lambda Layer -The GeoLambda project provides public Docker images and AWS Lambda Layers containing common geospatial native libraries. GeoLambda contains the libraries PROJ.5, GEOS, GeoTIFF, HDF4/5, SZIP, NetCDF, OpenJPEG, WEBP, ZSTD, and GDAL. For some applications you may wish to minimize the size of the libraries by excluding unused libraries, or you may wish to add other libraries. In this case this repository can be used as a template to create your own Docker image or Lambda Layer following the instructions in this README. +The GeoLambda project provides public Docker images and AWS Lambda Layers containing common geospatial native libraries. GeoLambda contains the libraries for GDAL, Proj, GEOS, GeoTIFF, HDF4/5, SZIP, NetCDF, OpenJPEG, WEBP, ZSTD, and others. For some applications you may wish to minimize the size of the libraries by excluding unused libraries, or you may wish to add other libraries. In this case this repository can be used as a template to create your own Docker image or Lambda Layer following the instructions in this README. This repository also contains additional images and layers for specific runtimes. Using them as a Layer assumes the use of the base GeoLambda layer. @@ -16,7 +16,7 @@ While GeoLambda was initially intended for AWS Lambda it is also useful as a bas | 1.1.0 | 2.4.1 | | | 1.2.0 | 2.4.2 | Separate Python (3.7.4) image and Lambda Layer added | | 2.0.0 | 3.0.1 | libgeotiff 1.5.1, proj 6.2.0 | -| 2.1.0 | 3.2.1 | libgeotiff 1.6.0, proj 7.2.1, openjpeg 2.4.0 | +| 2.1.0 | 3.2.1 | libgeotiff 1.6.0, proj 7.2.1, openjpeg 2.4.0, Python layer 3.7.9 | #### Environment variables