Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
47a75c9
working prototype with alb and api gateway
thisismana Dec 7, 2020
519ddd0
bumped sharp
thisismana Dec 9, 2020
5b0c94e
added etag, test invoke
thisismana Jan 8, 2021
ca4349e
updated terraform
thisismana Jan 11, 2021
97dce13
migrated from zip to docker deployment
thisismana Jan 14, 2021
f74ce69
github pr and deploy workflows
thisismana Jan 14, 2021
6af0440
deleted files not needed for our use cases
thisismana Jan 14, 2021
db1d79d
fixed tests
thisismana Jan 15, 2021
1789540
documentation
thisismana Jan 15, 2021
16acfa1
implemented http/410 gone for expired content
thisismana Jan 15, 2021
ede30d5
arg not supported on GH
thisismana Jan 15, 2021
bb9007c
reworked object expiry
thisismana Jan 18, 2021
4c43af0
fixed deployment-off-by-one issue by adding a waiter
thisismana Jan 18, 2021
002d30d
fixed expiry
thisismana Jan 18, 2021
794728a
expiry in seconds, not milliseconds
thisismana Jan 18, 2021
c669d1b
use deployment module (#1)
moritzzimmer Jan 21, 2021
2be18fb
feature / JSON logging (#10)
vmk1vmk Feb 11, 2021
76b0cbf
bumped gh action runs-on to latest ubuntu,
thisismana Feb 11, 2021
2ca8d9e
fixed docker build
thisismana Feb 11, 2021
43596e0
feature - log format (#13)
vmk1vmk Feb 16, 2021
319db14
fixed docker build
thisismana Feb 16, 2021
c01605b
Copy everything in Dockerfile with exception of files matching the .d…
vmk1vmk Feb 16, 2021
477bcce
use latest module version
moritzzimmer Feb 18, 2021
3975357
logging adaptions
thisismana Feb 18, 2021
2fc931c
move path to top level when logging
thisismana Feb 18, 2021
8dc31c4
more field excludes for logging
thisismana Feb 18, 2021
a5c3af1
test CloudWatch Lambda Insights
moritzzimmer Mar 15, 2021
7729b83
apply fixed deployment module
moritzzimmer Apr 1, 2021
c2f1dda
Lambda Insights extension
moritzzimmer Apr 14, 2021
0b276b7
new file system layout (#28)
thisismana Apr 19, 2021
2aae43b
V2 (#31)
thisismana May 6, 2021
5b81b39
updated image host
thisismana Oct 13, 2021
53cbdde
migrated back to zip deployment.
thisismana Oct 29, 2021
49b0ef6
added check for out-of-bounds
thisismana Oct 29, 2021
1501965
re-enabled logging
thisismana Nov 2, 2021
4855fc0
allow gif
thisismana Nov 15, 2021
f05cc6a
chore: use arm64 architecture
moritzzimmer Nov 24, 2021
f3a3605
chore: use arm64 insights layer
moritzzimmer Nov 26, 2021
e6393ed
migrated to oidc
thisismana Dec 9, 2021
caa5dc1
log level adaptions
thisismana Dec 9, 2021
d518178
fixed test
thisismana Dec 10, 2021
9119a11
chore: removed unused docker_tag var
moritzzimmer Dec 22, 2021
8548a2a
Update README.md
moritzzimmer Dec 22, 2021
21c7f9b
fix: run un x86_64 arch
moritzzimmer Dec 22, 2021
17695bb
tweaked logging even a bit more
thisismana Dec 22, 2021
412e63b
Respect custom status code in metadata (#92)
saefty Jan 3, 2022
daee23a
allow newbiz to write images into our bucket (#98)
thisismana Feb 1, 2022
287efe5
allow `GET` and `HEAD` @ ApiGW
thisismana Apr 29, 2022
1f7fb9b
refactor: use new log streaming lambda
moritzzimmer Apr 29, 2022
894e8c3
fixed regex to allow `\d+x\d+` in path (#131)
thisismana May 4, 2022
349a1b9
shorter cache times for 4xx (#135)
thisismana May 12, 2022
1eea5c4
hotfix for image requests without filename
thisismana Jun 3, 2022
d5166ac
replace `__WIDTH__` placeholder with default of `1800` pixels
thisismana Jun 10, 2022
5e35a26
feat: add user-agent to logs (#142)
moritzzimmer Jun 13, 2022
832dce6
convert `code` to string to not mess up the automated mapping
thisismana Jul 13, 2022
7f8b2a3
feat: First attempt to allow avif (BUZZ-2717) (#157)
TimSusa Aug 11, 2022
8a6bd2b
preparations for lambda-URLs
thisismana Aug 11, 2022
127e86f
fixing avif edge cases
thisismana Aug 11, 2022
4645a6a
- terraform function url
thisismana Aug 12, 2022
e751c41
added `immutable` to all `http/200` responses `Cache-Control` directive
thisismana Aug 12, 2022
fcc90b8
removed api gw
thisismana Aug 30, 2022
f6414a6
properly log `path` for events v1 && v2
thisismana Sep 14, 2022
4bd75fb
Improve `roundCrop` feature (#172)
thisismana Oct 11, 2022
05575b2
updated usage
thisismana Oct 11, 2022
29dbb65
chore: pre-commit for static terraform checks (#195)
moritzzimmer Dec 22, 2022
3367964
chore(lambda): updated terraform module to latest
moritzzimmer Jan 2, 2023
e896b8e
chore: disabled lambda insights (#219)
moritzzimmer Feb 20, 2023
8d61ca7
bugfix: handle zero cropping (#223)
thisismana Feb 28, 2023
fc4709d
feat: use new OpenSearch logging extension (#220)
thisismana Mar 14, 2023
0085c2c
chore: fixed logging permissions
thisismana Mar 15, 2023
8636998
feat: ignore `/authors/` prefix
thisismana Jul 14, 2023
c9a5881
chore: migrate from OAI to OAC
thisismana Jul 14, 2023
8d3d873
fix: added missing iam permissions for detecting http/404
thisismana Jul 31, 2023
85edd24
bugfix: char encoding (newsletter) and guard for non-matching regex
thisismana Jul 31, 2023
1bb46ac
fixed test
thisismana Jul 31, 2023
c612b87
added aws sdk to package
thisismana Jul 31, 2023
6506f5e
revert to node 14
thisismana Jul 31, 2023
365f19f
Revert "revert to node 14"
thisismana Jul 31, 2023
f109c47
thumb hash — initial check in (#234)
thisismana Aug 9, 2023
d708750
fixed Makefile/upload task
thisismana Aug 9, 2023
7b12b2a
fixed Makefile/upload task
thisismana Aug 9, 2023
fbc299a
fixed Makefile/upload task
thisismana Aug 9, 2023
6c41342
fixed Makefile/upload task
thisismana Aug 9, 2023
51713b6
feat: tweak image output settings
thisismana Aug 9, 2023
e5583dd
chore: bumped thumbs stack (#272)
moritzzimmer Sep 27, 2023
12ef7a8
chore: bump all the things (#279)
thisismana Oct 5, 2023
29f1375
[skip ci] enable bucket versioning
saefty Nov 24, 2023
fa12fd4
[skip ci] expire object delete markers
saefty Nov 24, 2023
83b0185
[Buzz 4210] migrate to typescript and update readme (#328)
Kev-Hoffmann Dec 13, 2023
d1dc32b
bugfix: remove body from logs
thisismana Dec 13, 2023
ac1c82b
cleanup/typings (#329)
thisismana Dec 18, 2023
0c22998
chore: bumped Lambda module (#340)
moritzzimmer Feb 1, 2024
d008f56
fix(gh): fixed deployment
moritzzimmer Feb 1, 2024
e3035bc
chore: deleted unused tf lock file
moritzzimmer Feb 1, 2024
dd52954
feat: modified robots.txt
thisismana Feb 29, 2024
d72261a
feat: modified robots.txt
thisismana Feb 29, 2024
fdf557e
feat: added `Vary: Accept` to all `http/200` responses (#408)
thisismana Jun 25, 2024
fd8cb72
feat(logging): log `accept` and `request_id` (#409)
thisismana Jun 26, 2024
52019c9
debugging headers
thisismana Jun 26, 2024
3e9a1fa
enable aws backup for master images
saefty Jul 12, 2024
a28473f
Image handler v2 (#411)
thisismana Jul 15, 2024
a008020
hotfix: stroeer seo image url de-suggering
thisismana Jul 15, 2024
de55039
feat(thumbhash): rewrote thumbhash with native nodejs/ts implementation
thisismana Jul 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[Buzz 4210] migrate to typescript and update readme (#328)
* migrate image-handler from javascript to typescript and update Readme

* feat: variant deployment

* feat: prod deployment

* fixed test

* fix: build command

* bump lambda to nodejs 20 runtime

---------

Co-authored-by: Matthias Naber <[email protected]>
  • Loading branch information
Kev-Hoffmann and thisismana authored Dec 13, 2023
commit 83b01854c7fdbda403fdd303d86bd21d134f733b
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ clean ::
@cd $(WORK_DIR) && rm -rf ./dist/ ./node_modules/

npm/install ::
cd $(WORK_DIR) && npm install
cd $(WORK_DIR) && npm install --cpu=arm64 --os=linux --libc=musl

npm/test ::
cd $(WORK_DIR) && npm run test
Expand All @@ -36,7 +36,7 @@ build ::
tf ::
rm -f $(WORK_DIR)/terraform/.terraform/terraform.tfstate || true
terraform -chdir=$(WORK_DIR)/terraform init $(TF_VARS) -reconfigure -upgrade=$(DO_TF_UPGRADE) $(TF_BACKEND_CFG)
if [ "true" == "$(DO_TF_UPGRADE)" ]; then terraform -chdir=$(WORK_DIR)/terraform providers lock -platform=darwin_amd64 -platform=linux_amd64; fi
if [ "true" == "$(DO_TF_UPGRADE)" ]; then terraform -chdir=$(WORK_DIR)/terraform providers lock -platform=darwin_amd64 -platform=darwin_arm64 -platform=linux_amd64; fi
terraform -chdir=$(WORK_DIR)/terraform $(MODE) $(TF_VARS)

invoke :: # invoke the running docker lambda by posting a sample API-GW-Event
Expand Down
172 changes: 98 additions & 74 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,92 +1,116 @@
**_Important Notice:_**
Due to a [change in the AWS Lambda execution environment](https://aws.amazon.com/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/), Serverless Image Handler v3 deployments are functionally broken. To address the issue we have released [minor version update v3.1.1](https://solutions-reference.s3.amazonaws.com/serverless-image-handler/v3.1.1/serverless-image-handler.template). We recommend all users of v3 to run cloudformation stack update with v3.1.1. Additionally, we suggest you to look at v5 of the solution and migrate to v5 if it addresses all of your use cases.
# Serverless Image Handler Readme

The serverless-image-handler mono repository contains the source code and documentation for the [image-handler](#image-handler) and [image-thumbs](#image-thumbs) AWS Lambda functions.

## Table of Contents

- [Serverless Image Handler Readme](#serverless-image-handler-readme)
- [Table of Contents](#table-of-contents)
- [Architecture](#architecture)
- [Image Handler](#image-handler)
- [Overview](#overview)
- [Prerequisites](#prerequisites)
- [Usage](#usage)
- [Building](#building)
- [Testing](#testing)
- [Infrastructure deployment](#infrastructure-deployment)
- [Image Thumbs](#image-thumbs)
- [Overview](#overview-1)
- [Prerequisites](#prerequisites-1)
- [Building](#building-1)
- [Test / Invoke](#test--invoke)
- [Infrastructure deployment](#infrastructure-deployment-1)

### Architecture

# AWS Serverless Image Handler Lambda wrapper for SharpJS
A solution to dynamically handle images on the fly, utilizing Sharp (https://sharp.pixelplumbing.com/en/stable/).
Published version, additional details and documentation are available here: https://aws.amazon.com/solutions/serverless-image-handler/
![Architecture](architecture.png)

_Note:_ it is recommended to build the application binary on Amazon Linux.
## Image Handler

## On This Page
- [Architecture Overview](#architecture-overview)
- [Creating a custom build](#creating-a-custom-build)
- [External Contributors](#external-contributors)
- [License](#license)
### Overview

## Architecture Overview
![Architecture](architecture.png)
The Image-handler is a solution Image Handler is a serverless image processing project built with Node.js 18 and the [Sharp](https://sharp.pixelplumbing.com/en/stable/) library.
It allows you to dynamically scale images on the fly and serves them through AWS Lambda, AWS CloudFront, and AWS S3.

The AWS CloudFormation template deploys an Amazon CloudFront distribution, Amazon API Gateway REST API, and an AWS Lambda function. Amazon CloudFront provides a caching layer to reduce the cost of image processing and the latency of subsequent image delivery. The Amazon API Gateway provides endpoint resources and triggers the AWS Lambda function. The AWS Lambda function retrieves the image from the customer's Amazon Simple Storage Service (Amazon S3) bucket and uses Sharp to return a modified version of the image to the API Gateway. Additionally, the solution generates a CloudFront domain name that provides cached access to the image handler API.
It aws originally forked from [aws-solutions/serverless-image-handler repository](https://github.com/aws-solutions/serverless-image-handler), but has been heavily modified to suit our needs.

_**Note**:_ From v5.0, all AWS CloudFormation template resources are created be [AWS CDK](https://aws.amazon.com/cdk/) and [AWS Solutions Constructs](https://aws.amazon.com/solutions/constructs/). Since the AWS CloudFormation template resources have the same logical ID comparing to v4.x, it makes the solution upgradable mostly from v4.x to v5.
### Prerequisites

## Creating a custom build
The solution can be deployed through the CloudFormation template available on the solution home page.
To make changes to the solution, download or clone this repo, update the source code and then run the deployment/build-s3-dist.sh script to deploy the updated Lambda code to an Amazon S3 bucket in your account.
- [Node.js](https://nodejs.org/en/) v18.x or later
- [Terraform](https://www.terraform.io/downloads.html)
- Make
- AWS credentials with sufficient permissions

### Prerequisites:
* [AWS Command Line Interface](https://aws.amazon.com/cli/)
* Node.js 12.x or later
### Usage

### 1. Clone the repository
```bash
git clone https://github.com/awslabs/serverless-image-handler.git
```
To scale images dynamically on the fly, you can make HTTP requests to the AWS CloudFront distribution URL.
The images are fetched from AWS S3, processed using Node.js 18 and Sharp, and then served through CloudFront.

### 2. Run unit tests for customization
Run unit tests to make sure added customization passes the tests:
```bash
cd ./deployment
chmod +x ./run-unit-tests.sh
./run-unit-tests.sh
```
Example URL:

### 3. Declare environment variables
```bash
export REGION=aws-region-code # the AWS region to launch the solution (e.g. us-east-1)
export DIST_OUTPUT_BUCKET=my-bucket-name # bucket where customized code will reside
export SOLUTION_NAME=my-solution-name # the solution name
export VERSION=my-version # version number for the customized code
```
```https://images.t-online.de/4k_hdr.jpg```

### 4. Create an Amazon S3 Bucket
The CloudFormation template is configured to pull the Lambda deployment packages from Amazon S3 bucket in the region the template is being launched in. Create a bucket in the desired region with the region name appended to the name of the bucket.
```bash
aws s3 mb s3://$DIST_OUTPUT_BUCKET-$REGION --region $REGION
```
For more details see the [Usage](docs/Usage.md) documentation.

### 5. Create the deployment packages
Build the distributable:
```bash
chmod +x ./build-s3-dist.sh
./build-s3-dist.sh $DIST_OUTPUT_BUCKET $SOLUTION_NAME $VERSION
```
### Environment variables

The following environment variables are used by the image-handler:

| Name | Description |
|---------------------------|-------------------------------------------------|
| `AUTO_WEBP` | Flag if the AUTO WEBP feature should be enabled |
| `CORS_ENABLED` | Flag if CORS should be enabled |
| `CORS_ORIGIN` | CORS origin. |
| `LOG_EXT_OPEN_SEARCH_URL` | URL of OpenSearch |
| `SOURCE_BUCKETS` | S3 Bucket with source images |

### Building

To build the package run:

```make build```

### Testing

Run tests using the following Make command:

```make npm/test```

### Infrastructure deployment

Deploy the infrastructure using Terraform with the following Make command:

```make tf```

## Image Thumbs

### Overview

Image-thumbs is an implementation of [thumbhash](https://evanw.github.io/thumbhash/) as a Rust AWS Lambda function URL providing a very compact representation of a placeholder for an image.

### Prerequisites

- [Rust](https://www.rust-lang.org/tools/install)

### Building

To build the package run:

```make SERVICE=thumbs build```

### Test / Invoke

See [cargo lambda](https://www.cargo-lambda.info/guide/getting-started.html)

Deploy the distributable to the Amazon S3 bucket in your account:
```bash
aws s3 sync ./regional-s3-assets/ s3://$DIST_OUTPUT_BUCKET-$REGION/$SOLUTION_NAME/$VERSION/ --recursive --acl bucket-owner-full-control
aws s3 sync ./global-s3-assets/ s3://$DIST_OUTPUT_BUCKET-$REGION/$SOLUTION_NAME/$VERSION/ --recursive --acl bucket-owner-full-control
# Terminal 1
cargo lambda watch
# Terminal 2
cargo lambda invoke --data-file apigw-request.json
```

### 6. Launch the CloudFormation template.
* Get the link of the `serverless-image-handler.template` uploaded to your Amazon S3 bucket.
* Deploy the Serverless Image Handler solution to your account by launching a new AWS CloudFormation stack using the S3 link of the `serverless-image-handler.template`.

## External Contributors
- [@leviwilson](https://github.com/leviwilson) for [#117](https://github.com/awslabs/serverless-image-handler/pull/117)
- [@rpong](https://github.com/rpong) for [#130](https://github.com/awslabs/serverless-image-handler/pull/130)
- [@harriswong](https://github.com/harriswong) for [#138](https://github.com/awslabs/serverless-image-handler/pull/138)
- [@ganey](https://github.com/ganey) for [#139](https://github.com/awslabs/serverless-image-handler/pull/139)
- [@browniebroke](https://github.com/browniebroke) for [#151](https://github.com/awslabs/serverless-image-handler/pull/151), [#152](https://github.com/awslabs/serverless-image-handler/pull/152)
- [@john-shaffer](https://github.com/john-shaffer) for [#158](https://github.com/awslabs/serverless-image-handler/pull/158)
- [@toredash](https://github.com/toredash) for [#174](https://github.com/awslabs/serverless-image-handler/pull/174), [#195](https://github.com/awslabs/serverless-image-handler/pull/195)
- [@lith-imad](https://github.com/lith-imad) for [#194](https://github.com/awslabs/serverless-image-handler/pull/194)
- [@pch](https://github.com/pch) for [#227](https://github.com/awslabs/serverless-image-handler/pull/227)
- [@atrope](https://github.com/atrope) for [#201](https://github.com/awslabs/serverless-image-handler/pull/201)
- [@bretto36](https://github.com/bretto36) for [#182](https://github.com/awslabs/serverless-image-handler/pull/182)

***
## License
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.<br />
SPDX-License-Identifier: Apache-2.0
## Infrastructure deployment

To deploy the infrastructure run:

```make SERVICE=thumb tf```
Binary file modified architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
41 changes: 7 additions & 34 deletions docs/Usage.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,7 @@
# Stroeer serverless image handler
# Usage

This is a customized variant of the [AWS Serverless Image Handler].
Its **code** is open source and can be found by following the _forked from_
link above or otherwise [on GitHub][AWS Serverless Image Handler Repository].

Under the hood it uses the [sharp image processing library][sharp].

It is recommended to have a look at the [output options][output options],
[resize operations][resize operations] and [image operations][image operations].

## Architecture

![architecture](arch.png)

* `pub` and `sub` lambda are described [here][peruggia]. Their task is to publish
the master images into the S3 bucket
* As for the image delivery, we're using CloudFront CDN as an entrypoint
* Its origin is an API Gateway
* Which then calls an AWS Lambda who is doing the actual work (image serving and manipulation).
It is recommended to have a look at the [output options](https://sharp.pixelplumbing.com/api-output),
[resize operations](https://sharp.pixelplumbing.com/api-resize) and [image operations](https://sharp.pixelplumbing.com/api-operation).

### Feature: AUTO WEBP

Expand Down Expand Up @@ -97,9 +81,9 @@ $ curl -v 'https://images.t-online.de/4k_hdr.jpg' \

## Filters

Most of the filter are documented on the [AWS Solution page][Serverless Image Handler Filters].
Most of the filter are documented on the [AWS Solution page](https://docs.aws.amazon.com/solutions/latest/serverless-image-handler/appendix-d.html).

Here is a tl;dr for the most important ones, demonstrated on an image from [the internet][img source]:
Here is a tl;dr for the most important ones, demonstrated on an image from [the internet](https://wallpapersafari.com/w/pEwDaY):
### #nofilter

will simply output the original image as is
Expand Down Expand Up @@ -187,20 +171,9 @@ black frame.

## Try it out

Visit the [Demo UI], enter the following
Visit the [Demo UI](https://master-images-053041861227-eu-west-1.s3-eu-west-1.amazonaws.com/index.html), enter the following

* `bucket name = master-images-053041861227-eu-west-1`
* `image key = oat.jpg`

There are many other images already automatically imported into the image bucket, just check the API for more `keys`.

[AWS Serverless Image Handler]: https://docs.aws.amazon.com/solutions/latest/serverless-image-handler/welcome.html
[AWS Serverless Image Handler Repository]: https://github.com/awslabs/serverless-image-handler
[sharp]: https://sharp.pixelplumbing.com/
[output options]: https://sharp.pixelplumbing.com/api-output
[resize operations]: https://sharp.pixelplumbing.com/api-resize
[image operations]: https://sharp.pixelplumbing.com/api-operation
[Serverless Image Handler Filters]: https://docs.aws.amazon.com/solutions/latest/serverless-image-handler/appendix-d.html
[Demo UI]: https://master-images-053041861227-eu-west-1.s3-eu-west-1.amazonaws.com/index.html
[img source]: https://wallpapersafari.com/w/pEwDaY
[peruggia]: github.com/stroeer/peruggia
There are many other images already automatically imported into the image bucket, just check the API for more `keys`.
1 change: 0 additions & 1 deletion docs/arch.drawio

This file was deleted.

Binary file removed docs/arch.png
Binary file not shown.
20 changes: 0 additions & 20 deletions source/image-handler/logger/index.js

This file was deleted.

20 changes: 0 additions & 20 deletions source/image-handler/logger/logger.js

This file was deleted.

63 changes: 0 additions & 63 deletions source/image-handler/logger/logger.spec.js

This file was deleted.

Loading