How to update Thumbor and Python Pillow

[onassar]

Thanks for all your work on this @stevemorad
Really appreciate it.

Unfort, it seems like there are some issues in Thumbor / Python Pillow that prevent this solution from being 100% ready for a production environment. Specifically, it doesn't work properly for a number of PNGs that have transparent backgrounds (which is quite a common use-case).

Here is the ticket I opened with Thumbor:
thumbor/thumbor#987

My question is: when Thumbor and Pillow are fixed/patched, what's the best process to update the Lambda function? Can I simply download Thumbor/Pillow exactly how they're released, download the Lambda package, replace the existing folders with the updated releases, and then re-upload?

[stevemorad]

Please post back when this issue is resolved. When it's fixed, your approach will work. We will also look to rebuild the published version to include updated Thumbor/Pillow after the issue is fixed.

[onassar]

Okay thanks I’ll check in with Thunbor

[stevemorad]

Version 2.2 now includes Thumbor 6.4.2 and Pillow 4.3.0. I believe this issue is now resolved. Would you confirm and close this issue if so?

[onassar]

Sure I'll test this after the easter-weekend and reply back here.

[onassar]

Just ran into this while testing the new template:
00:32:15 UTC+0100 CREATE_FAILED AWS::Lambda::Function CustomResource 1 validation error detected: Value '%%BUCKET_NAME%%-us-west-2' at 'code.s3Bucket' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[0-9A-Za-z\.\-_]*(?<!\.)$

I ran into something a while back; my bucket is named (something like):
app.local.domain.com.s3.amazonaws.com

Based on the log, it looks like the Lambda function doesn't like the naming convention of my S3 bucket. I ran into something like this a while back (listed below), whereby the OriginS3Bucket value was being rejected for similar things.

But in this case, it seems it's Lambda rejecting it.

--

This is the exchange I had with an AWS rep back in September:

Hi,

Thank you for contacting AWS Support. My name is J from Deployment team (Sydney/AU) and I'll be glad to assist with your query today.

To recap our call, you have engaged AWS Support to check the AWS solution template [1] since you were getting the template validation error message "Parameter OriginS3Bucket failed to satisfy the constraint: Must be a valid S3 bucket"

We found together that the parameter "OriginS3Bucket" having the "AllowedPattern" with value "[a-zA-Z][a-zA-Z0-9-]*" and not allowing the period but your S3 bucket name 'app.local.domain.com' having the period in its name.

The CFN deployment started successfully after removing the constraints "AllowedPattern" from the parameter "OriginS3Bucket" and now I noticed that your CFN stack 'ServerlessImageHandler' [2] deployment was successfully completed.

[1] http://docs.aws.amazon.com/solutions/latest/serverless-image-handler/deployment.html
[2] arn:aws:cloudformation:us-west-2:003016793786:stack/ServerlessImageHandler/4ae8d2f0-9da0-11e7-8325-500c33711099 

As I committed, I have created an internal feedback request to the respective team to review this issue and make necessary changes in solution template to accept S3 bucket with periods. This may take a bit of more time and currently, there is no ETA for when this feature will be available.

I am resolving this case since your CFN Stack [2] deployment was completed. If you have any questions or concerns, please do not hesitate to contact us by updating case or call us through support center or start a chat session and we'll gladly assist further. 

${print-hmd-poll}

Best regards,

J P S.
Amazon Web Services

Case ID4430157701

I hope this helps!

[stevemorad]

Did you test the new template directly from GitHub without running it through the build process? The bucket name variable does not appear to be configured properly (%%BUCKET_NAME%%-us-west-2).

If you want to test a precompiled version, please use the published template:
https://s3.amazonaws.com/solutions-reference/serverless-image-handler/latest/serverless-image-handler.template

Or make sure you follow the build process and use the build-s3-dist.sh file in the deployment directory to build the zip files using your own distribution bucket (e.g. ./build-s3-dist.sh mybucketname)

[stevemorad]

Also, I tested with sih.origin as the origin bucket and did not have any problems with the constraint or lambda functions using a bucket with a dot (.) in the name.

[onassar]

Hmm thanks didn't realize that was required. I'm looking at the one you just linked to, and see (in lines 106, 116 and 128) the string values solutions. My experience with these templates was that I didn't need to hardcode the bucket name directly into the template file. Rather, I'd be prompted when uploading to CloudFormation. Should I change the string values on those lines to be my respective bucket?

[stevemorad]

If you want to deploy our prebuilt version of the solution, then you do not need to modify the template above. Just configure CloudFormation to launch the template, and provide your image bucket as an input parameter when uploading to CloudFormation. Alternatively, use this link and configure your AWS region to the region you want to launch the solution:
https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=ServerlessImageHandler&templateURL=https:%2F%2Fs3.amazonaws.com%2Fsolutions-reference%2Fserverless-image-handler%2Flatest%2Fserverless-image-handler.template

More information about launching the prebuilt version is available here:
https://docs.aws.amazon.com/solutions/latest/serverless-image-handler/welcome.html

You only need to replace "solutions" in the template when you want to deploy your own/modified version of the solution. The solution has 3 parts, so you only need to modify the template to point to the specific part(s) you modified:

1. The main function that implements thumbor is called serverless-image-handler.zip and line Is possible use same url after upgrade? #106 tells the template what bucket you have put your modified version of this lambda function zip file. Line All output images are blurred after resizing #258 will actually take this bucket name and add "-region" to the end of it. We do this because CloudFormation requires the bucket with your lambda code be in the same region as you are launching the template. We stage copies of the lambda files in every region to support this requirement. So, the prebuilt zip files are located in solutions-region buckets (e.g. solutions-us-east-1).
2. The solution has a CloudFormation custom resource called serverless-image-handler-custom-resource.zip that helps with automating the solution deployment. For example, it deploys the demo website to your S3 bucket if requested. Line outputFormat option not honored #116 tells the template the bucket prefix for where this custom resource.
3. The demo website is zipped in serverless-image-handler-ui.zip and the bucket prefix is configured with line libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (nss) #128

We offer a prebuilt version so you don't have to worry about any of this if you just want to launch the solution. If you want to modify it, we provide the code and build scripts here on GitHub so you typically don't have to worry about modifying specific lines in the template (but that also means you will need to put the three zip files mentioned above into your own distribution bucket).

Hope this helps.

[onassar]

Thanks for all the details @stevemorad
I really appreciate it.

I was able to get the stack up and running, but wanted to document a couple things here for anyone who stumbles on this (along with a concern around this being production-ready; albeit it may be an issue with Pillow/Thumbor and not this repo).

I needed to change the stack region.
From this:
https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=ServerlessImageHandler&templateURL=https:%2F%2Fs3.amazonaws.com%2Fsolutions-reference%2Fserverless-image-handler%2Flatest%2Fserverless-image-handler.template

To this:
https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=ServerlessImageHandler&templateURL=https:%2F%2Fs3.amazonaws.com%2Fsolutions-reference%2Fserverless-image-handler%2Flatest%2Fserverless-image-handler.template

Notice the region value; otherwise it was creating a stack in the wrong region for me.

---

When defining the value for my bucket, I initially entered the wrong value.
I entered it as: app.local.domain.com.s3.amazonaws.com
The correct value was app.local.domain.com, since that's the actual bucket name. I was incorrectly appending .s3.amazonaws.com

Heads up in case anyone else does this by accident.

---

Finally: this version of the template is definitely an improvement.
I noticed:

• An increase in the quality of the thumbs produced (old version / new version); if you zoom in in your browser, you'll see a substantial improvement in the quality
• Some png/gif cases that errored out are now able to be served: ( before #1 / after #1, before #2 / after #2, before #3 / after #3)

The concerns around this being production ready:
In before/after cases number 2 and number 3 above, the resulting resized thumb isn't valid. What I mean by that is some or all of the background-transparent pixels are being converted to black. I'm going to list out below the two cases again, along with the original file, and what the resulting thumb should look like (which I currently use Cloudinary for conversion, which does it "correctly"):

Case number 2:
Original file: http://s3-us-west-2.amazonaws.com/app.local.getstencil.com/upld564d2305
Resized thumb using this version of the template: https://d3jqyot5ky4oh9.cloudfront.net/full-fit-in/304x212/filters:quality(100)/upld564d2305
What it should look like (served by Cloudinary): https://dx5683gi1tv0w.cloudfront.net/dtrjyhj9q/image/upload/if_ar_gt_1.0,h_212,c_scale/if_else,w_304,c_scale/s3.local/upld564d2305

Case number 3:
Original file: http://s3-us-west-2.amazonaws.com/app.local.getstencil.com/upldfc4d6cb3
Resized thumb: https://d3jqyot5ky4oh9.cloudfront.net/full-fit-in/304x212/filters:quality(100)/upldfc4d6cb3
What it should look like (served by Cloudinary): https://dx5683gi1tv0w.cloudfront.net/dtrjyhj9q/image/upload/if_ar_gt_1.0,h_212,c_scale/if_else,w_304,c_scale/s3.local/upldfc4d6cb3

I'm assuming these may be Thumbor/Pillow converting issues. But because of that, I can't rely on the CloudFormation for all my production needs.

The way I get around this is by only serving images that have the mime type value image/jpeg through this CloudFormation stack, and then using Cloudinary for all other types.

@stevemorad sorry if this is overly long. It took me a lot of work to figure out all this, so I'm hoping this is useful for this project to be improved, and/or anyone else who's considering using this in a production environment.

[npedrini]

Did you test the new template directly from GitHub without running it through the build process? The bucket name variable does not appear to be configured properly (%%BUCKET_NAME%%-us-west-2).

If you want to test a precompiled version, please use the published template:
https://s3.amazonaws.com/solutions-reference/serverless-image-handler/latest/serverless-image-handler.template

Or make sure you follow the build process and use the build-s3-dist.sh file in the deployment directory to build the zip files using your own distribution bucket (e.g. ./build-s3-dist.sh mybucketname)

Just FYI, I was also getting the error @onassar reported, but my problem ended up being that I was using the uncompiled template directly from the repo. Once I used the compiled version linked by @stevemorad I no longer got the bucket name error.