CloudFront Image Conversion

Created

Apr 8, 2022 8:13 AM

Department

Engineering

Convert Image from .jpg/.png to .webp

This is the continuation of the previous implementation where we serve rails assets through CloudFront. Now When a CloudFront URL is a hit we are going to convert .jpg/.png to a .webp image. Other formats will be served as it is.

The WebP format has become increasingly popular since Google introduced it in 2010. Its biggest selling point lies in its ability to produce much smaller file sizes while maintaining similar image quality. Faster load times = higher conversion rates.

WebP is a modern image format that provides superior lossless and lossy compression for images on the web. WebP lossless images are 26% smaller in size compared to PNGs. WebP lossy images are 25–34% smaller than comparable JPEG images at equivalent SSIM quality index. — Google

References

I would suggest going through the following articles and having an understanding of how the jpg

-> webp conversion flow works and then continues here.

Resizing images with amazon cloudfront lambdaedge aws cdn blog

Converting images to webp from cdn

I followed the step by step section mentioned in the AWS blog. This article contains the bug fixed code and some improvements. Some improvements includes

S3 URL with space and %20 bugs fixed
Dimension support was added. You can request for a particular dimension and if it is available then it will be served.

Step 1: Creating the Deployment Packages

First create a project folder, I named it image-optimization-service . I created a Docker file inside it. I also created the same folder structure as explained. Find the code repo here

image-optimization-service  # This is the root folder
image-optimization-service → lambda → viewer-request-function → handler.js
image-optimization-service → lambda → origin-response-function → handler.js
image-optimization-service → Dockerfile

Dockerfile

FROM amazonlinux:2

WORKDIR /tmp
#install the dependencies
RUN yum -y install tar && yum -y install gzip && yum -y install gcc-c++ && yum -y install findutils

RUN touch ~/.bashrc && chmod +x ~/.bashrc

RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.5/install.sh | bash

RUN source ~/.bashrc && nvm install 10.0

WORKDIR /build

image-optimization-service → lambda → viewer-request-function → handler.js

const userAgent = require('useragent')
const path = require('path')
const querystring = require('querystring');

const variables = {
  allowedDimension : [ {w:100,h:100}, {w:200,h:200}, {w:300,h:300}, {w:400,h:400} ],
  defaultDimension : {w:200,h:200},
  variance: 20,
  webpExtension: 'webp'
};

exports.handler = async (event, context, callback) => {
  const request = event.Records[0].cf.request
  const headers = request.headers

  // parse the querystrings key-value pairs. In our case it would be d=100x100
  const params = querystring.parse(request.querystring);

  const userAgentString = headers['user-agent'] && headers['user-agent'][0] ? headers['user-agent'][0].value : null
  const agent = userAgent.lookup(userAgentString)

  const browsersToInclude = [
    { browser: 'Chrome', version: 23 },
    { browser: 'Opera', version: 15 },
    { browser: 'Android', version: 53 },
    { browser: 'Chrome Mobile', version: 55 },
    { browser: 'Opera Mobile', version: 37 },
    { browser: 'UC Browser', version: 11 },
    { browser: 'Samsung Internet', version: 4 }
  ]

  const supportingBrowser = browsersToInclude
    .find(browser => browser.browser === agent.family && agent.major >= browser.version)
  let fwdUri = request.uri
  request.headers['originalKey'] = [{
    key: 'originalKey',
    value: fwdUri.substring(1)
  }]

  if (supportingBrowser && params.d ) {
    const fileFormat = path.extname(request.uri).replace('.', '')
    // read the dimension parameter value = width x height and split it by 'x'
    const dimensionMatch = params.d.split("x");

    // set the width and height parameters
    let width = dimensionMatch[0];
    let height = dimensionMatch[1];

    const match = fwdUri.match(/(.*)\/(.*)\.(.*)/);

    let prefix = match[1];
    let imageName = match[2];
    let extension = match[3];

    // define variable to be set to true if requested dimension is allowed.
    let matchFound = false;

    // calculate the acceptable variance. If image dimension is 105 and is within acceptable
    // range, then in our case, the dimension would be corrected to 100.
    let variancePercent = (variables.variance/100);

    for (let dimension of variables.allowedDimension) {
        let minWidth = dimension.w - (dimension.w * variancePercent);
        let maxWidth = dimension.w + (dimension.w * variancePercent);
        if(width >= minWidth && width <= maxWidth){
            width = dimension.w;
            height = dimension.h;
            matchFound = true;
            break;
        }
    }

    // read the accept header to determine if webP is supported.
    let accept = headers['accept']?headers['accept'][0].value:"";

    let url = [];
    // build the new uri to be forwarded upstream
    url.push(prefix);
    url.push(width+"x"+height);

    // check support for webp
    if (accept.includes(variables.webpExtension)) {
        url.push(variables.webpExtension);
    }
    else{
        url.push(extension);
    }
    url.push(imageName+"."+extension);
    request.headers['dimensionIncluded'] = [{
      key: 'dimensionIncluded',
      value: 'true'
    }]
    // fwdUri = url.join("/");
    fwdUri = fwdUri.replace(/(\.jpg|\.png|\.jpeg)$/g, '_'+width+'x'+height+'.webp')
    // final modified url is of format image_100x100.webp
    request.uri = fwdUri;
    request.query = request.querystring

    return callback(null, request);

  } else if (supportingBrowser) {
    request.headers['dimensionIncluded'] = [{
      key: 'dimensionIncluded',
      value: 'false'
    }]
    fwdUri = fwdUri.replace(/(\.jpg|\.png|\.jpeg)$/g, '.webp')
    // final modified url is of format image.webp
    request.uri = fwdUri;
    request.query = request.querystring

    return callback(null, request);
  }
  return callback(null, request)
}

image-optimization-service → lambda → origin-response-function → handler.js

const http = require('http');
const https = require('https');
const querystring = require('querystring');

const path = require('path')
const AWS = require('aws-sdk')

const S3 = new AWS.S3({
  signatureVersion: 'v4',
})

const Sharp = require('sharp')
const BUCKET = 'BUCKET-NAME'
const QUALITY = 75

exports.handler = async (event, context, callback) => {

  let response = event.Records[0].cf.response

  let request = event.Records[0].cf.request;

  const headers = response.headers

  const request_headers = request.headers

  const originalKey = request_headers.originalkey[0].value


  const dimensionincluded = request_headers.dimensionincluded[0].value

  // if (path.extname(uri) === '.webp') {
  if (parseInt(response.status) === 404) {

    const { uri } = request

    let params = querystring.parse(request.querystring);

    // read the required path. Ex: uri /images/100x100/webp/image.jpg
    let path = request.uri;

    // read the S3 key from the path variable.
    // Ex: path variable /images/100x100/webp/image.jpg
    let newKey = path.substring(1);

    // Ex: file_name=images/200x200/webp/image_100x100.jpg
    // Getting the image_100x100.webp part alone
    let file_name = path.split('/').slice(-1)[0]

    // get the source image file

    try {
      if (dimensionincluded == "false" || dimensionincluded == false) {
        let requiredFormat = file_name.split('.')[1]

        const bucketResource = await S3.getObject({ Bucket: BUCKET, Key: originalKey }).promise()

        // perform the resize operation

        const sharpImageBuffer = await Sharp(bucketResource.Body)
          .webp({ quality: +QUALITY })
          .toBuffer()

        // save the resized object to S3 bucket with appropriate object key.
        await S3.putObject({
          Body: sharpImageBuffer,
          Bucket: BUCKET,
          ContentType: 'image/webp',
          CacheControl: 'max-age=31536000',
          Key: newKey,
          StorageClass: 'STANDARD'
        }).promise()

        // generate a binary response with resized image
        response.status = 200;
        response.body = sharpImageBuffer.toString('base64');
        response.bodyEncoding = 'base64';
        response.headers['content-type'] = [{ key: 'Content-Type', value: 'image/' + requiredFormat }];
        callback(null, response);
      } else {
        // perform the resize operation

        // Getting the 100x100.webp part alone
        let dimension = file_name.split('_')[1].split('.')[0]

        let requiredFormat = file_name.split('_')[1].split('.')[1]

        let width = dimension.split('x')[0]

        let height = dimension.split('x')[1]

        const bucketResource = await S3.getObject({ Bucket: BUCKET, Key: originalKey }).promise()

        const sharpImageBuffer = await Sharp(bucketResource.Body)
          .resize(parseInt(width), parseInt(height))
          .webp({ quality: +QUALITY })
          .toBuffer()
        // save the resized object to S3 bucket with appropriate object key.
        await S3.putObject({
          Body: sharpImageBuffer,
          Bucket: BUCKET,
          ContentType: 'image/webp',
          CacheControl: 'max-age=31536000',
          Key: newKey,
          StorageClass: 'STANDARD'
        }).promise()

        // generate a binary response with resized image
        response.status = 200;
        response.body = sharpImageBuffer.toString('base64');
        response.bodyEncoding = 'base64';
        response.headers['content-type'] = [{ key: 'Content-Type', value: 'image/' + requiredFormat }];
        callback(null, response);
      }
    } catch (e) {
    }
  } else {
    headers['content-type'] = [{
      'value': 'image/webp',
      'key': 'Content-Type'
    }]
  }
  callback(null, response)
 }

Step 2: Building docker

Change directory to image-optimization-service that's the root of the project.

From the project root folder, run the following commands:

Note: Remember to change the Bucket Name and other values properly, because CloudFront deployment takes a long time.

The Dockerfile is configured to download Amazon Linux and install Node.js 10.0 along with dependencies.

docker build --tag lambci/lambda:build-nodejs10.0 .

Install the sharp and querystring module dependencies and compile the ‘Origin-Response’ function.

docker run --rm --volume ${PWD}/lambda/origin-response-function:/build lambci/lambda:build-nodejs10.0 /bin/bash -c "source ~/.bashrc; npm init -f -y; npm install sharp --save; npm install querystring --save; npm install --only=prod"

Install the querystring module dependencies and compile the ‘Viewer-Request’ function.

docker run --rm --volume ${PWD}/lambda/viewer-request-function:/build lambci/lambda:build-nodejs10.0 /bin/bash -c "source ~/.bashrc; npm init -f -y; npm install querystring --save; npm install path --save; npm install useragent --save; npm install yamlparser; npm install --only=prod"

Package the ‘Origin-Response’ function.

mkdir -p dist && cd lambda/origin-response-function && zip -FS -q -r ../../dist/origin-response-function.zip * && cd ../..

Package the ‘Viewer-Request’ function.

mkdir -p dist && cd lambda/viewer-request-function && zip -FS -q -r ../../dist/viewer-request-function.zip * && cd ../..

From the AWS console, create an S3 bucket in the us-east-1 region to hold the deployment files and upload the zip files created in the above steps. These would be referenced from the CloudFormation template during deployment. Since we thought this might be required in multiple projects and it has to be in us-east-1 we created a bucket like

BUCKET-NAME → project-name-environment → origin-response-function.zip
BUCKET-NAME → project-name-environment → viewer-request-function.zip

Step 3: Deploy the Lambda@Edge Functions

There were minor problems fixed from the AWS blog. IAM Role creation was having errors when cloud formation was deploying

Note: Remember to change the Bucket Name and other values properly, because CloudFront deployment takes a long time.

CloudFormation Template:

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: Rewrite jpg and png requests to webp if the browser supports webp
Resources:
  ViewerRequestFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: s3://image-optimization-service/viewer-request-function-1.zip
      Handler: handler.handler
      Runtime: nodejs10.x
      MemorySize: 128
      Timeout: 1
      Role: !GetAtt WebEdgeLambdaRole.Arn

  ViewerRequestFunctionVersion:
    Type: "AWS::Lambda::Version"
    Properties:
      FunctionName: !Ref ViewerRequestFunction
      Description: "A version of ViewerRequestFunction"

  OriginResponseFunction:
   Type: AWS::Serverless::Function
   Properties:
     CodeUri: s3://image-optimization-service/origin-response-function-1.zip
     Handler: handler.handler
     Runtime: nodejs10.x
     MemorySize: 512
     Timeout: 5
     Role: !GetAtt WebEdgeLambdaRole.Arn

  OriginResponseFunctionVersion:
    Type: "AWS::Lambda::Version"
    Properties:
      FunctionName: !Ref OriginResponseFunction
      Description: "A version of OriginResponseFunction"

  # ==== ROLES ==== #
  WebEdgeLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          -
            Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - "lambda.amazonaws.com"
                - "edgelambda.amazonaws.com"
  # ==== POLICIES ==== #
  PublishLogsPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: Allows functions to write logs
      Roles:
      - !Ref WebEdgeLambdaRole
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
            Resource: '*'

Steps to deploy cloudformation template:

To deploy using cloud formation template.

aws cloudformation deploy --template-file aws-sam-lambda-edge-webp.yaml --stack-name webp-conversion --capabilities CAPABILITY_NAMED_IAM --region us-east-1

After deploying the deployment finishes the CloudFront would have created the lambda functions for both viewer-request and the origin-response. Go to the viewer-request lambda function, check if version is released, copy the ARN of the release with version. For eg it will have the version at the end appended to else release version and copy the ARN.

Now go to cloudfront distribution → go to behaviours section → Go to lambda function section

Choose the viewer request from dropdown → Paste the ARN copied earlier → Include body

Go to the origin-response lambda function, check if version is released, copy the ARN of the release with version. For eg it will have the version at the end appended else release version and copy the ARN.

Now go to cloudfront distribution → go to behaviours section → Go to lambda function section

Choose the origin response from dropdown → Paste the ARN copied earlier

Setting S3 bucket policy:

There was a problem saving the files into the S3 bucket. The bucket policy of the S3 bucket that we are going to change needs to be accessible by the lambda functions. Here is the snippet, you need to change it to the respective bucket names and the lambda role names

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::test-cloudfront-doc/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::410230403282:role/webp-conversion-WebEdgeLambdaRole-Q6ACKG21A14M"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::test-cloudfront-doc/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::410230403282:role/webp-conversion-WebEdgeLambdaRole-Q6ACKG21A14M"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::test-cloudfront-doc/*"
    }
  ]
}

Now the deployment takes around 20 mins. After which if you hit the CloudFront URL it should convert the image and upload the .webp image to the same place.

Deployment Automation

The above settings can be done simply by cloning this repository and running the following script.

./webp_deployment_automation.rb create

Things to note

After deployment if S3 images are not accessible even after policy, Enable the read permission for everyone.
S3 URL with + and space have been handled here.