Creating and Deploying Custom Lambda Layers for Python Functions

September 2, 2024

AWS (Python) Lambda allows you to execute small code snippets in response to various triggers, such as API calls or the placement of an object in an S3 bucket. When a Lambda function is triggered, a compute instance is launched to execute the code. To minimize latency, the function's runtime environment is kept lightweight, typically consisting of only Python 3.12 and a few essential packages.

Problem: Adding Dependencies Beyond the Default Runtime

By default, a Python Lambda function includes only basic packages and boto3 for accessing AWS services. If your code requires additional packages, such as sqlalchemy, the function will fail unless those dependencies are included.

Solution: Creating a Custom Lambda Layer

A Lambda layer allows you to include additional packages that your function requires. While many prebuilt layers are available, you may need to create a custom layer to meet specific dependencies.

Step 1: Prepare the Requirements File

Create a requirements.txt file listing the packages your function needs:

txt

sqlalchemy

Step 2: Build and Package the Layer

Use the following bash script to install the required packages and create a zip file containing the dependencies:

bash

#!/bin/bash

# Remove old zip file if it exists
if [ -f "layers/custom-dependencies.zip" ]; then
    rm layers/custom-dependencies.zip
fi

# Remove old virtual environment if it exists
if [ -f "venv/" ]; then
    rm -rf venv/
fi

# Create a new virtual environment
python3.12 -m venv venv
source venv/bin/activate

# Install packages into the virtual environment
pip install --platform=manylinux2014_x86_64 --target=venv/lib/python3.12/site-packages/ --python-version=3.12 --only-binary=:all: -r requirements.txt

# Package the installed packages into a zip file
cp -r venv/ python/
zip -r -qq layers/custom-dependencies.zip python/lib/python3.12/site-packages -x '*boto3*' -x '*botocore*'

# Clean up
rm -rf python/
rm -rf venv/

echo "Zip file created successfully!"

The script will produce a zip file containing your custom dependencies.

Note: The zip file must not exceed 50MB, and the total size of all layers and the function code when unzipped cannot exceed 250MB.

Step 3: Upload the Layer to AWS

Navigate to the Lambda Layers page in the AWS Lambda console.
Click Create layer.
Provide a name and optional description for your layer.
Upload the zip file you created.
Choose x86_64 as Compatible architectures.
Choose Python 3.12 as Compatible runtime.
Click Create.
Copy the ARN of the newly created layer.

Step 4: Attach the Layer to Your Lambda Function

Go to the Lambda Functions page in the AWS Lambda console.
Select the function you want to configure.
Under Layers, click Add a layer (can be found at the bottom under Code).
Choose the appropriate layer source.
If using an ARN, enter the ARN and click Verify.
Click Add to attach the layer.

Creating and Deploying Custom Lambda Layers for Python Functions ​

Problem: Adding Dependencies Beyond the Default Runtime ​

Solution: Creating a Custom Lambda Layer ​

Step 1: Prepare the Requirements File ​

Step 2: Build and Package the Layer ​

Step 3: Upload the Layer to AWS ​

Step 4: Attach the Layer to Your Lambda Function ​

References ​