Skip to content

Latest commit

 

History

History
349 lines (258 loc) · 11 KB

readme.md

File metadata and controls

349 lines (258 loc) · 11 KB

Python Package Builder for AWS Lambda

Build AWS Lambda deployment packages and layers for multiple Python versions (3.10+) and architectures (x86_64/arm64).

CodeQL Dependabot Build and Release Github Pages

Inspired by LambdaZipper

Features

  • 🐍 Multiple Python versions: Support for Python 3.10, 3.11, 3.12, 3.13, and 3.14 (default)
  • 🏗️ Multi-architecture: Build for both x86_64 (amd64) and arm64 (aarch64) by default
  • 📦 Bulk packaging: Uses requirements.txt by default to package multiple dependencies
  • 🔄 Two output formats: Single combined archive (default) or individual archives per package
  • ⚡ Fast builds: Based on Amazon Linux 2023 for compatibility with Lambda runtime
  • 🚀 Zero-config: Run without arguments for sensible defaults

Quick Start

Default Usage (Recommended)

Simply run the build script with no arguments:

./build-multiarch.sh

Default behavior:

  • Uses Python 3.14 (latest)
  • Reads from requirements.txt in the current directory
  • Builds for both x86_64 and arm64 architectures
  • Creates single combined archive per architecture
  • Automatically uses the correct Amazon Linux version for Lambda compatibility

Result: combined-python3.14-x86_64.zip and combined-python3.14-aarch64.zip in ./output/

Note: This tool uses Amazon Linux 2 for Python 3.10-3.11 and Amazon Linux 2023 for Python 3.12+ to ensure compatibility with AWS Lambda Python runtimes.

Customizing the Build

# Use a specific Python version
./build-multiarch.sh --python 3.13

# Build only for x86_64
./build-multiarch.sh --skip-arm64

# Create individual archives per package
./build-multiarch.sh --individual

# Build a single package instead of requirements.txt
./build-multiarch.sh --package numpy

# Combine options
./build-multiarch.sh --python 3.12 --skip-arm64 --individual

Command Reference

build-multiarch.sh Options

Option Description Default
--python VERSION Python version (3.10-3.14) 3.14
-r, --requirements FILE Path to requirements.txt requirements.txt
--package NAME Single package name (overrides requirements.txt) -
--skip-x86, --skip-amd64 Skip x86_64 architecture build Build both
--skip-arm64 Skip arm64 architecture build Build both
--individual Create individual archives per package Single combined
--image NAME Custom Docker image name lambda-zipper
--help Show help message -

Usage Examples

# Default: Python 3.14, requirements.txt, multi-arch, combined archive
./build-multiarch.sh

# Use different Python version
./build-multiarch.sh --python 3.13

# Build only for x86_64 architecture
./build-multiarch.sh --skip-arm64

# Create individual archives for each package
./build-multiarch.sh --individual

# Use custom requirements file
./build-multiarch.sh -r ./my-requirements.txt

# Build a single package (ignores requirements.txt)
./build-multiarch.sh --package numpy

# Combine multiple options
./build-multiarch.sh --python 3.12 --skip-arm64 --individual -r prod-requirements.txt

Advanced Usage

Manual Docker Build (Lower-Level)

For direct Docker control, you can build and run containers manually:

Single Package Mode

Build a Docker image:

docker build --build-arg PYTHON_VERSION=3.14 -t lambda-zipper .

Package a single library:

docker run --rm -v $(pwd)/output:/package lambda-zipper numpy

Requirements.txt with Individual Archives

docker run --rm \
  -v $(pwd)/requirements.txt:/input/requirements.txt \
  -v $(pwd)/output:/package \
  lambda-zipper

Requirements.txt with Combined Archive

docker run --rm \
  -e SINGLE_FILE=true \
  -v $(pwd)/requirements.txt:/input/requirements.txt \
  -v $(pwd)/output:/package \
  lambda-zipper

Supported Python Versions

This project automatically uses the appropriate Amazon Linux version based on AWS Lambda Python runtime requirements:

Python Version Base OS Installation Method Lambda Runtime Compatibility
3.10 Amazon Linux 2 Built from source ✅ python3.10 runtime
3.11 Amazon Linux 2 Built from source ✅ python3.11 runtime
3.12 Amazon Linux 2023 dnf package ✅ python3.12 runtime
3.13 Amazon Linux 2023 dnf package ✅ python3.13 runtime
3.14 Amazon Linux 2023 Built from source ⚙️ Future runtime (Default)

Key Points:

  • Python 3.10 & 3.11: Use Amazon Linux 2 (matches Lambda runtime environment)
  • Python 3.12 & 3.13: Use Amazon Linux 2023 (matches Lambda runtime environment)
  • Python 3.14: Uses Amazon Linux 2023, compiled from source for latest features
  • All versions are built with development headers for compiling binary packages

Output Formats

Lambda Layer Format

python/
  lib/
    pythonX.Y/
      site-packages/
        [your packages here]

Upload as a Lambda Layer in the AWS Console.

Flat Deployment Format

[your packages at root level]

Extract into your Lambda deployment package alongside your code.

Architecture Support

  • x86_64 (amd64): Standard Lambda runtime
  • arm64 (aarch64): AWS Graviton2 Lambda runtime (cost-effective)

Choose the architecture that matches your Lambda function configuration.

Real-World Examples

Data Science Stack

Create requirements.txt:

numpy==1.26.4
pandas==2.1.4
scipy==1.11.4
scikit-learn==1.3.2

Build with defaults (Python 3.14, multi-arch, combined archive):

./build-multiarch.sh

Result: combined-python3.14-x86_64.zip and combined-python3.14-aarch64.zip ready for Lambda Layer

Web Scraping Stack

Create requirements.txt:

requests==2.31.0
beautifulsoup4==4.12.2
lxml==4.9.3

Build for Python 3.13, x86_64 only:

./build-multiarch.sh --python 3.13 --skip-arm64

Computer Vision with OpenCV

Create requirements.txt:

Pillow==10.2.0
opencv-python-headless==4.9.0.80
numpy==1.26.4

Build individual archives for selective deployment:

./build-multiarch.sh --individual

Result: Separate archives for each package, allowing you to pick and choose

AWS SDK Only (Single Package)

Build only boto3 for both architectures:

./build-multiarch.sh --package boto3

Result: boto3-python3.14-x86_64.zip and boto3-python3.14-aarch64.zip

Legacy Python Version Support

Build for Python 3.10 with specific requirements:

./build-multiarch.sh --python 3.10 -r legacy-requirements.txt

Project Structure

.
├── Dockerfile.al2         # Amazon Linux 2 image (Python 3.10-3.11)
├── Dockerfile.al2023      # Amazon Linux 2023 image (Python 3.12-3.14)
├── Dockerfile             # Default (points to AL2023)
├── package.sh             # Packaging script with requirements.txt support
├── build-multiarch.sh     # Helper script for multi-arch builds
├── requirements.example.txt
└── readme.md

How It Works

  1. Base Image Selection: Automatically chooses Amazon Linux 2 (for Python 3.10-3.11) or Amazon Linux 2023 (for Python 3.12+) to match AWS Lambda runtime environments
  2. Python Installation: Installs specified Python version (3.10-3.14) with development headers
  3. Package Installation: Uses pip with --target flag to install packages with proper dependencies
  4. Packaging: Creates zip archives with proper Lambda Layer structure (python/lib/pythonX.Y/site-packages/)
  5. Multi-arch: Leverages Docker buildx for platform-specific builds (x86_64 and arm64)
  6. Runtime Compatibility: Ensures binary compatibility with AWS Lambda execution environments

Troubleshooting

Docker buildx not available

Enable buildx in Docker Desktop settings or install manually:

docker buildx version

Requirements file not found

By default, the script looks for requirements.txt in the current directory. Specify a different file:

./build-multiarch.sh -r path/to/my-requirements.txt

Package has binary dependencies

The build includes gcc, g++, and cmake for compiling binary extensions. Most packages should work, but some may require additional system libraries. Modify the Dockerfile to add specific dependencies.

Lambda Layer size limits

Lambda has a 250MB (unzipped) limit for layers. For large packages:

  • Use --individual flag to split packages into separate archives
  • Deploy multiple smaller layers
  • Use Lambda deployment package directly
  • Consider AWS Lambda Extensions or Container Images

Build takes too long

Python 3.14 is compiled from source and takes longer. For faster builds:

  • Use Python 3.13 or earlier: ./build-multiarch.sh --python 3.13
  • Build only one architecture: ./build-multiarch.sh --skip-arm64
  • Docker caches layers, so subsequent builds are faster

Advanced Docker Usage

Custom Dockerfile Modifications

Add system dependencies:

# Add after the base install in Dockerfile
RUN dnf -y install postgresql-devel libxml2-devel

Building Without the Helper Script

# Build for specific platform
docker buildx build \
  --platform linux/amd64 \
  --build-arg PYTHON_VERSION=3.14 \
  -t lambda-zipper:custom \
  --load .

# Run with requirements (combined archive)
docker run --rm \
  -e SINGLE_FILE=true \
  -v $(pwd)/requirements.txt:/input/requirements.txt \
  -v $(pwd)/output:/package \
  lambda-zipper:custom

# Run with single package
docker run --rm \
  -v $(pwd)/output:/package \
  lambda-zipper:custom numpy

Output File Naming

Archives are named with the following pattern:

Combined archives (default):

  • combined-python{VERSION}-{ARCH}.zip
  • Example: combined-python3.14-x86_64.zip, combined-python3.14-aarch64.zip

Individual package archives (with --individual):

  • {PACKAGE}-python{VERSION}-{ARCH}.zip
  • Example: numpy-python3.14-x86_64.zip, requests-python3.13-aarch64.zip

Contributing

Feel free to submit issues and enhancement requests!

License

This project is open source and available for use and modification.