Serverless Event-Driven Data Pipeline on AWS Lambda and S3

A reference architecture for cost-efficient, auto-scaling ETL pipelines built entirely on AWS managed services — S3 event notifications, Lambda, Step Functions, and Glue.

Ingestion and orchestration

S3 event notifications trigger a Lambda function on new object arrival, which validates the file and starts a Step Functions state machine — this keeps orchestration logic out of individual Lambda functions and makes the pipeline’s error handling visible as a state diagram.

Transform and load

AWS Glue jobs handle the heavier transformation steps that exceed Lambda’s execution time and memory limits, while lightweight transforms stay in Lambda to avoid Glue’s higher cold-start latency for small files.

Cost and scaling characteristics

This pattern scales to zero when there’s no data flowing, which makes it substantially cheaper than an always-on cluster for bursty, unpredictable data volumes — the tradeoff is higher per-unit compute cost at sustained high volume, where a persistent EMR or Glue streaming job becomes more economical.