Event language
UI language
Your AWS Glue bills are burning a hole in your pocket?! If your monthly serverless ETL costs are making your eyes water, you're not alone. This talk isn't just about saving money; it's about reclaiming control. We'll share the gritty, honest truth of how we slashed our AWS Glue bill from a mind-numbing $10,000/month for 80 ETL pipelines to a mere $400, achieving a staggering 96% cost reduction and a 25x improvement, all while maintaining peak performance.<br><br>Talk Flow:<br>1. The Problem (5 mins)<br>Our AWS Glue setup: 80 ETL pipelines costing $10,000/month<br>Why serverless became expensive at scale<br>The breaking point that triggered the migration<br>2. The Solution Architecture (10 mins)<br>Why Apache Airflow over other alternatives<br>Designing for cost: EKS vs ECS on EC2 vs other options<br>The three-component architecture: Webserver, Scheduler, Workers<br>3. The Implementation Battle (15 mins)<br>Infrastructure as Code: Complete Terraform setup walkthrough<br>The Docker Strategy: Baking DAGs into images<br>The Redis Nightmare: Setting up Celery Executor (and why it made me cry)<br>Networking & Security: ALB, security groups, and internal DNS<br>4. The Migration Process (10 mins)<br>Converting Glue jobs to Airflow DAGs<br>Zero-downtime migration strategy<br>Monitoring and debugging in production<br>5. Results & Lessons Learned (5 mins)<br>Cost breakdown: From $10,000 to $400<br>Performance comparison<br>What we'd do differently<br>When to choose open source over managed services<br>Q&A (5 mins)
