Data Pipeline Engineering: Airflow, Kubernetes & Spark Course
Master the art of data pipeline engineering with our cutting-edge training program focused on Apache Airflow, Kubernetes, and Spark. Guided by industry experts, this comprehensive course equips you with essential skills to design scalable pipelines for modern data ecosystems. Enroll now to transform your career in data engineering.
Public Pricing
Corporate Pricing
Pax:
Training Provider Pricing
Pax:
Features
Classification
Focus Area Course
- Addresses skill gaps across industries
- Emphasizes emerging technologies and practices
- Enhances workforce capabilities
- Verified by industry experts
- Broadly applicable across sectors
Subsidies
What you'll learn
- Integrate Spark with Airflow for enhanced data processing capabilities
- Deploy applications on Kubernetes for efficient resource management
- Optimize pipeline performance for cost efficiency
- Implement error handling and monitoring strategies in data pipelines
- Design large-scale pipeline architectures using best practices
- Understand the architecture and components of Apache Airflow
- Create and manage DAGs with complex task dependencies
Why should you attend?
This course provides an in-depth exploration of data pipeline engineering, focusing on the integration and orchestration of Apache Airflow, Kubernetes, and Spark. Participants will begin by understanding the fundamentals of workflow orchestration and the architecture of Apache Airflow. Through hands-on exercises, learners will set up their first Airflow environment and create Directed Acyclic Graphs (DAGs) with multiple tasks. As the course progresses, students will delve into advanced topics such as scheduling with cron expressions, managing dependencies, and implementing error handling and monitoring strategies. The curriculum also covers best practices for DAG design patterns, testing, and optimizing hardware configurations to maintain peak performance. The course transitions into container orchestration with Kubernetes, where participants learn about its architecture, resource management, and security features like Role-Based Access Control (RBAC). Learners will deploy applications on Kubernetes, configure persistent storage solutions, and explore dynamic pod generation using the KubernetesPodOperator. Incorporating Spark into the pipeline is another key component of this course. Students will gain insights into Spark's architecture and data processing capabilities through practical exercises that involve writing basic Spark jobs and integrating them with Airflow via the SparkSubmitOperator. The course concludes with a comprehensive project where participants build a production-ready data pipeline utilizing all learned tools and techniques. By the end of this course, participants will have acquired robust skills in designing scalable data pipelines that efficiently integrate Airflow, Kubernetes, and Spark technologies.
Course Syllabus
Day 1 - Airflow Fundamentals & Setup
Short Break
15 minsRecap and Q&A
15 minsLunch
1 hourShort Break
15 minsRecap and Q&A
15 minsEnd of Day 1
Day 2 - Kubernetes & Airflow Integration
Short Break
15 minsRecap and Q&A
15 minsLunch
1 hourShort Break
15 minsRecap and Q&A
15 minsEnd of Day 2
Day 3 - Advanced Kubernetes & Spark
Short Break
15 minsRecap and Q&A
15 minsLunch
1 hourShort Break
15 minsRecap and Q&A
15 minsEnd of Day 3
Day 4 - Production Pipeline Architecture
Short Break
15 minsRecap and Q&A
15 minsLunch
1 hourShort Break
15 minsRecap and Q&A
15 minsEnd of Day 4
Ratings and Reviews
Instructor
Instructor
Instructor
Dr. Esther Suria Kala is a highly accomplished ICT consultant and corporate trainer with nearly 30 years of extensive experience in information technology, project management, and organizational development. As the founder of Angel Software Solutions, she has established herself as a versatile and dedicated professional who delivers cutting-edge training solutions across Malaysia and internationally.
Her academic credentials are exceptional, including a PhD in Psychology with a focus on Human Behaviour and Human Resources, an MBA in Finance, and degrees in both Computer Science and Human Resources. Currently pursuing a second PhD in IT Information Security, Dr. Esther demonstrates an unwavering commitment to continuous learning and staying at the forefront of technological advancement. Her educational foundation is complemented by an impressive array of professional certifications, including Microsoft Certified Trainer (MCT), CISSP, ITIL, SAP Certified Associate, Certified Cyber Security Professional, Data Scientist, AI Engineering, and ECBA certifications.
Dr. Esther's technical expertise spans an extensive range of domains, from traditional programming languages like Java, C++, and PHP to modern technologies including Python, AI coding, machine learning, RPA, and IR 4.0 microservices. She is proficient in database management systems including Microsoft SQL Server and Oracle, web development frameworks, and enterprise solutions like SAP Crystal Reports and SharePoint Server. Her knowledge extends to cybersecurity, ethical hacking, network administration, and data analytics, making her uniquely qualified to address both technical and strategic business challenges.
As a trainer, Dr. Esther excels in bridging the gap between technical complexity and practical application. She delivers comprehensive training programs for both IT and non-IT professionals, covering everything from software development and cloud computing to soft skills such as digital marketing, business analysis, customer service, and project management. Her ability to communicate effectively in both English and Bahasa Malaysia enables her to connect with diverse audiences across different organizational levels.
Her impressive client portfolio includes leading organizations such as Maybank, Telekom Malaysia, Malaysian Airlines, Tenaga Nasional Berhad, Bank Negara, Ministry of Education, Ministry of Finance, and numerous multinational corporations. This extensive experience across banking, telecommunications, government, aviation, and technology sectors demonstrates her adaptability and deep understanding of varied industry requirements. Recognized with the Outstanding Achievement Award in Information Technology from University Malaya in 2013, Dr. Esther continues to empower organizations through innovative training solutions that drive digital transformation and professional excellence.
Instructor
Instructor
Instructor Reviews
Tasmeea Rahman
Data Analytics & Engineering Training Specialist"Tasmeea was an excellent teacher. Very pleasant, willing to assist and answer concerns, and continuously asked how she could assist more. She made the course enjoyable as well as informative."
"Thank you so much for your invaluable guidance. I really enjoyed it and appreciated how much fun you made it! I feel much more equipped to cope with difficult challenges."
"The training was both fascinating and beneficial. Many members of the staff commented on how useful it was. Tasmeea delivers the topic with authority, wit, and passion."
Mohammad Mehdi Lotfinejad
Chief Data Officer & Data Science Trainer"Mehdi and I worked on several projects with company such as Petronas , Shell and CIMB Regional ETC. I must say Mehdi's training was highly appreciated by our clients as he was able to exhibit in full display his vast knowledge as a Data professional. I would highly recommend him to anyone looking for a top tier training expert."
"Not only knowledgeable but also having hands dirty on what he knows Friendly and building networks quickly."
"I had the pleasure of working with Mehdi together on some high-level initiatives such as the Petronas data scientist program and Shell's project to become a data-driven organization. During these projects, Mehdi received numerous accolades for his ability to share his knowledge and mentor up-and-coming data scientists. Based on our shared experiences, I have no hesitation in recommending Mehdi for any project or position he may be considered for."
FAQ
Frequently Asked Questions About This Course
Why should you attend?
This course provides an in-depth exploration of data pipeline engineering, focusing on the integration and orchestration of Apache Airflow, Kubernetes, and Spark. Participants will begin by understanding the fundamentals of workflow orchestration and the architecture of Apache Airflow. Through hands-on exercises, learners will set up their first Airflow environment and create Directed Acyclic Graphs (DAGs) with multiple tasks. As the course progresses, students will delve into advanced topics such as scheduling with cron expressions, managing dependencies, and implementing error handling and monitoring strategies. The curriculum also covers best practices for DAG design patterns, testing, and optimizing hardware configurations to maintain peak performance. The course transitions into container orchestration with Kubernetes, where participants learn about its architecture, resource management, and security features like Role-Based Access Control (RBAC). Learners will deploy applications on Kubernetes, configure persistent storage solutions, and explore dynamic pod generation using the KubernetesPodOperator. Incorporating Spark into the pipeline is another key component of this course. Students will gain insights into Spark's architecture and data processing capabilities through practical exercises that involve writing basic Spark jobs and integrating them with Airflow via the SparkSubmitOperator. The course concludes with a comprehensive project where participants build a production-ready data pipeline utilizing all learned tools and techniques. By the end of this course, participants will have acquired robust skills in designing scalable data pipelines that efficiently integrate Airflow, Kubernetes, and Spark technologies.
What you'll learn
- Integrate Spark with Airflow for enhanced data processing capabilities
- Deploy applications on Kubernetes for efficient resource management
- Optimize pipeline performance for cost efficiency
- Implement error handling and monitoring strategies in data pipelines
- Design large-scale pipeline architectures using best practices
- Understand the architecture and components of Apache Airflow
- Create and manage DAGs with complex task dependencies
Course Syllabus
Day 1 - Airflow Fundamentals & Setup
Short Break
15 minsRecap and Q&A
15 minsLunch
1 hourShort Break
15 minsRecap and Q&A
15 minsEnd of Day 1
Day 2 - Kubernetes & Airflow Integration
Short Break
15 minsRecap and Q&A
15 minsLunch
1 hourShort Break
15 minsRecap and Q&A
15 minsEnd of Day 2
Day 3 - Advanced Kubernetes & Spark
Short Break
15 minsRecap and Q&A
15 minsLunch
1 hourShort Break
15 minsRecap and Q&A
15 minsEnd of Day 3
Day 4 - Production Pipeline Architecture
Short Break
15 minsRecap and Q&A
15 minsLunch
1 hourShort Break
15 minsRecap and Q&A
15 minsEnd of Day 4
Instructor Reviews
Tasmeea Rahman
Data Analytics & Engineering Training Specialist"Tasmeea was an excellent teacher. Very pleasant, willing to assist and answer concerns, and continuously asked how she could assist more. She made the course enjoyable as well as informative."
"Thank you so much for your invaluable guidance. I really enjoyed it and appreciated how much fun you made it! I feel much more equipped to cope with difficult challenges."
"The training was both fascinating and beneficial. Many members of the staff commented on how useful it was. Tasmeea delivers the topic with authority, wit, and passion."
Mohammad Mehdi Lotfinejad
Chief Data Officer & Data Science Trainer"Mehdi and I worked on several projects with company such as Petronas , Shell and CIMB Regional ETC. I must say Mehdi's training was highly appreciated by our clients as he was able to exhibit in full display his vast knowledge as a Data professional. I would highly recommend him to anyone looking for a top tier training expert."
"Not only knowledgeable but also having hands dirty on what he knows Friendly and building networks quickly."
"I had the pleasure of working with Mehdi together on some high-level initiatives such as the Petronas data scientist program and Shell's project to become a data-driven organization. During these projects, Mehdi received numerous accolades for his ability to share his knowledge and mentor up-and-coming data scientists. Based on our shared experiences, I have no hesitation in recommending Mehdi for any project or position he may be considered for."
Public Pricing
Corporate Pricing
Pax:
Training Provider Pricing
Pax:
Features
Classification
Focus Area Course
- Addresses skill gaps across industries
- Emphasizes emerging technologies and practices
- Enhances workforce capabilities
- Verified by industry experts
- Broadly applicable across sectors
Subsidies
Ratings and Reviews
Instructors
Dr. Esther Suria Kala is a highly accomplished ICT consultant and corporate trainer with nearly 30 years of extensive experience in information technology, project management, and organizational development. As the founder of Angel Software Solutions, she has established herself as a versatile and dedicated professional who delivers cutting-edge training solutions across Malaysia and internationally.
Her academic credentials are exceptional, including a PhD in Psychology with a focus on Human Behaviour and Human Resources, an MBA in Finance, and degrees in both Computer Science and Human Resources. Currently pursuing a second PhD in IT Information Security, Dr. Esther demonstrates an unwavering commitment to continuous learning and staying at the forefront of technological advancement. Her educational foundation is complemented by an impressive array of professional certifications, including Microsoft Certified Trainer (MCT), CISSP, ITIL, SAP Certified Associate, Certified Cyber Security Professional, Data Scientist, AI Engineering, and ECBA certifications.
Dr. Esther's technical expertise spans an extensive range of domains, from traditional programming languages like Java, C++, and PHP to modern technologies including Python, AI coding, machine learning, RPA, and IR 4.0 microservices. She is proficient in database management systems including Microsoft SQL Server and Oracle, web development frameworks, and enterprise solutions like SAP Crystal Reports and SharePoint Server. Her knowledge extends to cybersecurity, ethical hacking, network administration, and data analytics, making her uniquely qualified to address both technical and strategic business challenges.
As a trainer, Dr. Esther excels in bridging the gap between technical complexity and practical application. She delivers comprehensive training programs for both IT and non-IT professionals, covering everything from software development and cloud computing to soft skills such as digital marketing, business analysis, customer service, and project management. Her ability to communicate effectively in both English and Bahasa Malaysia enables her to connect with diverse audiences across different organizational levels.
Her impressive client portfolio includes leading organizations such as Maybank, Telekom Malaysia, Malaysian Airlines, Tenaga Nasional Berhad, Bank Negara, Ministry of Education, Ministry of Finance, and numerous multinational corporations. This extensive experience across banking, telecommunications, government, aviation, and technology sectors demonstrates her adaptability and deep understanding of varied industry requirements. Recognized with the Outstanding Achievement Award in Information Technology from University Malaya in 2013, Dr. Esther continues to empower organizations through innovative training solutions that drive digital transformation and professional excellence.
FAQ
Frequently Asked Questions About This Course
Our Offers
Academy for Trainers Academy for Trainers
Teach what you love. Abundent Academy gives you the tools you need to run your own trainings! We provide you with the platform, the students, the materials, and the support you need to succeed!
- Higher trainer payouts
- Ready-made course materials
- Student management system
- AI digital marketing assistant
Academy for Corporates Academy for Corporates
Get unlimited access to all of Abundent Academy's carefully curated courses for your team, all organized according to learning paths and roles! Perfect for companies looking to upskill their workforce and stay ahead in the tech industry.
- Carefully curated courses
- Role-based learning paths
- Team progress tracking
- Gap Identification and Analysis
Academy for Partners Academy for Partners
White-label IT training delivery for training providers. We become your behind-the-scenes delivery arm so you can say yes to more clients without hiring more trainers.
- Expand your training catalog
- 40+ expert trainers ready
- White-label delivery
- You keep client relationships
Top companies choose Abundent Academy
