Big Data Mastery: Data Pipelines, Processing, and Messaging

Why should you attend?

This course offers an in-depth exploration of big data technologies and methodologies, focusing on the creation and optimization of data pipelines, processing frameworks, and messaging systems. Participants will begin by understanding the foundational architecture of Hadoop, including HDFS storage mechanisms and YARN resource management. Through hands-on exercises, learners will set up pseudo-clusters using Docker to gain practical experience. The course progresses into MapReduce programming, where students will learn about mapper, reducer, and combiner design patterns. Optimization techniques for the shuffle/sort phase are covered to enhance performance. Learners will also delve into Hive data warehousing concepts, comparing HiveQL with ANSI SQL syntax and exploring partitioning strategies to efficiently analyze large datasets. Spark's core concepts are thoroughly examined, highlighting the tradeoffs between RDDs and DataFrames. The Catalyst optimizer is explored in detail to understand how Spark executes queries efficiently. Students will engage in hands-on ETL pipeline development with complex joins to solidify their understanding. Advanced topics include Spark optimization techniques such as memory management and handling data skew. The course also covers batch processing patterns like Lambda and Kappa architectures, ensuring participants can handle late-arriving data effectively. Security within the Hadoop ecosystem is addressed through Kerberos authentication and Apache Ranger policy control. Finally, learners will explore Kafka's core concepts and Streams API for real-time data processing, culminating in a comprehensive end-to-end pipeline design project that integrates multiple big data technologies.