HDP Developer: Apache Pig and Hive (Hadoop) Training in New Zealand

Software developers who need to understand and develop applications for Hadoop.

• At the completion of the course students will be able to:
• Describe Hadoop, YARN and use cases for Hadoop
• Describe Hadoop ecosystem tools and frameworks
• Describe the HDFS architecture
• Use the Hadoop client to input data into HDFS
• Transfer data between Hadoop and a relational database
• Explain YARN and MapReduce architectures
• Run a MapReduce job on YARN
• Use Pig to explore and transform data in HDFS
• Understand how Hive tables are defined and implemented
• Use Hive to explore and analyze data sets
• Create and populate a Hive table that uses ORC file formats
• Explain and use the various Hive file formats
• Use the new Hive windowing functions
• Use Hive to run SQL-like queries to perform data analysis
• Use Hive to join datasets using a variety of techniques
• Write efficient Hive queries
• Perform data analytics using the DataFu Pig library
• Explain the uses and purpose of HCatalog
• Use HCatalog with Pig and Hive
• Define and schedule an Oozie workflow
• Present the Spark ecosystem and high-level architecture
• Perform data analysis with Spark's Resilient Distributed Dataset API
• Explore Spark SQL and the DataFrame API

• Use HDFS commands to add/remove files and folders
• Use Sqoop to transfer data between HDFS and a RDBMS
• Run MapReduce and YARN application jobs
• Explore, transform, split and join datasets using Pig
• Use Pig to transform and export a dataset for use with Hive
• Use HCatLoader and HcatStorer
• Use Hive to discover useful information in a dataset
• Describe how Hive queries get executed as MapReduce jobs
• Perform a join of two datasets with Hive
• Use advanced Hive features: windowing, views, ORC files
• Use Hive analytics functions
• Write a custom reducer in Python
• Analyze clickstream data and compute quantiles with DataFu
• Use Hive to compute ngrams on Avro-formatted files
• Define an Oozie workflow
• Use Spark Core to read files and perform data analysis
• Create and join DataFrames with Spark SQL