In this course we will learn how Pig Latin for scripts written to focus on ETL workflow logic. Pig on QDS enables programmers to quickly test scripts then move them into production on large-scale clusters. With Pig, QDS users can query data in the Hadoop Distributed File System (HDFS) or Amazon Simple Storage Service (S3) and store results in variables for reference in subsequent processing steps. Without worrying about managing the underlying infrastructure, developers can focus on building complex data pipelines by combining many data transformation steps in one script instead of a combination of SQL queries.
it easy to debug both active and historical jobs with a Spark Application UI. Results and logs are always available even without active running clusters.