Big data in practice using Spark

Nowadays everybody seems to be working with AI, Data Science and "big data". No doubt also you would like to interrogate your voluminous data sources (click streams, social media, relational data, cloud data, sensor data, ...) and are experiencing the shortcomings of traditional data analytics tools. Maybe you want the processing power of a cluster --and its parallel processing capabilities-- to interrogate your distributed data stores.

If fast prototyping and processing speed are a priority, Spark will most likely be the platform of your choice. Apache Spark is an open source processing engine focusing on low latency, ease of use, flexibility and analytics. It's an alternative to the MapReduce approach delivered of Hadoop with Hive (cf our course Big data in practice using Hadoop). Spark has complemented -actually superseded- Hadoop, due to the higher abstraction of Spark's APIs and its faster, in-memory processing.

More specifically, Spark allows to easily interrogate data sources on HDFS, in a NoSQL database (e.g. Cassandra or HBase), in a relational database, in the cloud (e.g. AWS S3) or in local files. Independent of this, a Spark job can easily run on either your local machine (i.e., in development mode), or on a Hadoop cluster (with Yarn), or a Mesos environment, or Kubernetes, or in the cloud. And all this through a simple Spark script or through a more complex (Java or Python) program or though a web based notebook (e.g. Zeppelin or Databricks).

This course builds on the context set forth in the Big data architecture and infrastructure overview course. You will get hands-on practice on Linux with Spark and its libraries. You learn how to implement robust data processing (in Python, Scala, or R) with an SQL-style interface.

After successful completion of the course, you will have sufficient basic expertise to set up a Spark or Databricks development environment, and use it to interrogate your data. You will be able to write simple Spark scripts and programs (with the Python based PySpark, or with the Scala based SparkShell) based on DataFrames and RDDs, and optionally also use the MLlib, GraphX, or Streaming libraries.