The term “Big Data” describes datasets that are either too big or change too fast or both to be processed on a single computer.
Big Data Processing provides an introduction to systems used to process Big Data. The main focus of the course is understanding the underpinnings of, programming and engineering big data systems; initially, the course explores general programming primitives that span across big data systems and touches upon distributed systems. Then, the course examines in detail the implementation of data analysis algorithms in Spark, in the context of batch processing applications, and Flink, in the context of streaming applications.
After the end of the course, all students should be able to:
5 ECTS: This means that you need to devote at least 140 hours of study for this course.
Online meetings: The course consists of 12 2-hour meetings. You are not required, but you are strongly encouraged, to attend.
Homework: In the homework assignments, you will have to write code or reply to open questions. You will always work in pairs.
Groups: The students are responsible to form pairs and communicate them to the course TAs, by registering them to Weblab.
Labs: 4 hours per week, designed to help you work together and get support from teaching assistants.
Teaching Assistants: Teaching assistants will be available during lab hours to provide your with feedback on your assignments. Do be active in asking questions, but don’t expect them to provide you with solutions.
| Week | Date | Topic | Teacher | Assignment (Deadline) |
|---|---|---|---|---|
| 1 | 5/9 | Course introduction,
Big and Fast data, Intro to course PL |
BKO | |
| 1 | 8/9 | Programming for Big Data (1): Basic data types, FP in a nutshell | BKO | |
| 2 | 13/9 | Data Engineering on the
Command Line (1), Diomidis’s slides |
DS | Unix (23/09) |
| 2 | 15/9 | Data Engineering on the
Command Line (2), Diomidis’s slides |
DS | |
| 3 | 20/9 | Programming for Big Data (2): Enumerating datasets | BKO | FP (07/10) |
| 3 | 22/9 | Distributed Systems | BKO | |
| 4 | 27/9 | Distributed Databases | BKO | |
| 4 | 29/9 | Distributed Filesystems | BKO | |
| 5 | 4/10 | Spark: RDDs and Pair RDDs | BKO | Spark (21/10) |
| 5 | 6/10 | Spark Internals, Spark SQL | BKO | |
| 6 | 11/10 | Stream processing | GG | Flink (04/11) |
| 6 | 13/10 | Stream processing systems | GG | |
| 7 | 18/10 | Graph Processing | BKO | |
| 7 | 20/10 | no lecture | ||
| 8 | 25/10 | Recap (no lecture) |
The schedule is subject to small changes during the term.
You can access the websites of the previous editions for 2022 and 2021. The lecture notes are based on the lectures taught by Georgios Gousios in 2017-2020.
The lab work is supervised by Thomas Overklift.
Portions of this course have been converted to online educational material by other TU Delft teachers. Please take a look at the following EdX MOOCs / ProfEds:
Use them at your discretion to improve your skills.
(TU Delft only): You can find the Collegerama recordings from 2019 here. Please note that the course contents have slightly changed, so do not base your exam studying on the old lectures.
You can find the course assignments on Brightspace and linked through this page.
All assignments are mandatory.
For submission, we will use Weblab. The course name is CSE2520: Big Data Processing
The student groups must submit each assignment before 23:59 on the day of the deadline.
Late submission: All submissions must be handed in time, with no exceptions. Any late submission will be discarded and will be graded with 0. In case of provable sickness, please contact the course teacher to arrange a case-specific deadline.
Lab assignments (40%): Grade calculated as mean grade for all assignments. There is no minimum grade per individual assignment. If you don’t submit an assignment, or the submission is late, you will get a 0. Each assignment counts for 25% of the lab part. The final lab grade has a minimum of 5.
Written Exam (60%): Closed-book exam. Minimum grade: 5
There will be an exam resit during Q2/3. You are allowed to transfer your assignment grade to the resit as a whole.
If the final assignment grade is in between 4 and 5.7, one of the previously submitted assignments can be repaired.
Partial grades (assignments or exam) cannot be transferred to subsequent years.
Disclaimer: information may change depending on unforeseen circumstances or measures (see: TER).
The course, by design, touches upon various current technologies; as such, there is no single source of truth. The following is an indicative list of resources where more information can be found. If you were to buy a single book about this course, we recommend [1].
This work is (c) 2017, 2018, 2019, 2020, 2021 - onwards by TU Delft and Georgios Gousios and licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.