Big and Fast Data
Georgios Gousios and Burcu Kulahcioglu Ozkan
31 August 2026
Big and Fast Data
What is big data?
An overloaded, fuzzy term
“Data too large to be efficiently processed on a single computer”
“Massive amounts of diverse, unstructured data produced by high-performance applications”
How big is “Big”?
Typical numbers associated with Big Data
~0.4 zettabytes (400 exabytes, 400 million TB) of data produced daily (by 2026)
~21 billion IOT devices with internet access (by 2026)
Google processes Google processes >5 trillion searches/year (~14 billion/day, ~160K/sec)
Amazon processes ~177 orders/second (~15.3 million orders/day)](https://capitaloneshopping.com/research/amazon-orders-per-day/)
ChatGPT processes billions of prompts a day:
- 2022: ~10M queries/day (~116/sec)
- 2024: >1B messages/day (~11.6K/sec)
- 2025: >2.5B prompts/day (~29K/sec)
- 2026: ~2.5B prompts/day (~29K/sec)
Hyperscale computing: Companies such as Google, Amazon, Microsoft, and OpenAI operate globally distributed infrastructure at enormous scale, processing massive volumes of data and computation.
How big is “Big”? – FaceBook
Numbers (\(\Leftarrow\)) are from 2014! Today on FB:
- 1.91 Billion active users per day
- 350 million photos per day (148k/min)
- Every min: 510k comments, 293k status updates
How big is “Big”? – Instagram
- More than 2B daily active users, clicking around the app
- ~95M photos & videos uploaded every day
- More than 3B monthly active users
- ~686M followers for the most-followed account
- ~680M followers for the most-followed person
How big is “Big”? – TikTok
- ~2 Billion monthly active users
- ~1.1 billion daily active users
- ~34 million videos uploaded every day
- ~100 minutes per user per day
How big is “Big”? – ChatGPT
- 1+ billion monthly users
- ~1 billion weekly users
- ~2.5 billion prompts every day
- ~29,000 prompts every second
The many Vs of Big data
Main Vs, by Doug Laney
- Volume: large amounts of data
- Variety: data comes in many different forms from diverse sources
- Velocity: the content is changing quickly
More Vs
- Value: data alone is not enough; how can value be derived from it?
- Validity: ensure that the interpreted data is sound, clean
- Veracity: can we trust the data? Does it come from reliable sources?
- Volatility: how long does data need to be kept for? When is it considered irrelevant?
- Visibility: data from diverse sources need to be stitched together
- Virality: how quickly data is spread and shared
Volume
We call Big Data big because it is really big:
- 90% of all the data ever was created in the last 2 years
- Every human created at least 1.7 MB of data per second in 2020.
- Every day, 306.4 billion emails are sent, and 500 million Tweets are made.
- The global big data and business analytics market was valued at [$138.9 billion in 2020 and is expected to grow to $229 billion in 2025] (https://www.precisely.com/blog/big-data/quality-data-big-data-worth).
Variety
- Structured data: SQL tables, format is known
- Semi-structured data: JSON, XML
- Unstructured data: Text, audio, video
We often need to combine various data sources of different types to come up with a result
Velocity
Data is not just big; it is generated and needs to be processed fast. Think of:
- Data centers writing to log files
- IoT sensors reporting temperatures around the globe
- Twitter: >500 million tweets a day (or 6k/sec)
- Stock Markets: high-frequency trading (latency costs money)
- Online advertising
Data needs to be processed with soft or hard real-time guarantees
Big Data processing
The ETL cycle
- Extract: Convert raw or semi-structured data into structured data
- Transform: Convert units, join data sources, cleanup etc
- Load: Load the data into another system for further processing
Big data engineering is concerned with building pipelines
Big data analytics is concerned with discovering patterns
How to process all this data?
- Batch processing: All data exists in some data store, a program processes the whole dataset at once
- Stream processing: Processing of data as they arrive to the system
2 basic approaches to distribute data processing operations on lots of machines
- Divide the data in chunks, apply the same algorithm on all chunks (data-parallelism)
- Divide the problem in chunks, run it on a cluster of machines (task-parallelism)
Desired properties of a big data processing system
- Robustness and fault-tolerance
- Low latency reads and updates
- Scalability
- Generalization
- Extensibility
- Ad hoc queries
- Minimal maintenance
- Debuggability
Large-scale computing
Not a new discipline:
- Cray-1 appeared in the late ’70s
- Physicists used super computers for simulations in the ’80s
- Shared-memory designs still in large scale use (e.g. TOP500 supercomputers)
What is new?
Large scale processing on distributed, commodity computers, enabled by advanced software using elastic resource allocation.
Software (not HW!) is what drives the Big Data industry.
A brief history of Big Data tech
- 2003: Google publishes the Google Filesystem paper, a large-scale distributed file system
- 2004: Google publishes the Map/Reduce paper, a distributed data processing abstraction
- 2006: Yahoo creates and open sources Hadoop, inspired by the Google papers
- 2006: Amazon launches its Elastic Compute Cloud, offering cheap, elastic resources
- 2007: Amazon publishes the DynamoDB paper, sketches the blueprints of a cloud-native database
- 2009 – onwards: The NoSQL movement. Schema-less, distributed databases defy the SQL way of storing data
- 2010: Matei Zaharia et al. publish the Spark paper, brings FP to in-memory computations
- 2012: Both Spark Streaming and Apache Flink appear, able to handle really high volume stream processing
- 2012: Alex Krizhevsky et al. publish their deep learning image classification paper re-igniting interest in neural networks and solidifying the value of big data
- 2017: Vaswani et al. publish their Transformer paper, removing the removing the sequential architectural bottleneck enabled highly parallel processing
- 2020: Brown et al. publish their GPT-3 paper, scaling that architecture to webscale text achiving emergent capabilities
The Big Data Tech Landscape (2019)
The ML, AI & Big Data Landscape (2024)
The technology landscape changed, but the fundamental problem remains: how do we process massive amounts of distributed data efficiently and reliably?
Progress is mostly industry-driven
D: Most advancement in Big Data technologies came from the industry. The universities only started contributing late. Why?
. . .
Typical problems solved with Big Data
- Modeling: What factors influence particular outcomes/behaviors?
- Information retrieval: Finding needles in haystacks, aka search engines
- Collaborative filtering: Recommending items based on items other users with similar tastes have chosen
- Outlier detection: Discovering outstanding transactions
- Graph analytics: Finding patterns and relationships in large networks, e.g., social networks, web graphs, recommendation networks
- Observability and log analytics: Processing logs, traces, and metrics to understand and monitor complex distributed systems
- AI/LLM data processing: Processing large collections of documents, interactions, embeddings, and evaluation data for retrieval-augmented generation, model evaluation, and AI applications
Image credits
- Data is the new oil picture (c) the Economist
Bibliography
Copyright
This work is (c) 2017, 2018, 2019, 2020, 2021 - onwards by TU Delft and Georgios Gousios and licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.