Big Data Technologies Study Notes

Big Data Characteristics, Adoption, Sources and Data Curation

What this topic is about and why it matters

Big data means datasets whose scale, speed or complexity need distributed storage, parallel processing or specialised analytical methods. In other words, they are too large, too fast-moving or too varied for a traditional single-machine database to handle well. This chapter is the starting point for the whole Big Data Technologies course. It explains what makes data "big", where it comes from, how organisations adopt big-data platforms sensibly, and how data is curated (looked after) so that it stays trustworthy and reusable from creation until the end of its life.

Later topics teach the tools: Hadoop (topic 2), HDFS (topics 3–4), MapReduce, HBase, Hive, Spark and Kafka. Those tools only make sense once you understand the problem they solve, which is the subject of this topic.

Prerequisites

• Basic Linux commands (ls, wc, head) and an idea of files and directories.
• SQL basics: tables, rows, columns, primary keys.
• Data size units. In this course, 1 KB = 1,024 bytes, 1 MB = 1,024 KB, and so on (binary units). Storage vendors often use decimal units (1 GB = 1,000,000,000 bytes), so always check which one a question uses.

KB -> MB -> GB -> TB -> PB -> EB -> ZB   (each step x 1,024 in binary units)
1 TB = 1,024 GB          1 PB = 1,024 TB = 1,048,576 GB
1 hour of 4K video is roughly 7 GB, so 1 TB holds roughly 146 hours of it

Big data: beyond the hype

The word "big data" was heavily marketed in the 2010s, and much of the talk overstated what it can do. A realistic view separates the hype from the substance:

• Size alone is not the point. A huge dataset with no business or research question produces cost, not value. Value comes from a clear question, good-quality data and people who can act on the answer.
• More data does not automatically mean better answers. A biased sample stays biased however large it gets, and with millions of rows almost any two variables will show some correlation. Correlation is still not causation.
• Big-data systems complement, not replace, relational databases. Banking transactions still need the ACID guarantees of an RDBMS (Relational Database Management System). Big-data platforms are added for workloads an RDBMS handles poorly.
• It is an engineering and organisational change, not just a product purchase. Skills, governance and data quality decide success more than the choice of software does.

Why traditional systems struggle

An RDBMS stores structured data in tables of rows and columns. It is based on E. F. Codd's relational model (1970) and is processed with SQL. Enterprise products such as Oracle, SQL Server, Db2, MySQL, PostgreSQL, MariaDB and SQLite use a client–server architecture with full ACID transactions. They are excellent for consistent, structured, moderate-sized data on one powerful server. They struggle when data reaches hundreds of terabytes, arrives at millions of records per second, or comes in dozens of formats (images, logs, JSON).

The classic illustration is disk speed. A single disk reading at about 100 MB/s needs 1,048,576 MB ÷ 100 MB/s ≈ 10,486 seconds, which is about 2.9 hours, to read 1 TB. If the same terabyte is spread over 100 disks that are read in parallel, each disk reads only 1/100 of the data, so the time drops to about 105 seconds, or 1.75 minutes. This is why big-data systems spread data across many machines.

Scale up versus scale out

• Scale up (vertical scaling) adds CPU, RAM or disk to one machine. It is simple, but it hits a hardware ceiling, gets very expensive at the top end, and the single machine remains a single point of failure.
• Scale out (horizontal scaling) adds more ordinary (commodity) machines and spreads storage and computation across them. This is how Hadoop, HBase and Spark grow.

Scaling out produces a distributed system: networked computers that coordinate their work, appear to users as one service, and keep working when some machines fail (partial failure). Distributed big-data platforms also use data locality. Instead of copying huge data blocks over the network to the program, they schedule the computation on or near the nodes that already hold the data. Moving a few kilobytes of code is cheaper than moving terabytes of data.

The five Vs of big data

Analyst Doug Laney (META Group, 2001) first described big data with three Vs: Volume, Velocity and Variety. Veracity and Value were added later, which gives the five Vs commonly taught today.

Typical velocity figures: YouTube receives roughly 500 hours of video per minute; stock exchanges process very high message rates; IoT (Internet of Things) deployments produce billions of sensor readings per day; e-commerce sites handle thousands of orders per second on sale days. Exact numbers change every year. Learn the idea, not the statistic.

Worked calculations

Velocity to daily volume. A stream of 6,000 events per second runs for 86,400 seconds per day:

6,000 x 86,400 = 518,400,000 events per day   (about 518 million)

Sensor data volume. There are 10,000 sensors, each sending one 200-byte reading per second:

10,000 x 86,400 x 200 = 172,800,000,000 bytes per day
= 172.8 GB per day in decimal units (about 160.9 GiB in binary units)

Yearly capacity planning. A platform ingests 2 TB of new raw data per day. In one year that is 2 × 365 = 730 TB. Distributed file systems usually keep several copies of each block for fault tolerance (HDFS keeps 3 by default, covered in topic 3), so the raw disk needed is 730 × 3 = 2,190 TB, which is about 2.14 PB. This is before compression and free-space headroom.

Types of data: structured, semi-structured, unstructured

Structured data follows a fixed, predefined schema of rows and columns and is easily managed by an RDBMS with SQL. Examples are bank transactions (ID, date, amount, account), employee records (EmpID, name, salary, department) and student grades.

ID  | Name  | Age | Salary
001 | Alice | 28  | 55000
002 | Bob   | 35  | 72000

Semi-structured data has some organisation through tags or keys, but no fixed table layout. Each record may have different fields. Common formats are JSON, XML and YAML. Typical sources are API responses, social-media posts and MongoDB documents:

{
  "post": "Weekend trip",
  "location": "Mumbai",
  "likes": 240,
  "tags": ["travel", "tech"],
  "image": "photo.jpg"
}

Unstructured data has no predefined data model: images (JPG, PNG), video (MP4), audio (MP3, WAV), free-form text, emails, PDFs and ML model files. It is usually kept in file systems or object stores and needs text processing or machine learning to search.

Industry surveys commonly estimate that about 80% of enterprise data is unstructured or semi-structured. That is one reason variety is a core big-data challenge.

Big-data skills and sources

Sources of big data

A big-data source is any operational system, sensor, application, log, transaction stream or external feed that produces data for a big-data platform. Sources are often grouped as follows:

Skills a big-data team needs

Big-data work is a team effort. Data engineering is the building and maintenance of systems that take in raw data and turn it into high-quality, consistent information for analysis and machine learning. A data engineer handles ingestion, storage design (data warehouse, data lake, NoSQL), transformation, pipeline automation and quality checks along the lifecycle:

Source -> Ingestion -> Storage -> Transformation -> Serving
(DBs, logs, APIs, IoT)                      (analysts, data scientists, BI apps)

Domain knowledge (finance, healthcare, science) and communication skills matter as much as tools.

Big-data adoption

Big-data adoption is the process by which an organisation picks valuable use cases, builds a suitable platform and makes data-driven work part of normal operations. A sensible adoption path looks like this:

1. Start from a business or research question with measurable value, for example "reduce card fraud losses by 10%".
2. Assess the data. What exists, where it lives, its quality, and who owns it.
3. Run a pilot (proof of concept) on a limited scope with clear success criteria.
4. Choose the platform: on-premises Hadoop cluster, cloud managed service, or a hybrid. Weigh cost, skills and data-residency rules.
5. Put governance in place. Governance is the set of policies, roles and controls that define data ownership, quality, security, permitted use, retention and accountability.
6. Scale and operationalise. Automate pipelines, monitor them, and train users.

Common barriers are the skills gap, data silos between departments, poor data quality, privacy and regulatory concerns, unclear return on investment, and culture ("we have always decided by gut feel"). Buying a cluster before knowing the use case is the classic adoption mistake.

Research and the changing nature of data repositories

A data repository is a managed place where data is stored, described and made available. Repositories have changed with the data:

• 1970s: programs read and wrote flat files directly, at kilobyte-to-megabyte scale.
• 1980s onward: relational databases (RDBMS) with SQL and ACID transactions.
• 1990s: data warehouses for analytical (OLAP) reporting.
• 2000s onward: NoSQL databases and distributed file systems (HDFS) for web-scale data. Later came data lakes and cloud object stores holding raw data of every type.

NoSQL systems relax the strict relational model to scale out. They often follow BASE (Basically Available, Soft state, Eventual consistency) rather than full ACID. The CAP theorem says that when a network partition happens, a distributed store must choose between Consistency (after an update, all clients see the same data) and Availability (every request gets a response). Partition tolerance means the system keeps working even when communication between servers is unreliable.

In a document store, an RDBMS table maps to a collection, a row maps to a document, and a column maps to a field. Two phone documents in the same collection can have different fields, for example one with GPS and Camera and another without.

In research, repositories have also changed. Funders and journals increasingly require data management plans and data sharing. This led to three kinds of research repository: institutional (run by a university), disciplinary (for one field, such as genomics or earth science) and generalist (Zenodo, Dryad, Figshare, Dataverse). Datasets receive persistent identifiers such as DOIs so they can be cited. The FAIR principles (2016) say data should be Findable, Accessible, Interoperable and Reusable.

Data curation

Data curation is the controlled selection, cleaning, documentation, organisation and preservation of data so it remains trustworthy and reusable over time. It directly addresses the veracity and value Vs.

The curation lifecycle

The Digital Curation Centre (DCC) Curation Lifecycle Model describes the sequential actions: conceptualise → create or receive → appraise and select → ingest → preservation action → store → access, use and reuse → transform. Occasional actions include dispose, reappraise and migrate. Description, metadata and preservation planning apply throughout.

The OAIS reference model (Open Archival Information System, ISO 14721) names three information packages:

Data sharing and reuse: implications for curation

Data is only reusable if a stranger can understand it. Sharing practices therefore shape what curators must do:

• Metadata and documentation: README files, data dictionaries or codebooks (column meanings, units, codes), and methods.
• Open, well-documented formats (CSV, JSON, NetCDF) rather than obsolete proprietary ones.
• Licences that state permitted reuse (for example CC0 or CC-BY).
• Privacy: anonymise personal data, and restrict access where consent requires it.
• Provenance and versioning: record where data came from and what changed between versions.

Overlooked and overrated data sharing

Some valuable data is overlooked: small "long-tail" datasets, negative results and lab notebooks that are never deposited and are quietly lost. Some sharing is overrated: uploading files with no documentation counts as "shared" but is practically unusable. Sharing also has real costs, such as preparation time, storage and privacy risk. Researchers hesitate because of worries about credit, misuse, effort and confidentiality. Good curation focuses effort on data with reuse value and makes it genuinely understandable.

Data curation services in action

Repository and library curation services typically offer:

• help writing data management plans;
• deposit and ingest with quality checks (file integrity, readable formats, complete metadata);
• metadata enrichment and format normalisation;
• DOI assignment and landing pages;
• access control and licensing;
• long-term preservation: multiple copies, periodic fixity checks (checksums that prove a file has not changed), and format migration before formats become obsolete.

Open exit: reaching the end of the data life cycle

Not all data should be kept forever. Retention schedules, legal requirements, cost and reappraisal can lead to deaccession, which means formally removing data from a collection. A responsible "open exit" is planned and transparent:

• document the decision and its reason;
• offer the data to another suitable repository if it still has value;
• securely delete where required;
• keep a tombstone landing page so the DOI still resolves and explains that the data was withdrawn and why.

Silently deleting files breaks citations and trust.

The current state of meta-repositories

A meta-repository is a registry or catalogue that aggregates descriptions of many repositories or datasets, so users can search across them. Examples:

• re3data.org, the Registry of Research Data Repositories, which helps find a suitable repository for a discipline;
• DataCite search across DOI metadata;
• Google Dataset Search;
• OpenAIRE.

Meta-repositories usually harvest metadata (for example through the OAI-PMH protocol) rather than hold the data itself. Their current weaknesses are inconsistent metadata quality, duplicate records, broken or stale links, and uneven coverage across disciplines.

Curation of scientific data at risk of loss: data rescue and dissemination

Data is at risk when it sits on obsolete media (magnetic tape, floppy disks), in unreadable formats, only on paper, or with a retiring researcher or unfunded project. Data rescue recovers such data and makes it usable:

1. Inventory the material and assess its scientific value.
2. Recover or migrate bits from old media, and digitise paper records.
3. Convert to open, current formats.
4. Document: add metadata, units, methods and provenance.
5. Validate against originals and record checksums.
6. Deposit in a trusted repository with a persistent identifier, and disseminate it so others can find and reuse it.

Rescued historical observations, such as old weather records, are valuable because they can never be measured again.

Common misconceptions

Worked mini-lab: curating a small sensor dataset

This lab mirrors the curation steps above on a tiny dataset using Python with pandas. It profiles the data (veracity check), removes a duplicate, drops a physically impossible reading, standardises city names, and records a SHA-256 checksum for fixity.

import hashlib
import io

import pandas as pd

raw = """sensor_id,ts,temp_c,city
S1,2026-03-01 10:00,21.5,Pune
S2,2026-03-01 10:00,,Mumbai
S2,2026-03-01 10:00,,Mumbai
S3,2026-03-01 10:00,999,pune
S4,2026-03-01 10:00,23.1,PUNE
"""
df = pd.read_csv(io.StringIO(raw))
print("rows:", len(df))
print("duplicates:", df.duplicated().sum())
print("missing temp:", df["temp_c"].isna().sum())

clean = df.drop_duplicates().copy()
clean["city"] = clean["city"].str.title()
clean = clean[clean["temp_c"].between(-50, 60) | clean["temp_c"].isna()]
print("clean rows:", len(clean))
print(clean.to_string(index=False))

csv_bytes = clean.to_csv(index=False).encode("utf-8")
print("sha256:", hashlib.sha256(csv_bytes).hexdigest()[:16])

Expected output:

rows: 5
duplicates: 1
missing temp: 2
clean rows: 3
sensor_id               ts  temp_c   city
       S1 2026-03-01 10:00    21.5   Pune
       S2 2026-03-01 10:00     NaN Mumbai
       S4 2026-03-01 10:00    23.1   Pune
sha256: d1eaf0570021e95a

Points to notice:

• drop_duplicates() treats the two identical S2 rows, both with a missing temperature, as duplicates. Five rows become four.
• The 999 °C reading fails the plausibility range and is removed. The missing S2 value is kept but documented, because deleting it would hide a sensor problem.
• str.title() turns pune and PUNE into Pune, which fixes a variety and consistency issue.

To finish the curation, save the cleaned data with clean.to_csv("readings.csv", index=False), record a small data dictionary (column, meaning, unit) in a README, and store the checksum. On Linux, fixity is checked from the shell with sha256sum; on macOS, use shasum -a 256. If the file changes, the hash changes. wc -l counts lines (one header plus three records), which is a quick completeness check:

wc -l readings.csv
sha256sum readings.csv

4 readings.csv
d1eaf0570021e95a...  readings.csv   (64 hex characters in full; same value as printed by the Python script)

Key points summary

• Big data is data whose volume, velocity or variety (plus veracity and value) exceeds what single-machine tools handle well. Laney's original three Vs were volume, velocity and variety.
• Data is structured (fixed schema), semi-structured (JSON/XML keys and tags) or unstructured (images, video, free text).
• Sources are human-generated, machine-generated, transactional, scientific and external. Big-data teams combine engineers, analysts, scientists, administrators and data stewards.
• Big-data platforms scale out across commodity machines and use data locality, because reading data from many disks in parallel is far faster than from one.
• Adoption should start from a valuable question, then a pilot, then platform choice, governance and scaling, not from buying technology.
• Repositories evolved from files to RDBMS, warehouses, NoSQL, HDFS and data lakes. Research repositories follow the FAIR principles and use DOIs.
• Curation keeps data trustworthy and reusable through the DCC lifecycle and OAIS packages (SIP, AIP, DIP), metadata, licences, fixity checks, planned exits with tombstones, meta-repositories such as re3data.org, and rescue of at-risk data.