NoSQL Databases

In the evolving landscape of data management, NoSQL databases (Not Only SQL) represent a paradigm shift from traditional relational database management systems (RDBMS). Coined in 1998 and re-popularized in 2009, the term describes non-relational data stores designed for specific use cases where RDBMS struggle: massive scale, flexible schemas, and specialized data models.

The advent of big data, cloud computing, real-time web applications, and agile development has driven demand for data management solutions that handle massive volumes of diverse data at high velocity. NoSQL databases offer alternatives or complements to RDBMS for scenarios requiring horizontal scalability, flexible schema design, and models suited to graphs, documents, or simple key-value structures.

NoSQL vs. RDBMS at a Glance

Essential terms for understanding NoSQL databases at the university level.

The CAP Theorem, formally articulated by Eric Brewer in 2000 and proven by Gilbert and Lynch, is a cornerstone of distributed systems design. It posits that a distributed data store cannot simultaneously guarantee Consistency (C), Availability (A), and Partition Tolerance (P).

Since network partitions are inevitable in any real distributed system, Partition Tolerance is essentially mandatory. This forces a binary choice between C and A when a partition occurs.

PACELC: Extending CAP

The PACELC theorem extends CAP by considering latency vs. consistency trade-offs even when there is no partition: if Partition, choose between A and C; Else, choose between Latency and Consistency.

NoSQL databases are broadly categorized by their fundamental data models, each optimized for different data types, access patterns, and use cases.

Key-value stores are the simplest NoSQL form: data is an opaque value associated with a unique key. The database has no knowledge of the value's internal structure -- it is a simple lookup table at massive scale.

Core Operations

Redis: Session Management Example

# Store session data with 30-minute TTL
SET sess:user:12345 '{"userId":"alice","cartItems":["itemA","itemB"]}'
EXPIRE sess:user:12345 1800

# Retrieve session
GET sess:user:12345

# Atomic increment (leaderboard score)
INCR leaderboard:player:alice
INCRBY leaderboard:player:alice 100

# Sorted set for real-time leaderboard
ZADD leaderboard 1500 "alice"
ZADD leaderboard 1200 "bob"
ZREVRANGE leaderboard 0 9 WITHSCORES   -- top 10 players

# Delete session on logout
DEL sess:user:12345

Consistent Hashing for Distribution

Key-value stores distribute data across nodes using consistent hashing. Keys and nodes are mapped onto a ring, and each key is assigned to the nearest node clockwise. When a node joins or leaves, only nearby keys migrate.

Use Cases

Document databases store data in self-describing documents (typically JSON or BSON) with flexible schemas. Documents can contain nested objects and arrays, naturally mapping to object-oriented application models and eliminating many joins.

MongoDB CRUD Operations

// INSERT -- flexible schema, no prior schema definition required
db.products.insertOne({
  "productId": "PROD001",
  "name": "Super Widget",
  "price": 29.99,
  "tags": ["electronics", "home"],
  "manufacturer": { "name": "WidgetCo", "country": "USA" },
  "stock": 150
});

// FIND -- rich query operators
db.products.find({
  "tags": "electronics",          // array containment query
  "price": { "$lt": 50 },         // range operator
  "manufacturer.country": "USA"   // nested field query
});

// UPDATE -- modify specific fields atomically
db.products.updateOne(
  { "productId": "PROD001" },
  { "$set": { "price": 27.50 }, "$inc": { "stock": -1 } }
);

// AGGREGATION PIPELINE -- multi-stage transformations
db.products.aggregate([
  { "$match": { "tags": "electronics" } },
  { "$group": { "_id": "$manufacturer.country",
                "avgPrice": { "$avg": "$price" },
                "count": { "$sum": 1 } } },
  { "$sort": { "avgPrice": -1 } }
]);

Indexing Strategies

// Single field index -- O(log N) queries on 'category'
db.products.createIndex({ "category": 1 });

// Compound index -- accelerates queries filtering by both fields
db.products.createIndex({ "category": 1, "price": -1 });

// Nested field index
db.products.createIndex({ "manufacturer.country": 1 });

// Text index for full-text search
db.products.createIndex({ "description": "text" });

Column-family stores, inspired by Google's BigTable, organize data into rows accessed by a row key, with each row containing columns grouped into column families. Unlike relational tables, different rows can have entirely different sets of columns within a family -- enabling sparse, wide-column storage ideal for time-series and high write-throughput workloads.

Cassandra: Time-Series IoT Data

-- Table design: composite primary key for time-series
CREATE TABLE sensor_readings (
  device_id    TEXT,
  reading_time TIMESTAMP,
  temperature  FLOAT,
  humidity     FLOAT,
  PRIMARY KEY (device_id, reading_time)
) WITH CLUSTERING ORDER BY (reading_time DESC);

-- INSERT -- appends a new row (fast, append-only log structure)
INSERT INTO sensor_readings (device_id, reading_time, temperature, humidity)
VALUES ('sensor_A', toTimestamp(now()), 23.5, 60.2);

-- RANGE QUERY -- efficient thanks to composite primary key
SELECT * FROM sensor_readings
WHERE device_id = 'sensor_A'
  AND reading_time >= '2024-01-01 08:00:00'
  AND reading_time <= '2024-01-01 09:00:00';

-- Tunable consistency per operation
CONSISTENCY QUORUM;  -- W+R > N guarantees strong consistency
CONSISTENCY ONE;     -- Fastest, eventual consistency

Use Cases

Graph databases make relationships first-class citizens alongside entities. Data is modeled as nodes (entities), edges (relationships), and properties (key-value metadata on both). This enables efficient traversal of deeply connected data that would require expensive multi-way joins in a relational model.

Index-Free Adjacency

The key performance differentiator for graph databases: each node directly stores pointers to its adjacent nodes and edges in memory, rather than relying on an index lookup. Traversal performance is O(1) per hop and proportional to the depth of traversal -- not the total graph size. A relational database would need a self-join for each hop, becoming exponentially more expensive at depth.

Neo4j Cypher Query Language

// Create nodes and relationships
CREATE (alice:Person {name: 'Alice', age: 30})
CREATE (bob:Person {name: 'Bob', age: 25})
CREATE (matrix:Movie {title: 'The Matrix', year: 1999})
CREATE (alice)-[:ACTED_IN {role: 'Neo'}]->(matrix)
CREATE (alice)-[:FRIENDS_WITH {since: 2019}]->(bob)

// Find all friends of Alice
MATCH (a:Person {name: 'Alice'})-[:FRIENDS_WITH]->(f:Person)
RETURN f.name, f.age;

// Find mutual friends of Alice and Bob
MATCH (alice:Person {name: 'Alice'})-[:FRIENDS_WITH]->(mutual)
      <-[:FRIENDS_WITH]-(bob:Person {name: 'Bob'})
RETURN mutual.name;

// Shortest path between two people
MATCH path = shortestPath(
  (alice:Person {name: 'Alice'})-[:FRIENDS_WITH*]-(target:Person {name: 'Charlie'})
)
RETURN length(path), [n IN nodes(path) | n.name] AS route;

// Recommendation: products bought by people who bought what Alice bought
MATCH (alice:Person {name: 'Alice'})-[:BOUGHT]->(prod:Product)
      <-[:BOUGHT]-(other:Person)-[:BOUGHT]->(rec:Product)
WHERE NOT (alice)-[:BOUGHT]->(rec)
RETURN rec.name, count(*) AS score ORDER BY score DESC LIMIT 5;

Use Cases

Consistency in distributed NoSQL systems spans a spectrum from strong guarantees to eventual convergence. Understanding the trade-offs is essential for designing correct distributed applications.

Quorum-Based Consistency: W + R > N

Vector Clocks & Conflict Resolution

Vector clocks track causality in distributed systems: each node maintains a vector of logical timestamps, one per replica. When two vector clocks are incomparable (neither is an ancestor of the other), they represent concurrent updates that conflict and need resolution.

Tunable Consistency Levels (Cassandra)

When should I choose a NoSQL database over a traditional RDBMS?

Choose NoSQL when your application needs horizontal scalability for massive data volumes or very high throughput, a flexible or evolving schema (e.g., IoT sensor data, user-generated content), specialized data models (graphs, documents, key-value), high availability during network partitions, or extremely fast reads or writes. For strong ACID guarantees, complex multi-table joins, or strict referential integrity, RDBMS are generally preferred. Most modern architectures use both in a polyglot persistence approach.

What is the "schema-less" nature of NoSQL, and what are its implications?

NoSQL databases are better described as "schema-flexible" or "schema-on-read" rather than truly schema-less. The database itself does not enforce a rigid schema, so documents in the same collection can have different fields. Pros: faster development, easier data model evolution, flexibility for diverse data. Cons: the application must validate and understand the data structure; lack of database-level enforcement can lead to inconsistent data if not carefully managed.

What is the primary difference between sharding and replication?

Sharding (horizontal partitioning) focuses on scalability by distributing data across multiple independent shards, each holding a unique subset of the total data. Replication focuses on availability and fault tolerance by maintaining multiple identical copies of data across nodes -- if one fails, others serve requests. They are often used together: each shard is typically replicated for high availability.

How does the CAP theorem affect NoSQL database design?

The CAP theorem states a distributed system cannot simultaneously guarantee Consistency, Availability, and Partition Tolerance. Since network partitions are inevitable in distributed systems, designers must choose: CP systems (e.g., HBase, MongoDB default) sacrifice availability during partitions to maintain strong consistency, while AP systems (e.g., Cassandra, DynamoDB) remain available but may return stale data, achieving eventual consistency.

How do I choose the right NoSQL database type for my project?

Consider four factors: (1) Data model -- key-value for simple GET/PUT, document for flexible nested objects, column-family for sparse/time-series high-write workloads, graph for relationship-heavy data. (2) Access patterns -- how will you query? By key, content, range, or relationships? (3) Consistency needs -- can you tolerate eventual consistency or do you need strong consistency? (4) Scale requirements -- do you need linear horizontal scaling? A polyglot persistence approach using multiple database types for different parts of an application is often optimal.

What are the trade-offs of eventual consistency in AP systems?

Eventual consistency means writes propagate asynchronously, so reads may temporarily return stale data. The "inconsistency window" is typically milliseconds to seconds. Applications must handle this gracefully: implement read-your-writes consistency for user-facing operations, use conflict resolution strategies (last-write-wins, CRDTs, application-level merging), and design UIs that tolerate brief inconsistencies. The benefit is higher availability and write throughput -- systems like Cassandra can sustain hundreds of thousands of writes per second.