Managing Data

Summary: Managing Data in the Cloud

The lecture surveys how cloud systems manage very large datasets. Cloud data can reside on infrastructure outside a customer’s direct control, raising security concerns. Providers also need to store replicas, support high-volume access, and scale capacity with minimal manual intervention. Traditional relational databases remain useful, but large-scale cloud workloads often require distributed storage and parallel processing.

Relational Databases and Parallel Architectures

• Applications typically query relational database management systems (RDBMSs) with SQL. A query optimizer can change the order of operations—for example, filtering tables before joining them—to reduce execution time.
• Database storage systems manage pages and indexes to reduce disk access. Traditional row-oriented storage suits transaction workloads, while column-oriented storage is often better for data warehouses, where queries aggregate selected columns.
• Parallel databases can use:
  • Shared memory: Processors share memory, commonly in multiprocessor servers.
  • Shared disk: Processors have separate memory but access common storage.
  • Shared nothing: Each server has its own memory and disk and communicates over a network.
• Parallel database features may include partitioning tables across processors, distributed query optimization and joins, transaction coordination, and failover to standby systems. The lecture names Oracle, DB2, and SQL Server for transaction processing, and Vertica, Teradata, and Netezza for data warehousing.

Distributed Cloud Storage

• Google File System (GFS) and the open-source Hadoop Distributed File System (HDFS) are designed for large files distributed across clusters of commodity servers.
• Files are divided into chunks or blocks and replicated across machines—and, in the described design, across physical racks—to improve fault tolerance.
• A master (or HDFS NameNode) tracks file metadata and chunk locations. Chunk servers or DataNodes store the data. Clients consult the master for metadata, then read from a data server.
• For writes and appends, the system coordinates replicas and designates a primary server to determine the write offset. Heartbeats help the master detect failed servers and assign a replacement primary when needed.

Bigtable

Bigtable is a distributed, structured storage system built on GFS. Data is addressed by row key, column key, and timestamp.

Columns are organized into column families. Families are set when a table is created, while labels within a family can be added dynamically. Cells can retain multiple timestamped versions.

Tables are split into ranges called tablets, which are managed by tablet servers. A master and metadata hierarchy track tablets and their locations. The design supports parallel reads and inserts, though inserts that maintain sorted order can require more work than simple appends.

Dynamo and Cloud Datastores

• Amazon Dynamo is presented as a distributed key–value store for high volumes of concurrent updates, particularly useful for web and e-commerce applications. Unlike Bigtable, it does not depend on an underlying distributed file system.
• Dynamo uses hashing to place objects across virtual nodes and replicates objects across physical nodes. Quorum-based coordination helps maintain eventual consistency during concurrent reads and writes, while the system also addresses transient failures and network partitions.
• Google App Engine Datastore and Amazon SimpleDB are introduced as cloud data services based on related scalable-storage ideas. The lecture describes column-family-style storage, horizontal partitioning, indexes, and lexicographically ordered keys that can support prefix and range queries. Index selection helps make different query types more efficient.

Overall, the lecture contrasts established relational and parallel database techniques with distributed file systems and scalable cloud data services, emphasizing their suitability for large datasets and high-volume parallel access.

Reviews, Guides, or Tutorials

No product review or hands-on tutorial is provided; this is an instructional overview of database and cloud-storage concepts.

Main Speaker or Source

An unnamed lecturer presents the course material. The discussion refers to systems and services from Google, Hadoop, and Amazon.