Distributed File Systems

How GFS, HDFS, Ceph and object stores such as S3 keep huge amounts of data on many ordinary machines: large blocks, one metadata service, copies or erasure codes across racks, and what each design gives up.

Overview

A distributed file system keeps files on many ordinary servers and still lets programs use them like one big disk, with directories, file names and byte offsets. It exists for data that is too big for one machine, or that must outlive the machines it sits on: logs, crawled web pages, analytics tables, machine learning training data and original video uploads.

This chapter starts with the Google File System (GFS), because HDFS and other later systems reuse its ideas, and then covers HDFS, the Hadoop file system that follows the same plan. Next come the three questions every design has to answer: where the copies go, whether to copy data or erasure-code it, and what to do about many small files. It ends with object storage, such as Amazon S3, and with Ceph, which finds data by computing where it lives instead of looking it up.

What you will be able to do

  • Explain how a client reads and writes a file in GFS and HDFS, and why file data never passes through the metadata server.
  • Say where HDFS places the three copies of a block, and what that protects against.
  • Keep the NameNode from being a single point of failure, and say what still limits it.
  • Compare replication with erasure coding using real overhead and repair numbers.
  • Explain how an object store such as S3 differs from a file system, and choose between them.

Key concepts

  • GFS
  • NameNode and DataNodes
  • Blocks and chunks
  • The write pipeline
  • Rack awareness
  • NameNode high availability
  • Erasure coding
  • The small-files problem
  • Object storage and S3
  • Ceph and CRUSH

Interview questions

  • Walk through how a client reads a file in HDFS or GFS.
  • How does a write work in HDFS, and what happens if a DataNode fails halfway?
  • Why are blocks 64 MB or 128 MB instead of a few KB?
  • How do you keep the NameNode from being a single point of failure?
  • Where does HDFS put the three copies of a block, and why?
  • When would you use erasure coding instead of replication? Show the numbers.
  • What is the small-files problem, and how do you fix it?
  • How is Amazon S3 different from a file system?
  • How does Ceph find an object without asking a metadata server?
  • Would you build a new analytics platform on HDFS or on object storage?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio