HDFS (Hadoop Distributed File System)
HDFS is a distributed file system called Hadoop Distributed File System. It is based on the master-slave architecture. An HDFS cluster consists of single name node called Master Node that manages the file system namespace and regulates the access to files by clients, and multiple data nodes called Saves usually one per node in the cluster, which manage the storage attached to the nodes that they run on.
Internally, a file is split into one or more blocks and these blocks are stored in a set of Data Nodes. The Name Node executes the file system namespace operations like opening, closing, and renaming files and directories. It also determines the mapping of blocks to Data Nodes.
The Data Nodes are responsible for serving read and write requests from the file system’s clients. The Data Nodes also perform block creation, deletion, and replication upon instruction from the Name Node.
HDFS Concepts
![HDFS Concepts]()
Name Node
- Manages the process of replication and distribution.
- Keeps track of block location and complete file system.
- Smart replication saves system from failure and provides resilience.
- Provides the data from replicas of name node if any of the node or rack fails.
Data Node
- Physically stores the data in files.
HDFS Ideas
- Handles very large file.
- 100s of MB, GB, TB and above.
- Data Access.
- Writes once and reads many times.
- Commodity hardware.
- Price ranged between $1,000 to $5,000. Not like in RDBMS $50,000.
HDFS is not designed to,
- Low-latency of data access (cannot read data quickly).
- Doesn’t fit OLTD database (needs RDBMS).
- Small files.
- More the number of small files, more the memory of name node be taken up.
- Arbitrary file modifications.
- Only appends are supported.
HDFS Blocks
- Block size is the minimum size of the data that can be read or written to a file system.
- Typically, for a disk, it is 512 bytes, but for HDFS, its default is 64 MB.
- A file smaller than a block size doesn’t occupy a full block size in HDFS.
Why the block size is so large
- Seek time is around 10ms.
- At transfer rate 100mbps, seek time is around 1% of the transfer time.
![HDFS]()
HDFS Architecture
![HDFS]()
Fig. HDFS Architecture
- HDFS is meant for very large files and won’t work efficiently with lots of small files.
- Write once, read many times is the principle. However, appends are supported.
- HDFS works on master-slave architecture. The name node is the master and Data nodes are workers/ slave nodes.
- Name node maintains namespace images and edit-log in the main memory.
- Failure of name node leads to a complete system failure called Single Point Of Failure (SPOF). Hence, its hardware has to be more resilient. Still, with updates, the hardware failures happen, so to counter those situations, resilience additions are,
- Remote NFS mount.
- Secondary Name Node.
- Secondary name node timely combines the edit log and namespace so as to avoid the name node to run out of memory.
- Specification of secondary name node should be equivalent to the primary name node.
- Name node is responsible for timely updating the namespace image and editing the log to local disk and on a remote network storage location.
- In case the failure of the last information from NFS mount is retrieved manually, the administrator to machine (secondary name node) configures it as name node. It takes approx. 30 minutes to get stable.
![HDFS]()