HDFS is hadoop distributed file system; in simple terms a file is stored in a distributed machines. The Hadoop framework was designed considering reliability, throughput, network I/O, and disk I/O; but compromised with latency, which is best in RDBMS.
These points are discussed in part I ie; HDFS Architecture HDFS Architecture .
How Read and Write operations are performed in HDFS
HDFS Write
![HDFS Architecture]()
- By default the replication factor(multiple copiesof blocks) for a block is 3.
- As Name Node receives write request from HDFS client (JVM), Name Node checks whether file is available or not as well as whether client is authorized or not (performs various checks) and returns multiple nodes.
- Step 3,4 and 5 will get repeated until the whole file gets written on HDFS.
- In case of Data Node failure-
- The data is written on the remaining two nodes.
- Name node notices under replication and arranges for replication.
- Same is the case with multiple node failure.
HDFS Write- Selection of the Data Nodes
How Data Nodes are selected by Name Nodes
![HDFS Architecture]()
- Any node within the cluster is chosen as the first node but it should not be too busy or overloaded.
- Second node is chosen as the first node is chosen.
- Third node is chosen to be on the same rack as the second one.This forms the pipeline.
Simulation on block distribution
- File is broken into blocks (64 mb) and then replicated and distributed across the file system.
![HDFS Architecture]()
- If one of the node/rack fails then also the replication of ( ) that block is available on other racks .
- Failure of multiple racks are more serious but less probable.
- Also, the whole procedure of selection and replication happens behind a curtain, no developer or client is able to see all this or has to worry about what happens in the background.
Node Distance
How is distance calculated in HDFS?
Idea of distance is based on bandwidth.
![HDFS Architecture]()
The only possible cases for calculating distance are-
- D=0, blocks on same node, same rack.
- D=2, blocks on different node,same rack.
- D=4, blocks on a node having different rack.
- D=6, blocks on a node having different data centre.
HDFS Read
![HDFS Architecture]()
If D7 fails, next D8 is picked
Failure cases,
- Data block is corrupted,
- Next node in the list is picked up.
- Data Node fails,
- Next node in the list is picked up.
- That node is not tried for the later blocks.