How to handle missing blocks and blocks with corrupt replicas in HDFS?

Posted on In Storage systems, Systems

One of HDFS cluster’s hdfs dfsadmin -report reports:

Under replicated blocks: 139016
Blocks with corrupt replicas: 9
Missing blocks: 0

The “Under replicated blocks” can be re-replicated automatically after some time.

How to handle the missing blocks and blocks with corrupt replicas in HDFS?

Understanding these blocks

A block is “with corrupt replicas” in HDFS if it has at least one corrupt replica along with at least one live replica. As such, a block having corrupt replicas does not indicate unavailable data, but they do indicate an increased chance that data may become unavailable.

If none of a block’s replicas are live, the block is called a missing block by HDFS, not a block with corrupt replicas.

Potential causes

Here are lists of potential causes and actions that you may take to handle the missing or corrupted blocks:

  • HDFS automatically fixes corrupt blocks in the background. A failure of this may indicate a problem with the underlying storage or filesystem of a DataNode. Use the HDFS fsck command to identify which files contain corrupt blocks.
  • Some DataNodes are down and the replicas that are missing blocks are only on those DataNodes.
  • The corrupt/missing blocks are from files with a replication factor of 1. New replicas cannot be created because the only replica of the block is missing.

Possible remedies

Some suggestion as by HDP

  • For critical data, use a replication factor of 3
  • Bring up the failed DataNodes with missing or corrupt blocks.
  • Identify the files associated with the missing or corrupt blocks by running the Hadoop fsck command.
  • Delete the corrupt files and recover them from backup, if it exists.

Reference: HDP doc and Cloudera doc.

Eric Ma

Eric is a systems guy. Eric is interested in building high-performance and scalable distributed systems and related technologies. The views or opinions expressed here are solely Eric's own and do not necessarily represent those of any third parties.

2 comments

  1. Hi. I believe there’s a mistake in the description of corrupt blocks: “A block is called corrupt by HDFS if it has at least one corrupt replica along.” I think that all replicas must be corrupted to be marked as corrupt rather than at least one.

    JIRA HDFS-7281 explains the following:
    1. A block is missing if and only if all DNs of its expected replicas are dead.
    2. A block is corrupted if and only if all its available replicas are corrupted. So if a block has 3 replicas; one of the DN is dead, the other two replicas are corrupted; it will be marked as corrupted.

    1. Hi Jim, yes, the description is not accurate. I fixed it to make it clearer – “A block is “with corrupt replicas” in HDFS if it has at least one corrupt replica along with at least one live replica.”

      It refers to the count from the HDFS report.

Leave a Reply

Your email address will not be published. Required fields are marked *