HDFS Commands Every Data Engineer Should Know – From Beginner to Expert

As a data engineer, mastering HDFS (Hadoop Distributed File System) commands is crucial. HDFS is the backbone of big data storage in Hadoop ecosystems, allowing scalable, reliable, and fault-tolerant storage of large datasets. In this blog, I’ll walk you through HDFS commands, starting from basic to expert-level operations.

Basic HDFS Commands

Before diving into advanced operations, it’s important to understand how to perform everyday tasks in HDFS, such as file management, directory structure, and permissions.

1. Checking File System Status

The first command is essential for knowing the health of the HDFS system.

hdfs dfsadmin -report

This provides an overview of the cluster’s status, including the total capacity, free space, and health of the DataNodes.

2. Listing Files and Directories

To list files and directories in HDFS, you’ll use the ls command:

hdfs dfs -ls /path/to/directory

You can add flags like -R for recursive listing or -h for human-readable sizes.

3 Uploading Files to HDFS

Moving local files into HDFS is fundamental:

hdfs dfs -put /local/path/to/file /hdfs/path/to/destination

This command copies the file from the local filesystem into HDFS. Alternatively, you can use -copyFromLocal.

4. Downloading Files from HDFS

Downloading files back to your local system is similarly easy:

hdfs dfs -get /hdfs/path/to/file /local/path/to/destination

Use -copyToLocal as another option

5 Creating Directories in HDFS

Create new directories in HDFS with the following:

hdfs dfs -mkdir /hdfs/path/new_directory

6 Deleting Files and Directories

To remove files or directories, you can use:

hdfs dfs -rm /hdfs/path/to/file
hdfs dfs -rm -r /hdfs/path/to/directory

Use -skipTrash if you want to bypass the trash feature.

Intermediate HDFS Commands

Once you’re familiar with basic file operations, you’ll want to explore more advanced functionalities for better data management and analysis.

7. Viewing File Contents

You can view the contents of a file directly in HDFS using:

hdfs dfs -cat /hdfs/path/to/file

For large files, use -tail to check the last few lines:

hdfs dfs -tail /hdfs/path/to/file

8. Checking Disk Usage

To find out the space usage of a directory in HDFS, use:

hdfs dfs -du -h /hdfs/path

This provides a human-readable breakdown of disk usage within the directory.

9. Moving and Renaming Files

You can move or rename files in HDFS using the -mv command:

hdfs dfs -mv /hdfs/source/file /hdfs/destination/file

This command works similarly to moving files in a regular file system.

10. Setting File Permissions

You can control file and directory permissions in HDFS:

hdfs dfs -chmod 755 /hdfs/path/to/file

Additionally, you can use -chown to change ownership and -chgrp to change the group:

hdfs dfs -chown user:group /hdfs/path/to/file

11. Copying Files Between HDFS Instances

For copying data between HDFS clusters:

hadoop distcp hdfs://source/path hdfs://destination/path

This command is powerful for migrating large datasets across clusters.

Advanced HDFS Commands

These commands require deeper expertise and are generally used for cluster maintenance, diagnostics, and optimization.

12. Balancing HDFS

To rebalance the HDFS cluster when some nodes are under-utilized or over-utilized:

hdfs balancer -threshold 10

The balancer redistributes blocks to ensure the disk space is evenly distributed across DataNodes.

13. Managing Snapshots

Snapshots in HDFS allow you to capture the state of a directory at a certain point in time. To create a snapshot:

hdfs dfs -createSnapshot /hdfs/path/to/directory snapshot_name

You can restore or view files from this snapshot later using:

hdfs dfs -ls /hdfs/path/to/directory/.snapshot/snapshot_name

To delete a snapshot:

hdfs dfs -deleteSnapshot /hdfs/path/to/directory snapshot_name

14 Checking HDFS Block Locations

Knowing where files are stored and their block locations can be crucial for diagnosing issues or optimizing performance:

hdfs fsck /hdfs/path/to/file -files -blocks -locations

This command shows the block size and replication factor of a file, helping you understand where it’s distributed across DataNodes.

15. Corrupt File Detection

In large clusters, file corruption can occur. To check for corrupted files:

hdfs fsck / -listCorruptFileBlocks

It lists all files with corrupt blocks so that you can restore them from backups or replication.

16. Setting Quotas

To manage space and file limits, you can set quotas:

hdfs dfsadmin -setSpaceQuota 100G /hdfs/path
hdfs dfsadmin -setQuota 100000 /hdfs/path

This command ensures directories don’t exceed storage or file count limits.

17. Audit and Data Integrity Check

For integrity checks, use:

hdfs dfs -checksum /hdfs/path/to/file

This generates a checksum, which you can compare against later to verify file integrity.

18. Safemode Operations

HDFS enters safemode for maintenance or recovery. To check or leave safemode:

hdfs dfsadmin -safemode get
hdfs dfsadmin -safemode leave

You can use this to perform critical operations like data recovery or cluster upgrades.

Advanced HDFS Commands for best practices

Here are some key practices:

19. Optimizing Block Size

For large files, increase the block size (default 128MB) to optimize I/O performance:

hdfs dfs -Ddfs.blocksize=256m -put /local/path /hdfs/path

Larger block sizes minimize the overhead associated with managing small files.

20. Efficient Replication Management

Ensure proper data redundancy by managing the replication factor. Set the replication factor based on criticality and access patterns:

hdfs dfs -setrep -w 3 /hdfs/path

You can also set it at the cluster level to avoid manual changes.

21. Archiving and Compression

To reduce storage consumption, use compression with HDFS archiving:

hadoop archive -archiveName archive_name.har /input_dir /output_dir

This compresses multiple small files into a HAR file while retaining efficient access.

Use -skipTrash if you want to bypass the trash feature.

Conclusion

From basic file management to advanced system tuning, these HDFS commands are essential for data engineers at all levels. Mastering them will enhance your ability to work efficiently within Hadoop ecosystems, whether handling everyday tasks or managing large clusters.