Disk Space and Inode Usage

Hopper provides a GPFS shared file system for users’ home directories and scratch space. While this file system provides outstanding performance and lots of space, there are limitations in regards to disk space and the number of files.

Some Background

Each unix file consists of two parts: 1) the data blocks that contain the contents of the file and 2) the inode that uniquely identifies the file and contains information about the file. When a file system is created, the data block size is established. This in turn determines the maximum number of files the file system can contain. Once set, you can’t easily increase the number of disk blocks or increase the number of inodes. In other words, there’s a finite limit to both inodes and disk space.

Running Out of Space

It follows that it’s possible to run a file system out of space in two ways :
1. No disk space is left.
2. No inodes are left.

Encountering either of these can crash the entire cluster. User quotas provide the means of protection from both of these events.

User Quotas

Hopper enforces hard and soft limits on quotas for home directories and for the scratch directory. A soft quota can be exceeded for a short period of time called a grace period. The hard quota cannot be exceeded under any circumstances. Here are the quotas for each:

Home Directory:

Space Quota: 2TB
Inode Quota: 9M ( soft ) 9.1M (hard)
Grace Period: 7 days

Scratch Directory:

Space Quota: none
Inode Quota: 10M ( soft ) 11M (hard)
Grace Period: 7 days

Monitoring Usage

It’s a good practice to periodically check your disk space and inode usage. Once you encounter the dreaded “Disk quota exceeded”, it’s too late. Use the checkquota script in /tools/scripts to monitor your usage.

For example, logged in as user abc0001:

Disk space usage for the home directory is 1.555T out of 2T.
Inode usage for the home directory is 5,306,589 out of 9,000,000.
Disk space usage for the scratch directory is 2.585T ( There’s currently no limit to disk space usage on scratch ).
Inode usage for the scratch directory is 4,432,981 out of 10,000,000.

Best Practices

Each user should monitor their usage and clean up as needed as neither home nor scratch are intended as permanent storage.

You may also like...

Leave a Reply