Lustre Filesystem Quotas

Overview

Three types of quotas can be enforced on our DDN storage system: user, group, and project quotas. Project quotas can effectively be made to work like directory quotas, and due to their conceptual simplicity shall be the most frequently used quota type at our site.

In terms of implementation, we shall treat each user directory as a "project" and assign quota limits to that project (represented internally by a unique ID -- but this can remain transparent to the user). In fact, because a single user may have two separate home directories on our system (one for the CPU cluster, raad2, and the other for the GPU cluster, raad2-gfx) we shall associate both /ddn/home/username and /ddn/gfxhome/username to one and the same project. If we were then to assign a block quota of 500G to the this project, the combined consumption across both directories would have to remain within 500 GB.

Directory-Based Default Quotas

1. Every user shall have a raad2 home directory (e.g. /ddn/home/fachaud74).
2. Every GPU cluster user shall also have a raad2-gfx home directory (e.g. /ddn/gfxhome/fachaud74).
3. The default quota of both home directories combined shall be 500GB.
4. By policy, no exceptions to increase the size of the home directories shall be granted.
5. However, additional space may be granted in a so-called “warehouse” directory under /ddn/warehouse (e.g. /ddn/warehouse/fachaud74).
6. The default quota for all warehouse directories shall be 1TB.
7. If even more space is required in the warehouse directory, higher limits shall be considered based on justification.

Rationale

Extremely large home directories (a) require a long time to back up fully (sometimes longer than a week!) and (b) cost us more money for cloud storage. Both home directories (/ddn/home & /ddn/gfxhome) are backed up fully every 4 weeks, with differential backups being made every other day. Furthermore, a single full backup of each home directory is also stored in the cloud for disaster recovery purposes. Every warehouse directory, on the other hand, shall be backed up fully only once every 2 months, with differentials being made once a week, and -- importantly -- none of the warehouse data shall be copied to the cloud for disaster recovery purposes.

How to Query a Directory Quota

If you already have a warehouse directory you can check it's current usage and quota limit with:

fachaud74@raad2a:~> lfs quota -h -p $(echo "11$(id -u $USER)") /ddn/warehouse/$USER

You can do the same with your home directory limit and usage:

fachaud74@raad2a:~> lfs quota -h -p $(echo "10$(id -u $USER)") $HOME
Disk quotas for prj 107001 (pid 107001):
     Filesystem    used   quota   limit   grace   files   quota   limit   grace
/ddn/home/fachaud74
                     0k      0k      0k       -       0       0       0       -
pid 107001 is using default block quota setting
pid 107001 is using default file quota setting
fachaud74@raad2a:~> 

However, if your home directory quota is not a directory based quota, it will show you the value of 0k for the used, quota, and limit columns of the output (as seen above). In this case, you should use the following instead:

fachaud74@raad2a:~> lfs quota -u $USER /ddn
Disk quotas for usr fachaud74 (uid 7001):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
           /ddn 200765701  524288000 524288000       -  139633       0       0       -
uid 7001 is using default file quota setting
fachaud74@raad2a:~>