New high performance storage for scratch data

Peter Childs Peter Childs Mar 20, 2019 · 2 mins read
New high performance storage for scratch data
Share this

We are pleased to announce a new scratch storage array that is based on fast NvME based hardware. This will hopefully make I/O related tasks much faster

Scratch space is the recommended location for temporarily storing data produced by cluster jobs. We often see jobs failing on the cluster due to users filling their home directory quota with data from job outputs, instead of using scratch space. Some users may be unaware that a P.I. can request a lab share and get the initial 1TB at no cost.

This new service replaces the old legacy /data/scratch service, which we switched off a few weeks ago, in order to re-use the same file paths. By reusing the old paths we hope to make the service as simple to use as possible.

The automatic deletion of old files is based on the same concept we have used in the auto-purging scratch space called /data/autoScratch/weekly/$USER and /data/autoScratch/monthly/$USER. We will be looking to phase these weekly/monthly services out within the next few weeks or months in favour of the faster and simpler /data/scratch/$USER file path. This is partly due to the disks these service are based on reaching end-of-life. All users should start changing their job scripts to use the new service instead. The long-term storage (home directories and lab shares) remains unchanged.

Files placed in /data/scratch are deleted 35 days after last modification. You will be sent warning emails that files are about to be deleted after 31 days (4 days before deletion). These numbers have been selected based on available capacity and observed behaviour. The service has loose capacity-based quotas of 1TB or 1 million files, and requests to extend this will be considered on an individual basis.

Please do not abuse this service by modifying timestamps of your files to make this a permanent location as it is unfair to other users. It is far better to ensure that files are correctly backed up in the right place.

Peter Childs
Written by Peter Childs
Research Storage Manager. Have you done your backups today?