Scratch Folder
Table of Contents
- What is a scratch folder?
- How can I access the scratch folder on Schooner?
- What is the Schooner scratch autodelete policy?
- Why do we have an autodelete policy?
- Where can I put my files?
What is a scratch folder?
Your /scratch folder is a space for temporary files, even large ones, and is ideal for jobs that do a lot of file accesses (reading, writing, etc.) and/or consume a lot of disk space.
You can modify your batch scripts so that the first step is to copy the files to /scratch before processing them.
Once all the processing is done, they can be copied to elsewhere. If you're really done with them, you should consider deleting them from /scratch as well, so that others can use the space.
If you're not done, you can leave them there to be used again.
Please use scratch as TEMPORARY storage only. We do not back up the scratch folder.
While OSCER currently does not enforce quotas on /scratch, we do have a two week deletion policy. See below.
How can I access the scratch folder on Schooner?
After logging into Schooner, you can go to your scratch folder by using the following command:
cd /scratch/yourusername
(replace yourusername with the username you used to log into Schooner).
What is the Schooner scratch autodelete policy?
- When a file is created in your /scratch directory, its clock starts ticking.
- Every file in your /scratch directory will get at least two warning e-mails before being effectively deleted but STILL RECOVERABLE.
- Those warning e-mails will be at least 5 calendar days apart.
- No file will be effectively deleted (but still recoverable) until the later of:
- at least 7 calendar days after the second such warning e-mail
or
- at least 14 calendar days after file creation,
whichever comes later. - Once a file is effectively deleted (but still recoverable), you will receive at least one more warning e-mail about it, at least 5 calendar days after the most recent warning e-mail. That'll be a warning about:
- that file being temporarily eligible to be restored (upon request);
- the file becoming eligible to be physically deleted soon. - No file will be physically deleted until the later of:
- at least 21 calendar days after it was initially created in your /scratch directory
or
- after you have received a total of at least 3 e-mail warnings about it,
whichever comes later. - Once a file is physically deleted, it is GONE FOREVER AND NEVER RECOVERABLE.
- NO OTHER Schooner filesystem is subject to automatic file deletion -- only /scratch.
Why do we have an autodelete policy?
Before we had an automatic deletion policy in place, our /scratch filesystems were typically between 95% and 100% full. That made it extremely difficult for anyone to get any work done -- either they couldn't write files at all, or writing files was very slow.
Our experience has shown that a predictable and recurring scheduled autodelete policy on scratch allows our users to better prepare for their file deletions. This is far better than ad-hoc and irregularly scheduled deletions that are often performed with little notice because the scratch storage has reached critical levels.
Where can I put my files?
Everyone has files that they need to retain for longer than 2 weeks. Where to put them?
- OURRstore
You can archive your files on the OURRstore and download them to /scratch when you need them. Here's what you'll need to do:- Request a OURRstore account (as an extension of your current OSCER account).
- Fill out and submit a OURRstore Use Agreement.
- Do an hour of training.
- Buy tape cartridges, which are incredibly cheap: see the OURRstore webpage.
(We recommend buying enough tape cartridges for dual copies of all your files, in case a tape cartridge breaks -- it doesn't happen often, but it definitely does happen.)
NOTE: Access to the OURRstore is available to researchers and educators at US institutions only.
- OURdisk
You can purchase one or more OURdisk portions to store your files and access them the same as you would on /scratch; see the OURdisk webpage.
