OU & Regional Research Store (OURRstore)
The OU & Regional Research Store (OURRstore) is a very large scale tape archive storage instrument being deployed and maintained by the OU Supercomputing Center for Education & Research (OSCER) at the University of Oklahoma (OU).
OURRstore will be in production for 8+ years (through at least March 2029), and will have capacity for over 160 Petabytes of tape (that is, over 160,000 TB).
OURRstore is enabling faculty, staff, postdocs, graduate students and undergraduates at the University of Oklahoma (OU), and at institutions across Oklahoma, as well as in other Great Plains Network (GPN) and Established Program to Stimulate Competitive Research (EPSCoR) states and territories, to build large and growing data collections, and to share these datasets with collaborators and others.
Via an innovative, low-cost business model, researchers buy their own tape cartridges, good for the lifetime of OURRstore (through at least March 2029), and pay zero usage charges (just tape cartridge and shipping costs).
OURRstore is expected to have hundreds of users.
OURRstore is intended for bulk data archiving, meaning small numbers of large files, not for zillions of tiny files. We can help you with that.
Please review OURRstore Policies and Procedures (below) for information regarding usage, best practices, costs and purchase process.
OURRstore is funded by a National Science Foundation (NSF) Major Research Instrumentation (MRI) grant (“Acquisition of a Regional Resource for Long-term Archiving of Large Scale Research Data Collections,” OAC-1828567, $967,755, 9/1/2018 - 8/31/2021, Principal Investigator Henry Neeman).
Interested in in using OURRstore? Contact OSCER at:
support@oscer.ou.edu
Table of Contents
SUMMARY
WHAT IS OURRSTORE'S CURRENT STATUS
WHO IS ELIGIBLE TO BE AN OURRSTORE USER
WHAT KINDS OF DATA ARE ELIGIBLE TO BE ARCHIVED IN OURRSTORE
MEDIA: TAPE CARTRIDGES
HOW TO GET ACCESS TO OURRSTORE
HOW TO GET AN OSCER ACCOUNT
OSCER Account Group Name
HOW TO GET OURRSTORE TRAINING
HOW TO PURCHASE MEDIA
FILE COMPRESSION
FILE SIZES
WRITE SPEED
RETRIEVE (READ) SPEED
REPLICATION FOR RESILIENCY
LIMITATIONS
USABLE FRACTION OF RAW TAPE CAPACITY
PURCHASING MEDIA: DETAILS
HARDWARE AND SOFTWARE SPECIFICATIONS
OURRstore is intended for large scale (bulk) datasets being stored for the long term (several years).
OURRstore’s business model is:
- OU’s NSF Major Research Instrumentation grant (see above) pays for most of the hardware and software, along with initial warranties.
- OU’s Chief Information Officer sponsors space, power, cooling, world-facing network and labor, as well as annual support/maintenance of hardware and software after the initial warranties expire.
- Researchers purchase tape cartridges, at market price, to be placed into OURRstore, and pay shipping costs to get the tape cartridges to OU (and in the case of non-OU users, to ship some of the tape cartridges back to the owner).
Researcher Cost Summary
Current market price is ~$100 minimum buy-in and ~$10 per usable TB, a one-time, up-front purchase, in production for the lifetime of OURRstore (at least March 2029, probably longer).
Researchers pay for tape cartridges and shipping charges ONLY. There are NO OTHER CHARGES.
Tape cartridge purchases are directly from relevant resellers, NOT from OU. We encourage you to ship them directly to OU.
All tape cartridge purchases paid by external funds such as grants are subject to Indirect Costs (IDC).
Data Transfer Method Summary
Files can be uploaded to OURRstore and/or downloaded from OURRstore via Globus (recommended), scp, sftp, bbcp or similar methods.
File Sharing Summary
A file owner can designate — via just a few mouse clicks — any file or folder/directory to be downloadable by any individual user, or by any user group, or by the whole world. (All files and folders/directories are PRIVATE BY DEFAULT.)
Required Tape Format Summary
OURRstore uses Linear Tape Open (LTO) tape cartridges, version 8 for ALL new purchases, and version 7 in "Type M" mode (explained below) for many older purchases. Later, we'll add support for at least version 9 and hopefully version 10, but not yet.
Currently Preferred LTO Version Summary
Currently, the preferred LTO version is LTO-8: 12 TB raw, ~10.2 TB usable per tape cartridge (~85% usable capacity). This is LTO’s lowest price per TB currently on the market.
Market Price Summary
LTO-8 tape cartridges are currently ~$50 each, which corresponds to ~$10 per usable TB for dual copies on separate tape cartridges (REQUIRED for non-OU users, STRONGLY URGED for OU users), not including Indirect Costs (if purchased via external funds such as a grant).
NOTE: Tape cartridges are considered (consumable) materials, NOT EQUIPMENT, so they ARE subject to IDC.
OURRstore Capacity Summary
OURRstore has room for a huge number of tape cartridges (initially ~11,000, with ~6000 more being added), and is expected to house over 160 PB (over 160,000 TB) of research data.
OURRstore’s first 824 tape cartridge slots are claimed by LTO-7 “Type M” tape cartridges holding files from the Oklahoma PetaStore, OU’s previous tape archive.
Of the remaining tape cartridge slots (over 16,000), half are reserved for OU users and the other half are available to all users, both halves on a first come, first served basis.
Prepaid Shipping of Tape Cartridges Containing Secondary Copies (Non-OU Users Only) Summary
Non-OU users are REQUIRED to provide prepaid shipping labels or prepaid shipping boxes for their secondary copies to be shipped back to them at their home institutions.
(OU users’ secondary copies will be stored in an environmentally controlled facility at OU, remote from the tape library that holds their primary copies, so OU users DON’T have to provide prepaid shipping labels or boxes.)
WHAT IS OURRSTORE’S CURRENT STATUS?
OURRstore has been in production since March 2021.
WHO IS ELIGIBLE TO BE AN OURRSTORE USER?
You may use OURRstore if you’re a researcher (non-academic professional, faculty, staff, postdoc, graduate student, undergraduate student, K-12) at any non-commercial research and/or education institution in any of the following categories:
- at U Oklahoma (all campuses);
- at any other non-commercial research and/or education institution in Oklahoma;
- at any non-commercial research and/or education institution in any other Great Plains Network (GPN) full member state (Arkansas, Kansas, Missouri, Nebraska, South Dakota) – this ISN’T limited to full member institutions, but applies to any such institution in these states;
- at any non-commercial research and/or education institution in any other Established Program to Stimulate Competitive Research (EPSCoR) jurisdiction (state or territory).
If you’re at a non-commercial research and/or education institution in a non-GPN/non-EPSCoR jurisdiction in the USA, you CANNOT get free access to OURRstore, but you CAN “buy in” to OURRstore, by purchasing a tape cartridge expansion cabinet and paying its annual support. If you’re interested in that option, please contact:
Please note that, because of data center floor space limitations, there are a VERY LIMITED number of such expansion cabinet opportunities available.
You MUST be an employee of, or a student at, an eligible institution (as described above), to be eligible to use OURRstore.
If you’re at a commercial firm, and/or if you’re an employee of, or a student at, any institution outside the USA, then you CANNOT use OURRstore AT ALL.
WHAT KINDS OF DATA ARE ELIGIBLE TO BE ARCHIVED IN OURRSTORE?
OURRstore is for Science, Technology, Engineering & Mathematics (STEM) research data ONLY, because OURRstore is funded by an NSF grant.
OURRstore is NOT for clinical research data, NOR for non-STEM data, likewise because OURRstore is funded by an NSF grant.
NOTE: The NSF’s definition of STEM includes physical sciences, biosciences, geosciences, engineering, mathematics, technology (for example, computer and information sciences), and social sciences.
OURRstore is intended for NON-CLINICAL STEM RESEARCH DATA that is NOT LEGALLY REGULATED.
Non-STEM data is currently FORBIDDEN on OURRstore, because OURRstore was funded by the NSF, and non-STEM data is outside the NSF’s mandate. (EXCEPTION: All data from the Oklahoma PetaStore is being copied to OURRstore, and that includes non-STEM data.)
Clinical data (for example, clinical studies of human disease) is currently FORBIDDEN on OURRstore, because OURRstore was funded by the NSF, and clinical data is outside the NSF’s mandate.
Legally regulated data (for example, HIPAA, Controlled Unclassified Information, ITAR/EAR) is currently FORBIDDEN on OURRstore, because it’s outside the stipulations in OU’s NSF grant text.
We ALLOW human subjects data that isn’t in any of the forbidden categories listed above. If your files are subject to one or more Institutional Review Board (IRB) agreement(s) governing human subjects research, whether with OU’s IRB and/or with another institution’s IRB, then it’s YOUR RESPONSIBILITY to ensure full compliance with your IRB agreement(s).
If you’re at OU and you have legally regulated data, please contact:
We’re working on a solution for OU users only, but it’s still in early stages.
If you’re at another institution, we DON’T expect to have a solution for you for legally regulated data, NOR for clinical data, NOR for non-STEM data.
OURRstore has room for a very large number of LTO tape cartridges (currently ~11,000 LTO tape cartridge slots total, now being expanded to ~17,000 LTO slots).
Each LTO-8 tape cartridge holds up to 12 TB raw, of which ~85% is usable in practice (~10.2 TB usable).
We strongly urge you to keep at least dual copies of every file, on two different pieces of media – so either buy twice as many tape cartridges, or store the second copy elsewhere than OURRstore (for example, at one of the national supercomputing centers).
NOTE: Non-OU users of OURRstore MUST keep at least dual tape copies, in accordance with the terms of our NSF grant. OU users of OURRstore are STRONGLY URGED to do so.
HOW TO GET ACCESS TO OURRSTORE?
Before you can use OURRstore, you MUST do ALL of the following:
- Get an OSCER account, if you don’t already have one (see below).
- Complete, sign and submit a Use Agreement form, as follows:
By e-mail (STRONGLY PREFERRED): support@oscer.ou.edu
By OU campus mail: Engineering Lab 212
By postal mail: Henry Neeman, Engineering Lab 212, 200 Felgar St, Norman OK 73019
Once we receive your form, we’ll acknowledge it so that you know you can start moving forward.
NOTE: We may have a new Use Agreement form later, in which case you’d need to do that one as well. - Purchase at least 2 tape cartridges and have them shipped to OU (see below). You DON’T need to wait for the tape cartridges you’ve purchased to arrive at OU to complete step 4, below, and then to start using OURRstore.
- Complete an OURRstore training session (~1 hour; see below).
If you don’t yet have an OSCER account, you can request an OSCER account at:
http://www.oscer.ou.edu/account_request/
In your OSCER account request, in your research description, please first state that you plan to use OURRstore.
If you’re from OU, or from another Oklahoma non-commercial research and/or education institution, or you’re actively collaborating on a research and/or education project that has an Oklahoma Principal or Co-Principal Investigator, then you can get a regular OSCER account that includes supercomputer access in addition to OURRstore access.
Otherwise, you’ll get an OURRstore-only OSCER account.
If you AREN’T the Principal Investigator (for example, lead faculty member) of your research team, then please consult with your research team’s PI to find out what research group name you should use.
If you ARE the PI of your research team, please pick a research group name as follows:
- 3 to 12 characters long, lower case letters and digits only, first character a lower case letter.
- DON’T use a research group name based on the name OURRstore, but instead choose a name based on the research your team does.
- Non-OU groups: Please prepend your institution’s abbreviation at the start of your group name. For example:
kublahblah (U Kansas)
ecublahblah (East Central U)
uamsblahblah (U Arkansas Medical Sciences)
NOTE: OU research teams DON’T need to prepend ou to their group name. - For the rest of the characters in your group name (up to 12 characters total, including the prepended institution abbreviation), if you aren’t sure what to name your research team’s group:
- Describe your research team’s topic area in 10 words or fewer.
- Distill that phrase down to the right number of characters.
For example, if you’re at U Nebraska Omaha and your research topic is Colliding black holes then your group name could be something like this:
unocolblkhol
Which perfectly fits in the 12-character limit!
- If you’re the PI of your research team and you want to name your research team based on your name, you can do that. In which case, please use your last/family name, followed by your first/given name or initial, if there’s room. (If you have a long family name, you might have to truncate it and skip the first initial.)
Examples:
umkcleekim
ksuhorvathw
ualrmelendez
uvmfitzgeral
HOW TO GET OURRSTORE TRAINING?
Once your OSCER account has been created, AND we’ve received your completed Use Agreement form, AND you’ve purchased your research group’s media (tape cartridges), THEN you can schedule an OURRstore training session with the OSCER Operations Team, to train you and/or your research team on how to use OURRstore properly.
To schedule your OURRstore training, please contact us at:
The training might be in person, or remotely via videoconferencing.
Some portion of the OURRstore training might be via written documentation and/or recorded video(s).
NOTE: You're allowed to take the training after you have purchased tape cartridges but before those tape cartridges have been delivered to OU.
HOW TO PURCHASE MEDIA (LTO TAPE CARTRIDGES) FOR OURRSTORE?
Purchases MUST be of the approved brand (IBM) and one of the approved model numbers.
For non-OU users of OURRstore, purchases can be from any reseller, subject to whatever rules your home institution imposes. (For example, some institutions require sole source justification or multiple quotes for purchases above a specific threshold dollar amount.)
For OU users of OURRstore, purchases below $5000 can be from any reseller, but purchases $5000 and up are subject to restrictions (see below).
WARNING: DON’T purchase ANY tape cartridges until you’ve consulted us first! Our warranty coverage FORBIDS the use of unapproved tape cartridge brands or model numbers, so any unapproved tape cartridges will be THROWN AWAY, or shipped back to you if you pay for shipping under the terms described below.
We’ll be happy to help you with your LTO tape cartridge purchase. Please contact us at:
for current pricing and further assistance.
Please also note that, because tape cartridges are classified as “materials” instead of “permanent equipment,” then if you’re buying the tape cartridge(s) via external funds (for example, a grant), then your purchase is subject to Indirect Costs (IDC), also known as “Facilities & Administration” or “Overhead.”
For OU Norman external funds, the IDC rate can be found here. (Typically you'll want the “Organized Research” rate.)
For OUHSC external funds, the IDC rate can be found here. (Typically you'll want “Research Grants and Contracts, and Federally Funded Clinical Trials: On Campus Rate”.)
For OU Tulsa, use the IDC rate that your unit is associated with (that is, OU Norman's IDC rate or OUHSC's IDC rate).
Other institutions typically publish their IDC rates on their Vice President/Vice Chancellor/Vice Provost for Research website.
For help with purchasing tape cartridges, please contact us at:
We’ll help you coordinate with the appropriate reseller(s) to get a quote and execute the purchase.
Details about purchasing can be found below.
DON’T BUY TAPE CARTRIDGE BARCODE LABELS
OU IT buys tape cartridge barcode labels in bulk, using our preferred numbering scheme.
So, please DON'T buy barcode labels; if you do, we’ll THROW THEM AWAY.
OURRSTORE IS FOR ARCHIVING, NOT FOR BACKUPS
OURRstore is for ARCHIVING ONLY – NO BACKUPS.
Backup: Frequent “incremental” copying of all files that are new or changed since the last incremental copy (for example, nightly), plus occasional “full dump” copying of all files (for example, weekly or monthly). Files of the same filename can exist in multiple versions, typically going back months (but typically not going back years, for files that have changed recently). Deleting a file from the source eventually (but not immediately) deletes the same file (all versions) from the destination, and if the file owner regrets deleting or changing the file from the source, they can request to have a recent backed up version of the file retrieved from the destination.
Mirror: (Reasonably) synchronous copy of all files. Files of the same filename exist in a single (most recent) version only. Deleting a file from the source (almost) immediately deletes it from the destination.
Archive: “Write once, read seldom.” Files have to be uploaded to the archive by the file owner intentionally (we're working on an automated way to do that, but that'll still require the file owner to choose manually which files to archive). Files of the same filename exist in a single (most recent) version only. Deleting a file from the source has zero effect on the destination copy; deleting the destination copy is a manual task by the file owner (and has zero effect on the source, if it still exists).
Hierarchical Storage Manager (HSM): Files reside on multiple “tiers,” with the most commonly and/or most recently accessed files on the fastest (therefore most expensive, therefore smallest) tier (for example, high speed SSD), and the least commonly and/or least recently accessed files on the slowest (therefore least expensive, therefore largest) tier (for example, tape), with automatic migration from tier to tier.
Using the definitions above, OURRstore is an archive, not a backup nor a mirror nor an HSM.
To save both (a) tape space (and therefore your money) and (b) disk space on the disk you’d retrieve your files to, we strongly urge you to pre-compress all your files.
We can help you learn how to do that. Contact us at:
For example, you can use gzip or Zip to pre-compress your files, before copying them to OURRstore.
OURRstore allows files of size 1 Gigabyte to 1 Terabyte, and recommends 20 GB to 200 GB.
Please note that OURRstore CANNOT be used for small files (smaller than 1 GB per file).
We strongly urge that all files stored on OURRstore be between 20 GB and 200 GB, although in principle you can store files as small as 1 GB or as large as 1 TB.
Each file being retrieved takes 30 to 120 seconds of preparation time (typically ~1 minute) before the relevant tape drive can read or write the relevant portion of the relevant tape cartridge:
- The tape cartridge that was already in the relevant tape drive might need to be rewound or fast forwarded to a different position.
- That tape cartridge is ejected from its tape drive.
- The tape robot collects that tape cartridge and carries it to its tape cartridge slot.
- The tape robot collects, from its tape cartridge slot, the tape cartridge that holds/will hold your file(s).
- The tape robot puts your tape cartridge into the relevant tape drive.
- The tape drive rewinds or fast forwards your tape cartridge to the appropriate position.
Therefore:
1 GB file: ~1 minute prep time (on average) + ~3 seconds read or write time – Prep time dominates!
20 GB file: ~1 minute prep time (on average) + ~1 minute read or write time – Prep time ≅ read time
200 GB file: ~1 minute prep time (on average) + ~11 minutes read or write time – Read time dominates!
1 TB file: ~1 minute prep time (on average) + ~1 hour read or write time – Read time dominates!
So, on the one hand, we don’t recommend files smaller than 20 GB, because far more time will be spent preparing to read or write the file than actually reading or writing it.
But, on the other hand, we don’t recommend files larger than 200 GB, because of long retrieval times (up to ~1 hour).
If you absolutely need to have file(s) of greater than 200 GB, please contact to us about that:
If you have many small files (under 1 GB each), then you can aggregate them into a Zip file or a gzipped tar file (Unix equivalent of a Zip file), in order to achieve the minimum acceptable file size for OURRstore.
We can help you learn how to do that:
If you have individual files of more than 1 GB each, we strongly recommend compressing them via Zip or gzip (or other comparable file compression methods), even if you aren’t aggregating multiple files into a single Zip or tar file.
We can help you learn how to do any of these tasks:
You’ll need to create the Zip file or gzipped tar file on the source disk system where the files are coming from, NOT on OURRstore.
That is, the files that you want to upload to OURRstore should ALREADY be compressed and/or converted to Zip or tar files, before uploading them to OURRstore.
Exception:
In the future, we’ll create a script that will automatically detect files in specific directories that are to be archived, and will then automatically create the appropriate tar or Zip file(s) and do the appropriate compression for you.
If and when that’s available at your institution, that’ll be even better than doing those tasks by hand.
How To Split Files Bigger Than 1 TB
If you have one or more files that are bigger than 1 TB, then you'll have to split them into subfiles that are each smaller than than 1 TB.
The Unix command to do this is:
split -d -b ########### the_filename the_filename.
EXCEPT:
(a) replace ########### with the number of bytes in each split file (expressed below as bytes_per_file),
and
(b) replace the_filename with the name of the file being split.
In the split command, notice the period at the very end of the second filename -- this period is VERY IMPORTANT.
We recommend the following:
number_of_split_files = ceiling(total_file_size / (1000 * 1000 * 1000 * 1000))
bytes_per_file = ceiling(total_file_size / number_of_split_files)
That way, you'll get the right number of split files, and they'll all be roughly the same size, instead of the last one being, for example, very small.
Once you've created the split files, you should upload the split files to OURRstore, NOT the original too-big file.
We’ve benchmarked file store (write) speeds into OURRstore as follows:
- ~X GB/sec from /scratch on OSCER’s current supercomputer, Schooner, for a non-parallel copy (1 TB should be ~X hours);
- ~X GB/sec from Schooner’s /scratch, for a parallel copy (1 TB should be ~X hours);
- ~X GB/sec from OURdisk (1 TB should be ~X hours) – OURdisk is OSCER’s current “condominium” disk offering;
- ~X GB/sec via Globus from a large scale, high performance filesystem outside OSCER;
- ~X GB/sec via sftp or scp from a large scale, high performance filesystem outside OSCER.
Note that uploading a file from a laptop/desktop PC will be slower than the above, as will uploading a file from a handheld (phone, tablet) – the bottleneck will be the PC or handheld.
NOTE: Because file uploads go to OURRstore’s disk instead of directly to OURRstore’s tape, your file uploads won’t spend any time waiting for your store-to-tape operation to get started; instead, that happens invisibly behind the scenes after your file upload completes, typically within ~X hours.
We anticipate file retrieval (read) speeds to be roughly the following:
- time spent waiting for a tape drive to become available, which will depend entirely on how many other file transfers are ongoing or waiting to begin,
PLUS
- time spent on preparation before reading (see above),
PLUS
- read time (see above),
PLUS
- copying your file(s) back to wherever you actually want to download the file to (see above).
Thus, other than time spent waiting for a tape drive to become available, total retrieval time should be:
- 1 GB file: ~1 minute;
- 20 GB file: ~2 minutes;
- 200 GB file: ~12 minutes;
- 1000 GB file: ~1 hour.
Note that, at least for the near future, your time spent waiting for a retrieval to get started should be very modest in most cases.
This is because OURRstore has 6 LTO-8 tape drives and gives priority to file retrievals (reads) when there are a mix of stores and retrieves waiting to get started.
Tape cartridges can break (and have done so). Therefore, you need MULTIPLE copies of your archived files, on MULTIPLE distinct tape cartridges.
You can choose which of the following replication policies to use, on a per-directory basis.
Non-OU users:
You’ll be able to store your files under either of the following replication policies:
- tape_2copies: One copy inside OURRstore’s tape library, one copy shipped back to you, using the prepaid shipping label that you provided us.
The disadvantage of tape_2copies is, if the primary copy in the OURRstore tape library fails (for example, if the tape breaks), then you’d have to ship your secondary copy to OU, which might be the ONLY REMAINING COPY OF THOSE FILES IN THE WORLD, so you’d be taking a BIG RISK in shipping it back to us.
- tape_3copies: One copy inside OURRstore’s tape library, one copy shipped back to you, and one copy carried by us to another OU campus to an environmentally controlled facility for offsite storage.
This costs 50% more than tape_2copies, but if the primary copy inside the OURRstore tape library fails, we can quickly restore your files from the tertiary copy located elsewhere at OU, instead of you having to ship us your secondary copy, so your RISK IS REDUCED A LOT.
OU users:
You’ll be able to store your files under either of the following replication policies:
- tape_2copies: One copy inside OURRstore’s tape library, one copy carried by us to another OU campus, to an environmentally controlled facility for offsite storage – we expect this to be the MOST POPULAR REPLICATION POLICY BY FAR for OU users.
- tape_3copies: One copy inside OURRstore’s tape library, two copies carried by us to another OU campus, to an environmentally controlled facility for offsite storage, so this costs 50% more than tape_2copies, with no major advantage other than ensuring that there are at least 3 copies total.
OU users only will have additional options, which are available ON REQUEST ONLY:
- disk_1copy_tape_1copy, if you’ve purchased disk capacity (EXPENSIVE!);
- disk_1copy_unsafe, if you’ve purchased disk capacity (EXPENSIVE!), for files that are available at other repositories (for example, at a national data repository);
- tape_1copy_unsafe, for files that are also available at other repositories (for example, at a national data repository).
NOTE to OU users: You can only choose a particular replication policy if you’ve purchased sufficient appropriate media (tape cartridges and/or disk drives) for it.
For example, if you only own tape cartridges but not disk drives, then you can choose options that involve tape only, not options that involve both tape and disk, nor options that involve disk only.
If you owned disk on the Oklahoma PetaStore, that disk ownership ENDED when the PetaStore was decommissioned, so you’d have to buy new disk for OURRstore.
Please note that we DON’T allow multiple copies on disk, because of the severe scarcity and limited expandability of disk capacity.
Please carefully read the Use Agreement form to understand the limitations of using OURRstore, including the kinds of files that CANNOT be stored on OURRstore, who CANNOT use OURRstore, and YOUR RESPONSIBILITIES when using OURRstore.
3, 2, 1 RULE
The 3, 2, 1 Rule recommends storing your files as follows:
- at least 3 copies of each file;
- at least 2 different media types (for example, disk and tape);
- at least 1 copy remote from the others.
If you want to follow the 3, 2, 1 Rule, you can achieve that via a combination of (a) OURRstore and (b) local disk at your institution.
Examples:
Non-OU Users: If you upload a file to OURRstore from disk at your local institution (for example, on a PC or on your local supercomputer), and you leave the file on disk at your local institution, then you’ll have one copy on your local institution’s disk, one copy inside OURRstore’s tape library at OU, and one copy shipped back to you at your local institution. So you’ll have 3 copies total, on two different media types (disk, tape), with one copy remote (at OU inside OURRstore).
OU Users: If you upload a file to OURRstore from any of (a) your local disk (for example, on your own PC), or (b) OSCER’s supercomputer, or (c) OURcloud, or (d) OURdisk, and you leave the file on disk, then you’ll have one copy on disk, one copy inside OURRstore’s tape library at OU Norman, and one copy in an environmentally controlled facility at OU Health Sciences Center in Oklahoma City (~24 miles from the OURRstore tape library). So you’ll have 3 copies total, on two different media types (disk, tape), with one copy remote (at a different OU campus).
NOTE: Not every research project needs to follow the 3, 2, 1 Rule, and there aren’t good rules of thumb for deciding whether your research project does need to, so use your project’s Data Management Plan (if you have one) as a guide.
USABLE FRACTION OF RAW TAPE CAPACITY: ~85%
LTO tape cartridge capacities are as follows:
- LTO-7:
6 TB raw, ~5.1 TB usable (85%) – NOT OFFERED (Why waste space?) - LTO-7 Type M:
9 TB raw, ~7.6 TB usable (85%) – NO NEW PURCHASES
- LTO-8:
12 TB raw, ~10.2 TB usable (85%) – currently the best $/TB
- LTO-9:
18 TB raw, ~15.3 TB usable (85%) – breakeven $/TB c. 2025
We recommend budgeting as if each LTO tape cartridge has 85% of the raw space available to use, in case of the following issues:
- no files are available that are small enough to fit in the last 15% of the tape cartridge;
- contingency, because we can’t guarantee that any particular tape cartridge will be 100% filled;
- unanticipated growth of your datasets.
In addition, you’ll want to think about whether your files will, over the long term, reside ONLY on OURRstore, or ALSO on some other long-term archive.
If these files will be stored ONLY on OURRstore, then we urge you to keep at least dual copies of all files on separate tape cartridges, which doubles the number of tape cartridges. (Non-OU users of OURRstore are REQUIRED to do this, by the terms of the OURRstore NSF grant.)
LTO-7 Type M (NO NEW PURCHASES)
LTO-7 Type M is just a regular LTO-7 tape cartridge, but formatted in a special way, using a special barcode label (which OU provides at no charge), and written/read by an LTO-8 tape drive (which OURRstore has several of).
So LTO-7 Type M at 9 TB raw/~7.65 TB usable has IDENTICAL PRICE to regular LTO-7 at 6 TB raw/~5.1 TB usable.
Because LTO-8 is now less expensive per usable TB, and consumes fewer tape cartridge slots per TB, OURRstore NO LONGER ALLOWS NEW LTO-7 Type M tape cartridges (extant ones will be kept in production for OURRstore's lifetime), and FORBIDS regular LTO-7.
Indirect Costs
Tape cartridges purchased on external funds such as a grant typically ARE subject to Indirect Costs (also known as “Facilities & Administration” or “Overhead”), because they’re considered consumable materials, not “permanent equipment.”
OU users: OU’s current IDC rate can be found here. Click on Download signed original agreement (PDF) and look for the current On Campus Organized Research rate (or whichever rate category is appropriate for your funding, if you know that it’s a different rate category).
Non-OU users: Other institutions’ IDC rates are typically found on the website of their local Vice President/Vice Chancellor/Vice Provost for Research.
Tape Cartridge Make and Models Allowed in OURRstore
For new tape cartridge purchases, OURRstore is allowed to have ONLY IBM-branded LTO tape cartridges, of ONLY the following model:
- IBM 01PL041 [LTO-8], NEW only
Extant tape cartridges in OURRstore, including LTO-7 and LTO-8, will be kept in production for the lifetime of OURRstore (at least 2029).
NEW tape cartridges only -- NO USED, NO RE-CERTIFIED
Any other tape cartridge brands, or any other IBM model numbers, are ABSOLUTELY FORBIDDEN in OURRstore, because they would risk voiding our warranty/support/maintenance agreements.
Some tape cartridge resellers also sell multi-packs (usually 5-packs and/or 20-packs), sometimes at a lower price per tape cartridge than buying large numbers of individual cartridges. If you want to purchase such multi-packs, you MUST confirm with the reseller, in writing, that the individual tape cartridges in the multi-pack are one of the above model numbers.
Shipping Address
ALL OURRstore tape cartridges, for both primary and secondary copies, should be shipped to:
Shane Smith, OURRstore, YOURGIVENNAME YOURFAMILYNAME
301 David L. Boren Blvd. Loading Dock 1028
Norman OK 73072
405-255-4308
smsmith@ou.edu
(but replacing YOURGIVENNAME with your first/given name and YOURFAMILYNAME with your last/family name).
By putting your name on the first line of the shipping address, we’ll know who these tape cartridges belong to (we receive many such shipments).
If you’re using a quote for the tape cartridges, please have the reseller put exactly the same shipping address in the quote, because some resellers only look at the shipping address in the quote, instead of the shipping address in the Purchase Order.
Tape Cartridge Resellers
You can purchase these tape cartridges from ANY RESELLER you want, subject to the caveat below for OU users.
Using your favorite search engine (for example, Google, DuckDuckGo, Bing, etc), search for:
IBM 01PL041
But, although LTO-9 tape cartridges are available, not only are they too expensive per TB, OURRstore also won’t be able to support them yet, because we don’t yet have LTO-9 tape drives to read and write LTO-9 tape cartridges.
From the web search above, you’ll see that there are several resellers of LTO tape cartridges.
OU users:
Tape cartridge purchases should be executed by your department's/unit's financial lead, or by the Shared Business Services Center (sbsc@ou.edu) if that's what your department/unit uses.
If you’re buying tape cartridges that total less than $5000, your department’s/unit’s financial lead can do a “Pcard” purchase. You DON’T need multiple quotes for that, nor a quote – your unit’s financial lead can purchase directly from a reseller’s website if they want.
If you’re buying enough tape cartridges to hit $5000 or more in a single purchase, then:
- You MUST get the required number of quotes, from different resellers, all for the same brand, model and quantity of tape cartridges. The number of quotes can be found on the OU Procurement Policies and Procedures webpage -- click on Informal Competition Policy. OSCER can help with this: support@oscer.ou.edu
- You MUST pick the least expensive quote, unless you have a compelling reason to do otherwise.
- Your department’s/unit’s financial lead must enter the purchase into OU’s internal purchasing system, “PeopleSoft Marketplace,” using the shipping address above and your relevant OU account (for example, a grant account).
- After your department’s/unit’s financial lead has submitted the purchase, OU Procurement can send a Purchase Order to the selected reseller.
Once tape cartridges are purchased, there AREN’T any recurring charges (for example, NO monthly service charge).
But, at OURRstore’s end-of-life, you might need to buy new tape cartridges (though we hope that won’t be the case).
Based on the current rate of tape cartridge capacity increase (typically doubling roughly every 2½ years), and assuming a 3% inflation rate per year and no change in the IDC rate, the cost in 8 years will be ~15% of today’s cost for the same usable capacity.
Obviously, there’s no way to know (let alone guarantee) that these rates will remain constant, so consider these numbers to be best guess estimates.
DON’T Buy Barcode Labels
OU IT provides barcode labels at no charge, so that our barcodes have the correct format and the correct numbering order.
Please DON’T buy your own barcode labels, regardless of which institution you’re at.
Pre-formatted Tape Cartridges
Please DON’T pay for pre-formatting of tape cartridges, unless you’ve contacted us first:
We’ll need to confirm with the tape cartridge reseller that the pre-formatting is of the correct type.
NON-OU USERS: PREPAID SHIPPING OF SECONDARY CARTRIDGES TO YOU
Non-OU users MUST provide prepaid shipping boxes and/or labels to OU, to ship your secondary tape cartridges back to you, preferably at your home institution.
(OU users DON’T need to do this.)
Shipping Weight and Dimensions for LTO Tape Cartridges
Without plastic “jewel cases”:
- LTO tape cartridge shipping weight, without jewel case, each: 0.441 lbs / 200 g
- LTO tape cartridge size, without jewel case, each: 4 in × 4.1 in × 0.8 in / 10.2 cm × 10.5 cm × 2.1 cm
With “jewel cases”:
- LTO tape cartridge shipping weight, with jewel case, each: 0.6 lbs / 270 g
- LTO tape cartridge size, with jewel case, each: 4 3/8 in × 4 1/2 in × 1 1/8 in / 11.1 cm × 11.4 cm × 2.9 cm
Prepaid Return Shipping Boxes or Shipping Labels for Non-OU Users
Non-OU users should please purchase one or more prepaid return shipping boxes or labels for the tape cartridges that contain secondary copies of files, which will be shipped back to you, preferably at your home institution.
Those can be sent to us physically (prepaid cardboard boxes and/or prepaid paper shipping labels), or electronically (for example, prepaid PDF shipping labels).
If you have to send us cardboard or paper instead of electronic, please alert us about that, so that we know to look for it.
Alternatively, you can wait until we’re ready to send you the tape cartridges, but the risk there is that your grant will expire (or run out of money) before you make the prepaid shipping box or label purchase.
You’re allowed to use any shipping carrier you want that can:
- provide a prepaid cardboard shipping box or prepaid paper or electronic shipping label (PDF preferred for electronic),
AND - schedule a pickup from the Four Partners Place data center at OU’s Norman campus, using the same address as below (this must be scheduled with the OSCER Operations Team).
NOTE: OU WON’T pay for shipping boxes and/or shipping labels of tape cartridges for ANY non-OU user, NOR will OU transport tape cartridges to a shipping carrier’s office or dropoff location, even if that shipping carrier has a dropoff location elsewhere on the OU Norman campus – we are NOT A TAXI SERVICE.
US Postal Service Prepaid Forever Priority Mail
An example of a service that meets our criteria is the US Postal Service’s Prepaid Forever Priority Mail (2-day), with the following pricing (as of Nov 18 2023):
Max Qty (tapes) Price per box Prepaid Forever Flat Rate Box Type
1 $10.20 Small (8 11/16 inch × 5 7/16 inch × 1 3/4 inch)
14 $17.10 Medium #2 (11 inch × 8 1/2 inch × 5 1/2 inch)
20 $22.80 Large (12 1/4 inch × 12 inch × 6 inch)
The US Postal Service will only sell these boxes in minimum lots of 5 of the same box size, so the minimum total shipping cost per box purchase is $51.00 for 5 × Small, or $85.50 for 5 × Medium, or $159.60 for 5 × Large.
Thus, the shipping price per tape cartridge varies from $10.20 each to $0.63 each, depending on how many are being shipped at a time.
If you expect to ever have at most 5 tape cartridges for OU to ship back to you, then you may be better served buying a 5-pack of Small boxes and having us ship your tape cartridges one per box, because the cheapest possible purchase is 5 Small boxes at $51.00 total.
If you expect to continue buying tape cartridges over OURRstore’s 8+ year lifetime, then you should buy 5 Large or 5 Medium boxes at a time, as needed.
(DON’T buy US Postal Service prepaid envelopes, only prepaid boxes, because we can’t put any padding in envelopes.)
The US Postal Service requires you to provide a return address for your Prepaid Forever Flat Rate Boxes, which in this case would be OU (which will use these boxes for shipping your tape cartridges to you). Please use:
Shane Smith
University of Oklahoma OURRstore YOURGIVENNAME YOURFAMILYNAME
301 DAVID L BOREN BLVD
STE 1000
NORMAN, OK 73072-7341
(Except replace YOURGIVENNAME with your given/first name, and replace YOURFAMILYNAME with your family/last name. And NO PUNCTUATION in the line that says University of Oklahoma. If you run out of room on that line, you may shorten the university’s name to “U Oklahoma,” or drop OURRstore, or shorten your name.)
That will cause the US Postal Service to ship these empty boxes (with prepaid shipping labels on them) to OU.
Then, e-mail us how many boxes of which sizes you’ve shipped to us, along with the address to ship your secondary tape cartridges to (preferably at your home institution).
Note that this return address is DIFFERENT from the address to have your tape cartridge reseller ship the tape cartridges to, above. (We tested the other shipping address, the one for buying tape cartridges, but the US Postal Service website won’t accept it, only this return address.)
If you want to use another carrier, you’ll need to have that carrier deliver their packing boxes to OU, and to provide OU with a prepaid shipping label or prepaid shipping box, and to schedule a pickup at OU, all at no charge to OU.
NOTE: In the context of OU shipping tape cartridges, the quantity of tape cartridges is the quantity to be shipped back to your institution (that is, either one half or one third of the total quantity of tape cartridges you’ve purchased, depending on whether you’re using tape_2copies or tape_3copies, respectively).
HARDWARE AND SOFTWARE SPECIFICATIONS
Tape Library: IBM TS4500
Maximum capacity, usable, as currently configured: ~100 PB (LTO-7 Type M and LTO-8, 10,940 tape cartridge slots in 9 frames)
Maximum capacity, usable, being deployed soon: ~160 PB (LTO-7 Type M, LTO-8, LTO-9 and probably LTO-10, 16,950 tape cartridge slots in 9 frames)
Current slot capacity: 10,940 tape cartridge slots, 28 tape drive slots
Tape drives, current: 6 × LTO-8 (can read and write LTO-8, LTO-7 Type M, LTO-7)
Tape drives, future: 4 to 6 × LTO-9 or LTO-10 (can read and write LTO-9, maybe LTO-10)
Frames, Norman (in production): 1 controller frame (L55), 1 drive/cartridge expansion frame (D55), 7 cartridge-only expansion frames (S55), might add additional S55 frame(s) and/or up to 2 S54 expansion frames upconverted from OSCER’s old IBM TS3500 tape library
Frames, OUHSC (coming): 1 controller frame (L55), 4 cartridge-only expansion frames (S55), might add additional S55 frame(s)
Tape cartridge format, still in production: LTO-7 Type M (in production for OURRstore's lifetime)
Tape cartridge format, current: LTO-8
Tape cartridge formats, future: LTO-9, MAYBE LTO-10
Tape cartridge capacity, each, uncompressed:
- LTO-7 Type M:
each 9 TB raw, ~7.65 TB usable - LTO-8:
each 12 TB raw, ~10.2 TB usable - LTO-9:
each 18 TB raw, ~15.3 TB usable - LTO-10:
each 36 TB raw, ~30.6 TB usable – MAYBE
Tape drive speed, uncompressed
- LTO-8 drive for LTO-7 Type M cartridge:
300 MB/sec peak per LTO-8 drive - LTO-8 drive for LTO-8 cartridge:
360 MB/sec peak per LTO-8 drive
- LTO-9 drive for LTO-9 cartridge:
400 MB/sec peak per LTO-9 drive
- LTO-10 drive for LTO-10 cartridge:
UNKNOWN – MAYBE
Disk Subsystem (as front-end “landing pad” for files coming on and off tape): IBM FlashSystem 5030
- Disk drive format: Nearline SAS 7200 RPM
- Disk drive capacity, each, raw: 14 TB
- Current number of drives: 60
- Current capacity: 456 TB usable
- Enclosures: One Large Form Factor (3.5-inch) Control Enclosure (12 drive bays), 4 LFF Expansion Enclosures (12 drive bays each)
- Network, Fibre Channel: 8 ports 16 Gbps
Servers, Tape Control: IBM LC922 model 22P
- CPUs: dual POWER9 16-core
- RAM: 128 GB (8 × 16 GB) DDR4
- Disk: quad Nearline SAS 2 TB 7200 RPM
- Network, Ethernet: Intel XL710 10GE SFP+ quad-port with transceivers
- Network, Fibre Channel: QLogic QLE2692SR dual-port 16 Gbps
- Operating System: Red Hat Enterprise Linux
Servers, Disk Control: Lenovo ThinkSystem SR650
- CPUs: Intel Xeon Cascade Lake 4214 12-core 2.2 GHz
- RAM: 192 GB (12 × 16 GB), DDR4 2933 MHz
- Disk: dual SAS SSD 800 GB PM1645a
- Network, Ethernet: Lenovo 10GE SFP+ quad-port with dual transceivers, Intel i350 GigE dual-port
- Network, Fibre Channel: QLogic QLE2692SR dual-port 16 Gbps
- Operating System: Red Hat Enterprise Linux
Switches, Fibre Channel: IBM SAN24B-6
- 24 Fibre Channel 32 Gbps ports per switch, 11 Fibre Channel 16 Gbps transceivers per switch
Switches, Ethernet:
- Dell S4128F-ON: 2 switches, each 28 10GE SFP+ ports, 2 100GE QSFP28 ports, connected/failover via Virtual Link Trunking
- Dell N3048ET-ON: 2 switches, each 48 GigE RJ45 ports, 2 10GE SFP+ ports, connected/failover via stacking
Software, Tape Control: IBM Spectrum Archive
Software, Disk Control: IBM Spectrum Scale (formerly GPFS) Standard Edition
INTERESTED IN USING OURRSTORE?
Contact OSCER at: support@oscer.ou.edu
