IRADS Frequently Asked Questions
What is NEXRAD?
NEXRAD is an acronym that stands for NEXt Generation RADar, but more formally refers to the "Weather Surveillance Radar - 1988, Doppler," also known as the WSR-88D. The NEXRAD radar network comprises 142 operational WSR-88D radars in the continental United States, 120 of which are managed by the National Weather Service with 21 managed by the Department of Defense. An additional 4 and 7 radars are located in Hawaii and Alaska, respectively, managed by the Department of Transportation, with radars also located in Puerto Rico, Guam, South Korea, Japan and the Azores. For more information, visit the Radar Operations Center. Thus, a total of 158 operational radars are deployed in the United States and at selected overseas locations. Eight research and development radars are used for a variety of purposes, with the dual-polarization NEXRAD test bed located in Norman, Oklahoma.
What is Level II data?
Level II data are the digital radial base data (Reflectivity, Mean Radial Velocity, and Spectrum Width) and Dual Polarization variables (Differential Reflectivity, Correlation Coefficient, and Differential Phase) output from the signal processor in the Radar Data Acquisition unit. The output also includes status information required to properly interpret the data (e.g., information on synchronization, calibration, date, time, antenna position, Nyquist velocity, and operational mode). Level II data are used for many purposes including: support of operations, maintenance and developmental activities at the ROC; activities directed toward algorithm and product improvement; research by universities; and private sector uses/applications. The Level II data are sent to IRADS by the NWS WSR-88D Level II Central Collection, Distribution and Archive Network. Level II data are continuously transmitted and archived from all DOC and select DoD and DOT WSR-88D systems. The interface control document describing the Level II format. ICD for Archive II/User – Current build information here.
Why would I want to receive Level II data?
Level II data are unique among ground-based atmospheric observations because they provide the highest spatial and temporal resolution information available within three-dimensional volumes on a continuous basis, regardless of the weather present. Level II data are used for real time warning of hazardous spring and winter weather, for initializing numerical weather prediction models, for verifying the occurrence of past events, such as the location of damaging hail, and for scientific and educational purposes. Level II data also are used for non-meteorological purposes, including bird and insect migration studies, bird strike avoidance, urban pollution transport, and the tracking of hazardous atmospheric releases. Level II data collected by the University of Oklahoma, as part of Project CRAFT (pdf) was instrumental in tracking debris from the Shuttle Columbia disaster.
What software do I need?
A public-domain software package, known as the Local Data Manager (LDM), is required to ingest streaming Level II data. LDM was developed, and continues to be enhanced, by the Unidata Program at the University Corporation for Atmospheric Research in Boulder, Colorado (Unidata was a CRAFT partner), and is highly configurable and scalable. LDM has the ability to ingest other types of data as well. IRADS provides free access to the LDM software, in collaboration with Unidata, and as an additional service IRADS will provide individual users with support in downloading, installing, configuring, and continuing management of LDM (such services cannot be provided by Unidata). http://www.weather.gov/code88d/
What hardware do I need?
You will need a computer that runs the Unix/Linux operating system and a high-bandwidth connection to the Internet (see below). The processing power required to run LDM is not significant, and top-of-the-line processors are not required. In other words, LDM is not "processor intensive" but is "input/output intensive". LDM constantly writes to the computer's hard disk, so we recommend a hard disk with speeds of at least 7200 RPM. The amount of disk space required is based upon the amount of radar information you wish to store. The maximum amount of Level II data one could receive in a 24-hour period is approximately 4.3 gigabytes uncompressed for a single radar. Because the data are compressed using the loss-less BZIP2 package, (bzip2) which is in the public domain, with a factor of 8 to 10 compression achieved, on average, the daily disk space requirement for a single radar should not exceed about 400 megabytes. In practice, the numbers are much lower, with a maximum amount of approximately 100 megabytes per radar per day. Memory requirements are linked to the number of radars to be received. If you plan at any time to receive the entire set of Level II data from IRADS, we recommend no less than 1 gigabyte of RAM. See bandwidth requirements below.
What type of network connection do I need?
Your bandwidth usage will be determined by how many radar feeds you request. Bandwidth numbers provided by the NOAA Radar Operations Center state an average of 20 Mbps (bits) and up to 27 Mbps during large events.
What level of system support is required?
LDM system support can be largely automated, but in some instances you may need to manually correct or re-configure LDM. IRADS offers a variety of support services, including technicians formally trained in LDM management (these individuals assisted numerous LDM users during the CRAFT project). Further, IRADS offers 24-hours a day, seven days a week support via its 1-800 hotline, and other custom services also are available. 1-866-33-IRADS (866-334-7237)
How do I read Level II data?
Reading a new data format can be a challenge! IRADS provides detailed documentation on Level II data formats including the file linked below. The ICD linked below is directly from the Radar Operations Center (ROC) and is the latest ICD they provide. IRADS members have access to other documentation including a document born out of experience that we feel will provide a more thorough explanation and give you a head start in getting your processing underway.
Level II Data Interface Control Documents
What software is available for using Level II data?
Here are several links to existing software that can read, convert and/or view Level II data:
NOTE: IRADS does not support any of the stated software packages.
- WDSS-II from NSSL
- NOAA's Weather and Climate Toolkit
- Integrated Data Viewer (IDV) from the UCAR/Unidata Program
- NWS CODE & ORPG Software Radar Data Software Library
How are NEXRAD Level II data sent from the radar to me?
Level II data is transmitted from each radar, which is located near a Weather Forecast Office (WFO), via NWSNet to an ingest server located in Washington D.C., and from there the data is transmitted directly to the three top-tier providers (one of which is IRADS).
Can I get some sample data?
We have some sample data available for you! We have setup a sample directory structure that closely resembles our actual live data structure. Within these directories you will find sample storm data for your examination. Level II and CANRAD Sample data
How can this data be used?
We have some excellent examples of how the data is currently being used. Look at this Daily Real Time Analysis and Forecast System developed at the University of Oklahoma!
What radars do you monitor?
Level II radar data feeds monitored by IRADS (pdf)
What's the best way to establish data/server redundancy for IRADS data?
The IRADS pricing model accounts for the bandwidth needed for a single active connection, so if two servers are actively pulling data simultaneously, there would be an additional charge for the second connection.
The most common alternative is to have the backup (second) site/server pull data from the primary (first) site/server. Using this configuration, the second site only gets data directly from IRADS if the first is unavailable. This can be accomplished automatically with the PRIMARY and ALTERNATE directives in the ldmd.conf config file. Both sites can also be set up to pull data from each other in case of network issues. You can request more information here: irads@ou.edu
We have an LDM server but are unfamiliar with LINUX, can IRADS help with setup?
IRADS can help with new user install LDM, but not with full OS installation. IRADS can provide assistance in LDM config and user documentation. You'll need a Sys admin with LINUX experience, however, the LDM install is very simple. You may send questions to: irads@ou.edu
How does LDM process live data?
LDM is the software used by the NWS to transmit the data from each radar site to the forecast offices and aggregation sites. They use LDM for moving around all sorts of information besides radar data including watches, warnings, and forecast data. UCAR in Boulder, CO, developed and maintain this application and others used throughout the NWS.
With each new volume scan, LDM takes 100 radials, compresses it, dumps into an LDM queue as a packet, ships it upstream to the client LDM, then writes the LDM file. The customer reads the file as it’s arriving, and can dump it straight into another process, or wait until the full volume scan for a radar station is complete. There are 156 radar stations, so 156 files every 11 or 12 minutes for clear air mode. As for data processing: IO processing - writing files to disk is the biggest load on a machine. For this reason a solid state drive is highly recommended.
How does data compression work?
For Level II data, the compression used is Beta2. As data arrives, it is written as a compressed file, and IRADS decoder provides decompression (undozip2) for the raw file, producting bits of uncompressed data. It leaves the compressed version on the LDM box. The Interface Control Documents (ICD, UCAR/UNIDATA) has data format information, (ICD for archive 2/user – build 12) which defines the data structure.
The bz2 extension is a bit of a misnomer. At the beginning of each received chunk (except the first in a scan), the first four bytes are a control word that indicates the length of the upcoming compressed data. The first chunk is a bit different. It starts with a 24-byte header, then the control word and compressed diagnostic messages.
Archive II Format: https://www.roc.noaa.gov/wsr88d/PublicDocs/ICDs/2620010E.pdf
Message Types: https://www.roc.noaa.gov/wsr88d/PublicDocs/ICDs/RDA_RPG_2620002P.pdf
Where can I find software to structure and manipulate the L2 data?
NOAA software, Common Operating Development Environment (CODE) creates products on RPG (written in C). CODE is an algorithm development environment for the WSR-88D radar. CODE contains the software and guidance required to create the algorithm development environment on an Intel PC running Red Hat Enterprise Workstation Linux. CODE public edition removes proprietary components for public use, and provides libraries for quick utilization and makes the data structure recognizable.
Header information: The diagnostic information (Message Types 2, 3 , and 5) allow you to do basic checking - to see that all chunks were received. Perhaps the easiest method is to concatenate the chunks together as they arrive. The entire scan isn’t needed to display the lower elevations, but the first chunk contains metadata needed to step through the following data. The first 24 bytes make up the Volume Header Record, and contain the filename, start date/time, and call sign of the radar. The next four bytes are a control word that indicates how long the first record is. The four-byte control words let you step through the entire scan.
More information on the formatting and allowed directives for these files is in the Configuration Files section of the LDM Basics page: http://www.unidata.ucar.edu/software/ldm/ldm-current/basics/index.html
For using the data, NOAA’s Radar Operations Center provides the definitive collection of Interface Control Documents: https://www.roc.noaa.gov/WSR88D/BuildInfo/Files.aspx
There is a lot of data in the ICDs, but the most useful is likely the Message Type definitions: https://www.roc.noaa.gov/wsr88d/PublicDocs/ICDs/RDA_RPG_2620002P.pdf
Actual radar data arrives as message types 1 and 31. Message type 5 contains VCP data. It will list what the radar plans to do during the upcoming scan, including number of elevations, VCP, resolution, etc. Message type 2 and 3 contain health monitoring information about the radar and associated equipment.
Can you help us conduct testing with the L2 data?
Yes. Once the LDM box is running, you can access archived L2 test data. It's the same as live data. IRADS offers data samples of level II and CANRAD data. These are raw, compressed data files. Note: undozip2 will open these files.
What happens after successful testing?
You provide IRADS an IP address and we push the data. We typically allow a one week test run to ensure the data is arriving properly and processing of the live data is going smoothly. Next, as needed, we offer a phone consult to answer questions or resolve any problems. Finally, OU sends you a contract for signature. The entire process may be 1-2 weeks. Note: IRADS can begin the contract process while you're testing the data to speed up the process.
What about accessing archived data?
IRADS doesn’t archive data. We have 12 roughly hours of data on our system. NCDC NOAA has L2 archives. IRADS customers are primarily focused on real-time L2 data. Amazon Web Services offers archived NOAA radar data (NEXRAD on AWS).
What are typical latencies seen from radar site to the LDM server?
Max time is 10 seconds, but typical times are 1-2 seconds from the RPG (radar product generator). 2-3 second latency is a good overall average for all radars.
Note: This is true for radar sites within the Continental United States. Sites in the Pacific, Korea, and Japan, may be closer to 30 seconds due to distance and network limitations between the sites and NOAA’s aggregation points.
What are the bandwidth requirements for receiving the full radar feed?
Expect 2+ MB/s for stormy weather with a total file size of up to 50MB (for a 3-5 minute scan). On a clear day, a whole scan may only be 2-3 MB total (for a 10-11 minute scan). Note: the smallest bit is 1 chunk, which is 120 radials, 3 scans per level. The volume scan header explains the mode of the particular radar.
More information is available from the Interface Control Documents provided by the NOAA Radar Operations Center: https://www.roc.noaa.gov/WSR88D/BuildInfo/Files.aspx
What redundancy does IRADS utilize?
Both stacks are located in Norman, OK., in a hardened datacenter facility, but are hosted in different network IP spaces on completely separate network hardware.
Is there a simpler dataset than Level II?
The NWS makes radar data available in two formats: Level 2 and Level 3. The Level 2 data is the raw, high resolution data directly from the RDA (that's what IRADS provides). Level 3 data products are lower resolution, pre-rendered and produced by the radar product generators located at each forecast office. The pre-rendered products include reflectivity for each elevation, radial velocity, echo tops, total storm precipitation, and a host of others. This suite of products is generated and sent out every time a radar completes a volume scan. This page describes the products and has links to recent data. IRADS makes this L3 data available in real time via the NEXRAD3 feed in LDM. (See our NOAAPort data offering). https://www.roc.noaa.gov/WSR88D/Level_III/Level3Info.aspx
Level II Rendering & Troubleshooting
Please refer to the following resources:
- Warning Decision Support System: http://www.wdssii.org/
- NOAA’s Climate Toolkit: https://www.ncdc.noaa.gov/wct/
- WSR-88D CODE: http://www.weather.gov/code88d/
- Unidata IDV: http://www.unidata.ucar.edu/software/idv/ - IDV can be installed and pointed at a directory of live L2 data
- Unidata GEMPAK: www.unidata.ucar.edu/software/gempak/ - this software can serve as a test site to render and monitors select radar sites through IRADS
