Data Processing and Visualization
While the measurements made with remotely piloted systems are valuable on their own, not all data is good data. Any sensor is vulnerable to inaccuracies induced by damage, use, and other factors. Derived variables don’t appear in sensor logs at all. To create the UAS datasets researchers and forecasters need to improve their understanding of the ABL system, we have a group dedicated to processing raw data, creating meaningful visualizations, and packaging them for distribution with other scientists.
CASS UAS autopilots store flight diagnostics in real-time to an onboard micro-SD card in binary format. These files contain information about the aircraft’s attitude angles (roll, pitch, and yaw), GPS location, altitude, accelerations, and sensor output. The Profiles Python package developed by CASS scientists synthesizes these files along with information from flight logs, calibration coefficients, and geographical data files to create self-describing, quality datasets. This software performs a statistical analysis of the different measurements to produce a post-processed quality-controlled product in netCDF format that contains metadata about platform specifications, flight conditions, mission objectives, and flight path. Currently, this post-processing workflow is performed offline after a set of flights is completed. However, CASS scientists are working to streamline this process to be performed in near-real time utilizing cloud computing resources. This will allow for rapid visualization and dissemination of data to both the public and decision makers. The planned cloud-based infrastructure will also provide a more robust data archival process thanks to automated post-processing capabilities.
Data visualization is also a key component to this research nucleus. As CASS continues to usher in a new paradigm of PBL observational techniques, the way scientists visualize and interpret these data must evolve. The data processing team is continually developing innovative data visualizations to provide the maximum impact from the data collected in the PBL. Additionally, the data processing nucleus is focusing on developing an architecture to control a UAS remotely to support the 3D Mesonet concept outlined in Chilson et al. (2019).
Some specific current and future projects related to data processing and visualization include:
- Development of post-processing algorithms for wind estimation and other derived variables
- Optimization of parameters calculated in-flight versus post-processing
- Data directory management and automation
- Spatial analysis of simultaneous flights in different locations
- Streamlining of sensor calibration procedures
- Development of architecture for live data streaming and cloud archival