ARM Priorities

Tasked by DOE, Mark Ivey and Beat Schmid have produced an implementation plan for a new ARM UAS program. It will consist of Tethered Balloon Systems, small-rotor craft, small UAS, and mid-size UAS. The program will be implemented and managed jointly by SNL and PNNL as spelled out in the implementation plan. This ECR covers the PNNL actions resulting from the UAS Implementation Plan. A corresponding ECR will be submitted by SNL. Actions so far have been documented under ECO 01057: ARM Next Generation - Aerial Measurements. But separate ECO's are needed now.

ARM Archive received many recommendations and feedback to improve the data discovery tool. The scope of this ECR is to address the following high level changes:

• Fix the display and search issues, and corresponding metadata
• Improve the data discovery home page (revise current highlights, include new ones such as special campaigns, radar data etc..)
• Improve the search & display capabilities
• Provide additional functionalities (radar data – default measurements for every order)
• Integrate NCVWeb for interactive visualizations and also improve the current plot viewer and DQR reports
• Update recommended measurements to include radar and other new data streams by gathering input from the PIs
• Develop a design to consolidate the FC data delivery using the data discovery model

The overall look-and-feel framework will not be changed, we will include that as part of the planned ARM website refresh.

Detailed tasks list will be developed using the above groupings and tracked using the ServiceNow project tasks. Separate design reviews will be carried out for the major features using the Agile - sprint plan & reviews.

This is a component of ENG 1061 ARM Next Generation - Computing Architecture for Big Data Processing and High Resolution Modeling. This ENG will be used for tracking the ADC processing and visualization cluster design and implementation.

Many of the background information are captured in ENG 1061.

This is a component of ENG 1061 ARM Next Generation - Computing Architecture for Big Data Processing and High Resolution Modeling. This ENG will be used for tracking the LASSO cluster design and implementation.

Many of the background information are captured in ENG 1061.

The CDR for the LASSO cluster is already completed and it is captured in the ENG 1061. This ENG will be primarily used for procurement, deployment and initial use case developments.

This ECR/ECO will serve as collection point for design documents related to a new facility to be installed at Oliktok Point, Alaska.

New airborne atmospheric measurement systems and supporting ground systems will be developed and deployed at Oliktok Point, Alaska. This ECR/ECO serves as the collection point for design documents and communications.

ARM Next Generation organization, planning, and schedules require a series of Engineering Change Orders to initiate the analysis and production of information for use in determining the measurement strategy, contract preparation for arriving equipment, and process reinforcement/development for transitioning to the SGP Supersite with Enhanced Mobile Facility.

Identifying Non-Atmospheric Returns in The Doppler Spectra Data from The Radar Wind Profiler for Improved Estimation of Moments and Winds

Science Lead: Virendra P. Ghate Instrument Mentor: Richard L. Coulter Developer: Jonathan J. Helmus Translator: Scott M. Collis

Introduction and Motivation The Department of Energy (DOE)’s Atmospheric Radiation Measurement (ARM) program operates total of nine Radar Wind Profilers (RWP). Four RWPs are part of the instrumentation present at the Southern Great Plains (SGP) observing facility, while the facilities at the North Slope of Alaska (NSA), Eastern North Atlantic (ENA) and the three ARM Mobile Facilities (AMF) have one RWP each. The RWPs operate at 915 MHz or 1290 MHz, and collect data in two operational modes; a precipitation mode (PR) and boundary layer mode (BL). The precipitation mode provides un-attenuated and un-saturated estimates of Doppler spectrum moments during heavy precipitating conditions, when cloud radars often have difficulties due to signal attenuation. While, the RWP boundary layer mode yields wind speeds and wind directions in the lowest part of the atmosphere providing valuable information on atmospheric turbulence. The PR and BL modes were set to operate in two pulse length settings, with recently the BL mode being operated only in a single pulse length setting. With recent renewed interest (Tridon et al. 2013) in the retrievals that can be made using the data from the RWPs, we have begun to explore improvements to the existing data products that ARM provides from the RWP. During our previous work, we have discovered that although scientifically useful, the RWP Doppler spectrum and moments data suffer from contamination from non-atmospheric targets (trees, power lines etc.) termed as clutter. An example of clutter signal observed in the Doppler spectrum moment data collected by the RWP present at the SGP central facility is shown in Fig. 1 and Fig. 2. If left unmitigated, the clutter signals introduce artifacts into the calculated RWP moments and contaminate retrieved atmospheric parameters such as winds, boundary layer depth, vertical velocities, and hydrometeor distributions.
The presence of non-atmospheric clutter has also been observed in RWPs operated by other organizations (NOAA, EPA etc.). Past studies based on data collected by those RWPs have demonstrated that identifying and removing the clutter signals from the RWP Doppler spectra has resulted in substantial improvements in the quality of the moments and the retrieved meteorological parameters (Bianco et al., 2008). To improve the value of the data collected by the RWP present at the ARM observing facilities, we propose to develop and implement clutter removal technique for the RWP present at the ARM SGP central facility that can be eventually applied to all ARM RWPs.

Data and Proposed Methods The ARM RWP have gone through multiple changes in operational settings since the original setup in 1992. Until 2011, the ARM RWPs were operated in traditional setup with two pulse lengths, dwell time of ~30 seconds, three beam angles and 64 FFT points. With improvements in technology and adapting to the science needs, the operational settings were changed to have dwell time of 14 seconds in 2011. Further modifications were made to the operational settings in early 2013 to have only one pulse length. This work will primarily focus on the data collected by the RWP in the boundary layer modes during the summer of 2012 and 2013, and will eventually extend it to other modes and previous periods. The RWP operates in the BL model with two pulse length settings; short pulse (SP) with range resolution of ~rameters. The moments are calculated using the data at the bins having sufficient atmospheric score,nals. Our initial analysis revealed that although novel, the method cannot be directly applied to the data collected by ARM RWP due to significant differences in operational settings of the ARM RWP and the RWP used in the study by Bianco and Wilczak (2002). Particularly, the membership functions of fuzzy logic algorithm need to be significantly modified for the technique to be applicable to the data collected by ARM RWPs. We propose to not only modify the membership functions of the fuzzy logic algorithm but also use data collected by collocated instruments such as the ceilometer and Doppler lidar in identifying clutter signals

Sites/Conditions The data collected by the RWP present at the SGP central facility during the summer months of 2012 and 2013 will serve as a test-bed for this effort. We will particularly use data collected during clear-sky convective periods as determined by the ceilometer and the ECOR measurements. We anticipate the developed algorithm to be readily applicable to all of the nine RWPs part of the ARM program.

Testing and Analysis Procedure Radiosondes are launched at the ARM SGPe RWP clutter free moments. The Doppler Lidar present at the ARM SGP central facility measures the vertical velocity at a high temporal and range resolution. It also observes the horizontal wmoments. Also, the vertical air motion within cloud layer as reported by the RWP can be compared to that observed by the cloud radar.

Input Datastreams The algorithm will read data from the foll create a new value added product (VAP). The VAP will have clutter free Doppler spectrum and moments data in a CF/radial compliant format.

Proposed Beta Users Laura Bianco (NOAA – Earth System M. Wilczak, 2002: Convective boundary layer depth: improved measurements by Doppler radar wind profiler using Fuzzy Logic Methods. J. Atmos. and Ocean. Tech., 19, 1745-1758. Bianco, L., J. M. Wilc5, 1397-1413 Tridon, F., A. Battaglia, P. Kollias, E. Luke, C. R. Williams, 2013: Signal postprocessing and reflectivity calibration of the Atmospheric Radiation Measurement Program 915-MHz Wind Profilers. J. Atmos. Oceanic Technol., 30, 1038–1054.

At the May 2014 SGP High-Resolution Modeling workshop, one of the core recommendations was to deploy a network of sites around the SGP central facility that would provide continuous profiles of temperature, humidity, and wind that would support efforts to develop a continuous model forcing data set. It was noted that there are remote sensors that could readily provide this information for the boundary layer but the free troposphere will be much more challenging. The current thinking is that four profiling sites would be deployed on a radius of ~30-50 km from the central facility but that they would designed to be portable so that the horizontal domain spanned by the network can be changed relatively easily in the future. The elements of these profiling sites are currently planned to be:

• an AERI for T/RH profiles in the boundary layer
• a Doppler lidar for horizontal and vertical winds in the boundary layer
• a microwave radiometer as a column constraint on water vapor. In principle this could be replaced with a microwave profiler, though it is not clear that there is sufficient information gained to warrant that change

It is also possible that these sites could be augmented with wind profilers. The radar wind profilers would provide information into the free troposphere but with lower resolution than the Doppler lidar. We need to consider the cost/benefit of including both instruments.

It is also envisioned that these boundary layer profiling sites would include measurements of surface radiation and heat fluxes and soil properties for a fully integrated view of the surface and boundary layer.

Where possible, these sites would leverage instruments coming back from the Tropical Western Pacific and other spares around the program, but would also require some new procurements including the portable infrastructure.

We would like to move ahead with the implementation of these sites in FY15 to begin operation in FY16 and are seeking input for the optimum design of this profiling array.

This request represents a sub-component of the overall SGP reconfiguration umbrella task, ECO-01059. It also relates directly to the just-submitted ECR-01118. That request seeks to refurbish a number of older AERIs for a variety of applications including this profiling network.

The report from the SGP ARM/ASR High-resolution modeling workshop is available at: http://science.energy.gov/ber/news-and-resources/ under the “Workshops” section (it is currently the first report listed under that section).

The current plan is to 1) reposition the Southern Great Plains Site (SGP) for contributions to LES model studies, 2) redeploy equipment from Tropical Western Pacific (TWP) sites, 3) improve/upgrade the unsatisfactory performance of some ARM precipitation sensors and 4) upgrade precipitation observations at NSA. This will improve observations of rainfall, frozen precipitation and the rainfall drop size distribution at ARM sites.

The MFRSRs (and MFRs and NIMFRs) currently have narrowband sensors at 415, 500, 615, 673, 870, and 940 nm, and one open/broadband sensor. We propose replacing the open/broadband sensor with a 1625nm sensor.

Scientific Motivations:

The addition of a near-IR channel to the ARM MFRSR/MFR/NIMFR instruments will allow for improved retrievals of aerosol and cloud properties. The longer wavelength, sensitive to larger particles, will help constrain the size distribution of coarse mode aerosol particles, allowing for more accurate retrievals of aerosol optical properties. We also expect to improve cloud optical depth and mean effective cloud particle size retrievals using a combination of this near IR wavelength and a mid-visible one. When implemented for up- and down-pointing paired MFRSR and MFR units, the longer wavelength will better constrain spectral surface albedo estimates, especially for vegetation with a strong spectral dependence across the visible and near-IR.

We are currently considering the need to either modify or replace the current filter profile bench at SGP to accommodate the new 1625 nm channel. A filter bench that operates at that wavelength range would be used to measure the filter profile of the 1625 nm channel, along with the six other channels. This characterization of the filter profile is critical for applying accurate calibration coefficients and for using measurements among different instruments in tandem, such as up- and down-looking units for measuring surface albedo. Without a new bench, we would move forward by measuring the profiles of the current six channels every time a head is run through the cal facility, and using the original profile of the 1625 nm interference filter measured before mating to the sfice to monitor.

This Change Request is to integrate a subset of the MAOS infrastructure (1 container and baseline instrumentation) within the AMF1. The proposal and implementation strategy is summarized in a White Paper that was submitted to the Program Manager, Sally McFarlane for initial review. The White Paper was written in response to the need to replace the AMF1 AOS instrumentation that is being retired in FY16 after the completion of the GOAmazon campaign in Brazil and as a cost-effective mitigation strategy to deal with the high Operational and Mentor costs to run the MAOS. This ENG will track the discussion in considering the integration of the MAOS into AMF1 while also defining a redefined solitary MAOS container to be used with any AMF or fixed site.

The implementation strategy outlined in the White Paper is divided into 2 Phases over FY16 - FY18:

Phase 1, Incorporation of MAOS-A into AMF1 for LASIC, includes: (1) Replacement of AMF1 AOS with MAOS-A, (2) Use of the Original AMF1 AOS Instrumentation, (3) Impact to the LASIC Campaign.

Phase 2, AOS Standardization and the Use of MAOS, includes: (1) AOS Standardization, (2) MAOS Deployment, (3) Impact to CACTI and Future Campaigns.

The White Paper was submitted to the AMSG in January for review and implementation advice by the end of the month in order to prepare for the upcoming LASIC and CACTI deployments. The AMSG was asked to evaluate the proposed implementation in terms of how to best outfit the AMF1 and MAOS based upon the AMSG recommendations for optimal (and realistic) recommendations for the overall standardization of ARM AOS baseline instrumentation.

Install six digital cameras to generate stereo-photogrammetric 3D cloud masks

1. Background

Shallow Cu are difficult to measure quantitatively due to their low liquid-water content (which produces a reflectivity that is often below the detection threshold of cloud radar), varied sizes and large inter-cloud spacing (making vertically pointing instruments subject to the vagaries of whatever may or may not pass overhead), and manifestly three-dimensional (3D) structure (which cannot be characterized using a total-sky imager). Stereo photogrammetry, which uses multiple visible-spectrum cameras to map the surfaces of clouds in 3D, can overcome these limitations. Stereo cameras can: measure clouds with extremely low liquid-water content; map multiple clouds simultaneously; and, with a sufficient number of cameras, generate a gridded product that defines the 3D volume occupied by a field of clouds. Since these gridded products can be generated at arbitrarily high frequency (although once every 10 seconds is likely sufficient), the lifecycle of shallow Cu can be characterized in ways that are not possible with any other instrument. The ability to measure height-resolved and time-resolved cloud sizes and cloud area fractions is particularly important for the LES ARM Symbiotic Simulation and Observation (LASSO) project, which needs these data as a ground truth for validation of the LES.

2. Requirements

Six cameras will be deployed at the Southern Great Plains (SGP) site at a distance of roughly 5 km from the Central Facility (CF). These cameras will be grouped into three pairs. To provide optimal coverage, each pair will ideally be located 120 degrees from the other pairs, measured with the CF at the origin. Within each pair, the two cameras must be located a distance of roughly 500-1000 meters apart in a direction roughly perpendicular to the line connecting that pair to the CF. One possibility is to use the two new SACR sites as locations for two of the cameras (one camera from each of two pairs). These locations would provide two of the cameras easy access to power and internet. All of the cameras must have a clean line of sight of a cube of atmosphere roughly 4 km to a side centered on the CF. The cameras must also have power and internet access (for data transmission, remote troubleshooting, and time server synchronization). The cameras must be mounted on sturdy structures that prevent wind-driven shaking (which would introduce errors into the stereo reconstructions). Each camera's power supply must be sufficient to power the camera (<= 5W), attached computer (Raspberry Pi or equivalent, <= 5W), and, if needed, radio link for internet connectivity (<= 5W). Data transfer rates will be on the order of 1-2 MB/minute, depending on the image quality and image frequency. Frequent time synchronization is important to keep the computer clocks faithful and to keep the image acquisition synchronized. If this is not done through the internet, a solution using GPS may be an alternative. Each camera and associated electronics (e.g., Raspberry Pi, GPS receiver) must be housed in a weather-proof enclosure with a glass window.

3. Tasks

One of the first tasks will be to draw up plans for a mount, enclosure, power supply (PV and balance of system if nothing else is available), and radio link (for the internet connectivity). These decisions will need to be made in close collaboration with relevant experts in the ARM program. In tandem, LBNL will draw up plans for the appropriate combination of camera, lens, and computer to optimize image quality while holding power consumption to a minimum. Note that these two sets of es; and there are several issues that must be taken into consideration regarding the compatibility oe hardware will be acquired for a single camera installation. This single camera will be deployed for testing at either LBNL or SGP. This will give us the opportunity to ensure that all of the components work and work well with each other before acquiring all six cameras and associated hardware.

4. Timeline

Due to the novelty of designing standalone cameras without wired power or internet, there is some uncertainty in how long it will take to move from the drawing board to an operational installation of six cameras. The following is subject to revision as we learn by doing. (Note that the computational requirements for performing stereo reconstruction on the images will be specified in a separate ECR. This ECR is intended to install the six cameras and bring them to an operational status, by which we mean the collection of images that are then stored at the ARM archive.)

2/2016 - 3/2016: Decide on hardware components for standalone camera with neither wired power nor wired internet. This will include the camera, lens, computer, enclosure, mount, PV panel, balance of system, radio link, and GPS receiver (if needed).

4/2016 - 5/2016: Procure, assemble, and test the components for a single camera installation. Programming of the computer and calibration of camera optics will take place at LBNL. Assembly of the other components will occur at either LBNL or SGP. Final testing in the field will take place at either LBNL or SGP.

6/2016 - 7/2016: Procure and install all six cameras at the SGP site.

5. Budget

The cost of each camera installation is expected to be less than 5k, for a total cost under 30k for all six cameras. Personnel costs at LBNL during the 6-month period from February 1 to July 31 are expected to be 2-months FTE of a Project Scientist (Oktem), 2-months FTE of the PI (Romps), and 1-month FTE of an LBNL IT staff (Lee or Thomas).

Dual polarization radars represent a strategic investment by ARM, providing unique capabilities and benefits for end users. One common challenge raised by data users is the inability to relate low level radar products with physically meaningful parameters. Radars are complex instruments with complex data products that users often find difficult to interpret. The production of these complex products is carried out through what is known as a data pipeline. Historically, successful radar data production is achieved "on-line". By "on-line", we mean a continuous process from data capture to advanced data products production without manual intervention. Such a pipeline needs to be maintained and monitored from timeseries (IQ) data to derived products because of the complex interactions between system designs, signal-processing algorithms, operating mode, calibration, and radar retrieval algorithms. This is particularly relevant to the ARM radars due to the wide variety of radar types, modes, and scanning strategies. These complex interactions between the various stages of the data production process result in a propagation of errors (bias and variance), which can be monitored at various stages in the pipeline. The resulting data and retrievals at the end of the pipeline will be delivered to end users through the ARM archive. To maximize the use and impact of these data, they must be released as regular products, at regular intervals, without requiring significant manual intervention.

These products have huge implications for users, as it is through these data that we interact with and receive feedback from them. Additionally, we are able to monitor the effectiveness of the operations and engineering strategy through these data, providing direction and feedback for the internal ARM infrastructure. The proposed pipeline enables the Radar Engineering and Operations to perform various tasks, including:

1. Minimize the delay in improving the product.
2. Analyze the metadata from retrievals to trace problems.
3. Improve system-monitoring capability.
4. Improve data quality.
5. Design operational configurations.

There are several operational considerations that need to be addressed and these shall be discussed during the next few months, culminating with design reviews. Some, but not all, of the considerations for the radar data pipeline are listed here.

1. Large data buffer at DMF to process data for calibration and quality control.
2. Methods for agile reprocessing to correct errors and introduce additional features. The complexity of radar data necessitates the ability to improve and reprocess data products from source/raw data.
3. Ability of mentors, translators, and developers to modify and improve the algorithms used in calibration and quality control (QC). Allow the coordination of uncertainty quantification into the generated products.
4. Assign DQPR and DQR to data including retrievals and VAPs. This includes data provenance on DQRs. DQRs in products need to incorporate DQRs from source data.
5. Metadata and archiving of data products.

This Change Request is to implement an operational data flow pipeline from radar base data to retrievals. Such a system is not new to ARM and has been/is currently in place for some of the instruments/products. However, such a seamless pipeline does not exist for the radars and radar retrievals. The proposed pipeline is critical to provide high quality data in a timely manner to the end-users, while providing the platform for operations to evolve additional data quality improvement methodologies.

Affected instruments XSAPR and XSAPR2 CSAPR and CSAPR2 SACR and SACR2 KAZR and KAZR2 WACR MWACR

Complete data reduction to get calibrated solar spectral irradiances and optical depths from Aug 2009 to Mar 2014 RSS deployment; estimate RSS costs to restore operation

The Rotating Shadowband Spectroradiometer (RSS) ceased operating on March 10, 2014. Inspection showed a badly burned section of the control board in the heater-controller area. The vacuum system that keeps the CCD clean continues to function. The decision was made in early 2015 to finish the data reduction of the Aug 2009 to Mar 2014 data set to evaluate its usefulness to the ARM/ASR communities. A decision would then be made whether to attempt to repair the instrument. The data reduction task is not wrapped up, although many essential pieces are completed; a report outlining the progress toward that end in attached. In the next year, we would propose to automate the data reduction to complete the task of producing calibrated global and diffuse horizontal plus direct normal spectral irradiances for 4.5 years. We would also produce aerosol optical depths for the 362-1072 nm wavelength range, and, to the extent possible, retrieve NO2, O3, and H20 using the bands that fall in this wavelength region. Further, assuming a desire to restore the RSS to operation based on the usefulness of the data set resulting from the effort above, funds will need to be expended to hire a consultant familiar with the RSS electronics to acquire a new control board, redesign the heater-controller to prevent another meltdown, and install the modified board to bring the RSS back on line.

The purpose of this request is to track activities associated with the ARM Radar Engineering and Operations Work Plan through FY2017.

The use of radar data to study and retrieve cloud properties is not a simple task. Extracting cloud properties based on radar observations requires well-characterized radar and a good understanding of the microphysical processes that occur in the cloud systems, which enables the development of retrievals and products. A well-characterized system is one where the uncertainties of calibrations, waveforms, signal processing, and system errors are well estimated and measured. The errors and quality of the retrievals and products also provide a feedback on the quality of the radar observations. The proposed plan is an agile plan that relies both on engineering evaluation and analysis of multiparameter radar observations.

The essence of the plan is relatively simple: focus attention on sets of radars using a phased approach with the goal of addressing core issues for each radar over a period of approximately two years. The key elements of the proposed strategy is the coordination of sustained effort towards: (1) characterizing the radars (2) calibrating the radars, and (3) streamlining the development of radar products and associated feedback for data quality. In order to realize the key elements of the strategy the operational setup must be abstracted into two categories: phased effort bringing radars to a characterized operational status and end-to-end data flow.

The radar science group has expressed a strong interest in evaluating the Precipitation Imaging Package (PIP) at Oliktok. The PIP provides images of ice crystals and so provides important information about hydrometeor habit to complement radar measurements. The PIP has previously been deployed by NASA at Hyytiala in conjunction with the BAECC campaign. Walt Petersen, the lead for NASA GPM (Global Precipitation Measurement) ground validation has offered to loan ARM a PIP to deploy at Oliktok for the purpose of evaluating at that site with the dual goal of augmenting ARM radar measurements and supporting GPM validation. The purpose of this request is to deploy PIP at Oliktok, initially for one year, to evaluate the potential of that instrument at the Oliktok site.

The CSAPR2 is a transportable weather radar that can be deployed for field campaigns. The CSAPR2 receiver electronics and antenna are installed on a trailer mounted pedestal without a radome. Radome protects the antenna and pedestal from severe weather such as hail and high winds. The lack of a radome inhibits radar operations in severe weather. This request is to procure and deploy a radome for the CSAPR2.

. Overview

Spatial variability is of critical importance to many scientific studies, especially those that involve processes (e.g., precipitation, clouds, and radiation) of great spatial variations at high temporal frequency. High-density atmospheric radiation measurement (ARM) sites deployed at Southern Great Plains (SGP) allow us to observe the spatial patterns of variables of scientific interests. The upcoming super site at SGP with enhanced spatial density will facilitate the studies at even finer scales. However, currently the data is only reported at individual site locations at different time resolutions for different datastreams. It is difficult for the users to locate all the data they need and requires extra effort to synchronize the data. The ARMBE 2D gridded surface data (ARMBE2DGRID) will merge various datastreams together and interpolate them onto a common 2D grid with a uniform temporal resolution of one hour interval, the same as that used in current ARMBE datasets. In addition, we also provide an hourly station-based surface dataset (ARMBESTNS) that contains the same variables as in ARMBE2DGRID. Higher temporal resolutions will be considered based on research needs and data availability. Besides a more user-friendly format, these value-added products (VAPs) will apply the following necessary stringent quality control (QC) procedures (Zhang et al. 2001ab) to further improve the data quality over the raw sources:

1). Maximum and minimum check 2). Outlier check 3). Temporal variability check

2. Implementation Plan

1). Development of ARMBE2DGRID and ARMBESTNS for MC3E

We plan to start the development of ARMBE2DGRID and ARMBESTNS with processing data for the Midlatitude Continental Convective Clouds Experiment (MC3E) at the SGP site. The grid for MC3E is at 0.25x 0.25 deg and covers the domain at 34.75N—38.75N and 95.25W—99.75W. We derive the surface quantities in each small grid box. If there are actual measurements within a 0.25 x 0.25 deg grid box, simple arithmetic averaging is used to obtain the value for that grid box. Some variables are available from several instruments. They are merged in the arithmetic averaging process with a weighting function depending on the quality of the measured data. If there is no actual measurement in a 0.25 x 0.25 deg box, then the Barnes scheme (Barnes, 1964) is used with the length scale of (Lx=50km, Ly=50km, and Lt=6hr) to fill the missing data. The purpose of this task is to use MC3E to refine the algorithm needed for development of these two VAPs and to get the data released to the community quickly for this major ARM field campaign in support of several major cloud modeling studies.

2). Development of ARMBE2DGRID and ARMBESTNS for multiple years at SGP

Once the MC3E data is released, we will apply the code and algorithm to the time periods where continuous forcing data is available so that we provide a more comprehensive dataset for statistically studying cloud systems and evaluating high-resolution cloud models

3. Variables

surface winds, temperature, relative humidity, turbulent fluxes, radiative fluxes, precipitation, liquid water path, precipitable water, soil moisture, soil temperature, soil heat flux, total cloud cover, surface albedo, boundary layer height, vegetation height, and land cover.

4. Output

The output variables of the ARMBE2DGRID data are the same variables as input, but at the resolution of 0.25 x 0.25 x 1 hr.

The output variables of the ARMBESTNS data are the same as those of ARMBE2DGRID data, but reported at the original site locations.

5. Uncertainty estimates

Uncertainties in ARBME2DGRID and ARMBESTNS wil Data Quality Office and original data developers.

7. Effort

The task requires 1 FTE for one y

This ECR describes the planned creation of a new VAP that will (1) classify cloud types from ARSCL data, and then (2) further identify times with shallow cumulus. This VAP is primarily motivated by the needs of the routine LES modeling project. The shallow cumulus identification will be used by the LASSO team in deciding which days to model. The more general cloud classification will be useful as an index to help users find times of meteorological interest. It is also envisioned that the cloud classification can be used as part of the data discovery by meteorological regime described by the LASSO team, as that is being developed.

A summary of the VAP plan is provided here, but a full VAP implementation plan is available at the following link:

https://engineering.arm.gov/~riihimaki/ImplementationPlan_cldclass_20151014.pdf

Algorithm Outline:

1) Cloud-type classification

Each cloud layer at the SGP site will be classified based on the method by Mace et al. (2006). However, differently from Mace et al. (2006), in which cloud optical depth from the multi-filter rotating shadow-band radiometer (MFRSR) is used to define the cloud-types, the cloud thickness, that is directly calculated from a cloud top and base height following McFarlane et al. (2013), is utilized. These cloud boundaries will come from ARSCL and include information from the cloud radar, MPL, and ceilometer. In addition, the cloud top height may be refined using a fuzzy logic-based algorithm (Chandra et al., 2013), which can help eliminating insect radar echoes in the boundary layer, and thus improve the detection of the cloud top of low clouds. This classification method will be compared to the existing cloud classification evaluation product available from 1999-2001 that uses the fuzzy logic algorithm of Wang and Sassen (2004), and a determination of what is the most efficient method to make operational that meets our needs will be determined.

2) Sub-category of the low clouds, shallow cumulus clouds

The times/dates of shallow cumulus clouds are another interest in our VAP to complement the DOE’s next generation LES modeling efforts. There are several studies in the literature that manually identify shallow cumulus clouds from ARM data (Berg and Kassianov, 2008; Chandra et al., 2010, 2013; Zhang and Klein, 2013), but there is not yet an automated version. Thus we intend to start by investigating to what extent an automated algorithm using ARSCL cloud boundaries (including MPL, cloud radar, and ceilometer data), TSI images, and GOES satellite data can reproduce these manually identified shallow cumulus clouds.

First, the periods with low clouds will be further sorted by choosing the dates/times having single-layered fair-weather low clouds. Additional conditions such as a small fractional cloud cover and a time-varying cloud fraction, which can take account of features of shallow cumulus clouds, are under consideration to pick out the times/dates of shallow cumulus clouds among the classified low-clouds cases. Furthermore, the use of the Geostationary Operational Environmental satellites (GOES) 0.5 hourly data will help us to identify homogeneity of identified shallow cumulus clouds. The GOES data can help exclude optically thick clouds and compare regional homogeneity in cloud cover or properties.

The identified times/dates of shallow cumulus clouds will be simultaneously compared with a cloud classification generated from image processing of the Total Sky Imager (TSI) images being developed by Jessica Kleiss. The method uses a support vector machine (SVM), a supervised, nonparametric classifier used frequently in computer vision classificatiocommunity feedback. That will take place over the next few months. Then the chosen algorithm will be16 (September 30, 2016), and to make that operational the following fiscal year.

The effort this fiscal year (to produce an evaluation product) is estimated to take 500 hours, but will be determined more precisely when the algorithm has been finalized.

Suggested reviewers: Translators Kyo-Sun Lim Larry Berg Yunyan Zhang Jessica Kleiss Bill Gustafson Andy Vogelmann Heng Xiao Tim Wagner

Title: Develop a development roadmap for Py-ART

The Python-ARM Radar Toolkit is an open source community code for working with and extracting insight from radar data. Py-ART has a very large and diverse user base. Each week the Py-ART repository has over 800 page views and 120 unique visitors. Py-ART has 46 “forks”, multiple instances being developed by other GitHub users. The Py-ART email list is very active list and has 65 members. Most importantly there are 14 known active developers with nearly 10,000 lines of code contributed by Non-ARM funded developers and scientists.

As Py-ART grows and a palpable appetite in the community to contribute grows there is a need to develop a roadmap to guide this growth. In making decisions on support development needs to be guided by the needs of the community and with DoE ARM contributing significant resources though the time of the Argonne translator team and the help of PNNL radar mentors it is important that Py-ART meets the needs of DoE (CESD) stakeholders.

This proposal covers the work to compile the thoughts of the development and science leads (Helmus and Collis) and the thoughts of the rest of the community including some who do not currently use Py-ART but who are part of the ASR science community, specifically asking what would entice them to use the toolkit (suggestions requested). We will be using this issue on GitHub as a starting point: https://github.com/ARM-DOE/pyart/issues/390

We will aim to have a good document by the time of ARM/ASR radar meeting

60/40/10 (100) hours Collis/Helmus/Weber 15 hours putting together a focus group and review committee. 45 hours surveying, discussing and background researching. 50 hours writing the document, proofing and reviewing with groups.

Reviewers: Translators Helmus, Weber Radar Science: Kollias, Clothiax (Non-user?), Nesbitt, Schumacher, Kumjian, Giangrande, Van leir-Walqui, Oue, Varble (non-user), Dolan, Rutledge, Protat, Chandra, Matrosov (non-user) Verlinde (non-User?), Luke (Non-User), Rhyzkov,(Non-User?) Miller(Non-User) Bharadwaj, Hardin Gaustad (ADI lead, non-user)

Note: Non-User also implies no supervised employees or students benefit from Py-ART

New AERI optimal estimation (AERIOE) VAP

Proposed VAP name: aerioe1turn Science Sponsor: Dave Turner (dave.turner@noaa.gov) Translator: Laura Riihimaki (laura.riihimaki@pnnl.gov) Developer: Tim Shippert (tim.shippert@pnnl.gov)

Motivation

The Atmospheric Emitted Radiance Interferometer (AERI) measures downwelling spectral infrared radiance in the wavelength range 3.3-19.2 microns (520-3020 cm-1 wavenumbers). This is the terrestrial radiation (LW) wavelength range so gives a great deal of information about temperature and humidity profiles in the boundary layer, and liquid water for liquid water paths less than 100 g m-2 or so.

This plan proposes to implement Dave Turner’s new optimal estimation retrieval [Turner and Löhnert, 2013] as an operational VAP. AERIOE offers a number of improvements over the AERIPROF VAP that is ARM’s current data product for thermodynamic profiles from the AERI. First, AERIOE is not limited to clear sky cases, and in fact also retrieves the LWP of clouds less than 100 g m-2, a critical need for shallow cumulus measurements. Second, the optimal estimation methodology is less sensitive to having a good initial guess to implement in the forward model. This gives higher accuracy, and also allows the VAP to be run at sites beyond SGP, something that has not been done with AERIPROF. Finally, the optimal estimation methodology produces uncertainties along with the thermodynamic profiles.

Implementing AERIOE as an operational VAP has been identified as a high priority for LASSO modeling activities. The retrieved LWP’s are a critical cloud property, and the thermodynamic profiles are part of the retrievals for the boundary facilities that will be used in forcing the model runs.

Development Tasks

Dave’s code is already well developed and he has run it robustly at multiple sites. Development for a few things is needed for our purposes.

1. One of the major challenges of running the code operationally is that the algorithm is computationally intensive so it will strain computing resources. Parallelization will be needed, but it may also tax DMF resources so the right computing environment will need to be chosen for operational and historical processing. (e.g. Currently it takes ~19.5 hours wall clock time to process 24 hours of native resolution data using all 32 processors on DMF machine Nickel, running 16 parallel processes.)
2. Decide on temporal resolution for operational processing. The cloud property retrievals require high temporal resolution (ideally the native ~20 second resolution). However, it is possible that the thermodynamic profile retrievals are better when averaging the data to a coarser resolution to reduce noise. In addition to accuracy, the chosen resolution substantially impacts the computational resources needed for the processing.
3. Report uncertainties as needed for data assimilation. Dave recommends scaling uncertainties by a profile-dependent factor to account for the lack of information content above the boundary layer.
4. Implement using Doppler lidar cloud base height rather than that from ceilometer at boundary facilities.
5. Port to ADI and follow ARM data standards.

References

Turner, D. D., and U. Löhnert (2013), Information Content and Uncertainties in Thermodynamic Profiles and Liquid Cloud Properties Retrieved from the Ground-Based Atmospheric Emitted Radiance Interferometer (AERI), Journal of Applied Meteorology and Climatology, 53(3), 752-771, doi:10.1175/JAMC-D-13-0126.1.

The scope of this ECR is to explore a new way of providing large scale data analysis and visualization services for ARM data. The current search for ARM data is performed by using its metadata, such as the site name, instrument name, date, etc. NoSQL technologies are being explored to improve the capabilities of data searching, not only by their metadata but also by using the data values. Few technologies have been explored for this purpose. However, the two that are currently being tested for ARM data are Apache Cassandra [noSQL database] and Apache Spark [noSQL based analytics framework]. Both of these technologies were developed to work in a distributed environment and hence can handle large data for storing and analytics. Cassandra nodes can be expanded or shrunk seamlessly depending on the data volume needs which is either the size of the data itself or number of concurrent users just accessing the data or running analysis on them. Spark can be used to calculate statistics and implement some machine learning techniques, as required. Storing data in a database makes it easier to access and therefore can be used by D3.js. D3.js is a JavaScript library that can generate interactive data visualizations in web browsers by making use of commonly used SVG, HTML5 and CSS standards.

One of the value added products [VAP] from ARM, armbeatm datastream, was first used for the assessment of the two noSQL tools. The data for this VAP was collected from 3 sites [SGP, NSA, TWP]. Each file had a year’s worth of data and there are about 10 to 15 files from each site. The primary measurements were stored as individual columns in Cassandra. A couple of interactive graphs, such as multiline plots and parallel coordinates, were created using d3.js. A few statistics were run on this datastream, such as minimum temperature during the past 10 years and correlation between temperature and relative humidity, using the Spark framework. The unreleased VAP product, siganalmwr, was also used to test interactive graphs generated using d3.js.

NoSQL can also be taken advantage of for the LASSO work. LASSO will require the ability for users to analyze both observations and LES model output either individually or together across multiple time periods. LASSO’s intent is to present the simulation output, corresponding observations, associated statistics, estimated uncertainties, and metadata in an accessible and easily used form by combining them into a unified package called a Data Cube. This cube will include both raw and processed simulation information. The example listed in the LASSO implementation strategy suggests that enormous data storage is required to store the above mentioned quantities, and noSQL can potentially provide a powerful means to store the data and subsequently provide efficient user access via tools such as Spark and D3.js.

First, we propose to explore efficient ways to store the multidimensional LASSO data using noSQL. This will involve understanding requirements for user access, such as how to handle the geospatial nature of LASSO data. We will also estimate the data storage needs and associated hardware necessary to accommodate LASSO. Second, we will investigate means to analyze the LASSO data stored within the noSQL database. This will require designing queries that can merge disparate data with different temporal and spatial characteristics. It will also involve determining a minimum set of requirements for analysis needs that would need to be implemented to make the noSQL approach useful, along with optional additions that would add value if they were to be implemented. Third, we will investigate ways to interface the noSQL database to the ARM website to make it easier for users to do basic analyses. This would most likel

The LES ARM Symbiotic Simulation & Observation (LASSO) pilot project is testing two LES models, and possible configurations of them, for doing routine LES runs at SGP. This engineering request encompasses the portions of LASSO that involve the modeling workflow and tasks oriented toward making a final model selection. Pieces of this work include configuring the models, comparison of the models to identify pros and cons of each, and the software needed to automate running the models. LASSO can be viewed as producing a series of ARM products that when strung together provide the tools necessary to generate ongoing LES simulations and accompanying metrics and diagnostics. This engineering request focuses on producing the model output from the LES models. The workflow will integrate pieces from other engineering tasks, such as using the cloud classification product being developed in ECO-01245 and LES forcing generation (ENG0003092). Output from this workflow will be passed to the LASSO Data Cube (ENG0001246). Model costs will also be estimated for use in Computing Infrastructure (ENG0001061).

This ECR can be viewed as two primary pieces. The first is choices surrounding configuring the WRF and SAM LES models. This will involve evaluating appropriate physics choices, domain size, grid spacing, boundary condition methodology, and forcings to be used on an ongoing basis. This will be done using a series of test simulations that compare results for a number of cases to determine optimal choices. Most testing will be done at the OLCF and NERSC computing facilities while ARM is in the process of purchasing their own high-performance computing cluster.

The second piece of this ECR is the software development to automate the models. The pilot project will provide the steps that need to be automated to run the models. This will involve working with ARM software developers to provide them with the knowledge they need to automate the modeling process within the ARM computer ecosystem. The code will be designed to work both within ARM's computers and at other DOE computing facilities, which will both assist with the model testing and provide a means for LASSO users to reproduce the LASSO simulations and variants of them as necessary to further their research.

Potential reviewers to include on this ECR: Jennifer Comstock William Gustafson Andy Vogelmann Chitra Sivaraman Giri Palanisamy David Troyan ARM representative for working with Globus and high-speed offsite file transfers

The SACR ADVance Velocity-Azimuth Display (SACR-ADV-VAD) product is expected to complement infrequent soundings by providing profiles of horizontal wind speed and direction in-cloud every time the SACR operates the Hemispherical Sky Range-Height Indicator scan strategy, which is typically every 30 to 60 minutes.

The Velocity Azimuth Display (VAD) technique was historically used to retrieve wind field properties using weather radars (Lhermitte and Atlas, 1961). This VAP relies on SACR radial mean Doppler velocity observations from the SACRCORR VAP which are corrected for aliasing, noise, insects and second trip echo.

This VAP outputs a daily NetCDF file containing profiles of cloud level horizontal wind speed and direction derived every time the HS-RHI scan strategy is performed (this could result an irregular time stamp). This product is produced at a 50 m height resolution.

SACR-ADV-QVP provides a quasi-vertical representation of azimuthal averaged polarimetric variables such as differential reflectivity (ZDR), specific differential phase (KDP) and correlation coefficient (rho hv), and linear depolarization ratio (LDR) to reveal important signatures in ice and mixed-phase clouds. SACR-ADV-QVP are produced using a sequence of plan-position-indicator (PPI) scans.

PPI scans are constant elevation with changing azimuth scans. This scan strategy typically covers 3 different elevation angles and is repeated every 30 minutes.

The scientific basis of the algorithm is described in Ryzhkov et al. (2016). For the three elevation PPIs, observations at constant range are averaged over azimuth angles. The standard deviation over azimuth angles is used as an error estimate.

The SACR2CORR VAP generates the main inputs of this VAP. They include corrected reflectivity, differential reflectivity, spectrum width, differential phase, linear depolarization ratio, and correlation coefficient all of which are free of noise, insects and second trip echo and all of which are in native radar polar coordinates.

This VAP outputs a daily NetCDF file containing quasi-vertical profiles of all aforementioned polarimetric variable every time the PPI scan strategy is performed (this could result an irregular time stamp). This product is produced at a 10 m height resolution at an elevation angle of 20 degrees.

Remote sensors provide continuous high-frequency high-resolution upper air profiles, especially in the boundary layer. To support high resolution modeling studies such as the Large-Eddy Simulation (LES) ARM Symbiotic Simulation and Observation (LASSO) workflow, SGP has been reconfigured to create a "megasite" with a network of boundary sites (planning 4 sites) around the SGP central facility that provides continuous profiles of temperature, humidity and wind from remote sensors. The newly established remote sensors which can provide vertical profiles for the variational analysis include: • Atmospheric Emitted Radiance Interferometer (AERI): temperature and humidity profiles for the lowest couple kilometers (~ 3km). The products planned to be used in variational analysis are temperature and humidity retrievals using the Optimal Estimation Technique (AERIoe) (http://www.arm.gov/data/pi/96). An updated version of AERIoe VAP is currently ready. • Doppler lidar: horizontal and vertical winds in the boundary layer. The products planned to be used in variational analysis are derived wind profiles using velocity-azimuth-display algorithm (http://www.arm.gov/data/eval/86).

The goal of this task is to develop a method to incorporate measurements from the new ARM boundary layer profiling network, particularly the remote sensing data (e.g. AERI, radar/lidar profiles), into variational analysis to support the high-resolution modeling studies. Data retrieved from remote sensing measurements could be used to as background data in the variational analysis. However, the background data from NWP analysis is still necessary since most of these remote sensors are limited to the lowest couple kilometers and/or are not reliable when precipitation exists. Previous AERIoe VAP uses climatology value in upper troposphere where remote sensing profiles are unreliable. Our preliminary work shows that this will cause bias in the large-scale forcing data because the climatological profiles do not capture the features of synoptic systems. An updated AERIoe algorithm, which used the Rapid Refresh (RAP) analysis (updated version of RUC after May 2012) as the background field for upper levels (the same as what is used in the variational analysis), is being tested. The new AERIoe product will have a better representation of the large scale state in the upper levels and improve the consistency with the variational analysis.

Another potential problem to merging analysis data and remote sensing profiles is the discontinuity at the level where remote sensing profiles become unreliable. We plan to use a vertically-changing weighting function to combine remote sensing profiles with analysis data in which remote sensing profiles will have large weight in boundary layer when they are reliable and small weight in upper layer when they are unreliable.

We plan to use the data from the remote sensor sites at SGP reconfigured in 2016 to derive the forcing data. Since the reconfigured boundary layer profiling network is much smaller than the default 300 km × 300 km domain used in traditional continuous forcing, the new algorithm will be re-designed for a smaller domain of 75 km × 75 km.

To fully use the advantages of the boundary layer profilers, we will run variational analysis with higher resolutions comparing to the standard run: 1-hourly in temporal and 10-mb in vertical. A set of experiments are designed to test the sensitivity of the forcing data to different resolution and background data: Experiment 1 will be the control run which is RAP only in 1 hr and 10 mb resolution. Experiment 2 is RAP + AERIoe in 1hr and 10 mb resolution Experiment 3 is RAP + AERIoe (with different merging method) in 1 hr and 10 mb resolution Experiment 4 is RAP + AERIoe in 3 hr and 10 mb resolution. Experiment 5 is RAP + AERIoe in 1 hr and 25 mb resolution. After these experiments, one method will be chosen for the future development of large-scale forcing data with boundary layer remote sensing profiles.

The OGRE-CLOUDS framework will build upon past efforts by the PI and Co-I to: 1) produce vertically resolved cloud and precipitation properties (with accompanying uncertainties) in the column above an ARM site under all cloud conditions with 2) the ability to implement new, conditional (i.e., applicable under specified cloud conditions) retrieval techniques and 3) a diagnostic package for comparison and evaluation of the new retrieval. We will demonstrate the use of this framework by implementing two state-of-the-art retrieval algorithms.

The development activities will include:

1) The use of an improved version of the MICROBASE algorithm as the underlying background for the new framework. The current MICROBASE algorithm will be updated to fully integrate it into the ARM Data Integrator (ADI) framework, include regular unit testing, quantification of uncertainties (incorporating the work of Zhao et al. 2013) and improved modularity to facilitate testing of new algorithms.

2) Implementation and testing of state-of-the art retrieval algorithms for ice clouds (Szyrmer et al. 2012) and drizzling clouds (O'Connor et al. 2005; Luke and Kollias 2013). This implementation will include the need for a "Cloud Condition Identification" algorithm and dataset, which is retrieval dependent, for quantitatively defining the conditions under which each new retrieval is applicable.

3) A pathway towards four diagnostic components including radiative closure using the ARM BBHRP framework, comparison to existing retrievals and in situ observations, and forward modeling of independent instrument observations.

4) The new framework will be developed within the construct of the ADI, hosted in GitHub, and made available as a resource for the community to facilitate developing and testing new retrieval algorithms.