In 2019, ESS-DIVE funded a small number of community projects to work on critical data management needs. The focus was on developing standardized formats, vocabulary and metadata requirements that would be broadly adopted by the ESS community.
Each of these efforts involved extensive work to identify appropriate existing data/metadata standards, or develop new standards by considering input from other researchers within the DOE community who collect or use similar data. The work also involved developing crosswalks to other major standards relevant to the data type, and ultimately provided a recommendation for the standard of choice.
If you are interested in learning more about finalized data standards (reporting formats) developed by the Community Partners and ESS-DIVE, review the ESS-DIVE Data Reporting Formats page. Additionally, email ess-dive-support@lbl.gov if you are interested in suggesting additional data standards.
  |
In Progress: Unoccupied Aerial System (UAS) data and metadataKim Ely and Shawn Serbin, BNL This reporting format will provide metadata requirements and format guidelines for uploading data from small (≤ 25 kg) Unoccupied Aerial Systems (UASs) to the ESS-DIVE archive. This includes basic recording of core flight data (e.g. telemetry), when available, and primary flight metadata (e.g. flight location, flight paths, altitude, date, start/end time), basic UAS payload and instrumentation information (e.g. camera make and model), and the general data product and file metadata needed for interpreting and accessing derived UAS datasets (e.g. pixel resolution, spatial reference system). Our objective is to develop a community-standard basic metadata format that works for any UAS platform and instrument payload while providing some guidance on how to expand on this with advanced metadata levels and suggested data storage formats that are optimized for cloud access. |
   |
Generic standard for file-level metadata and comma-separated data filesRanjeet Devarakonda, Susan Heinz, and Terri Velliquette, ORNL This project developed two standards for data types important to the ESS community: 1) a generic standard for the structure of data stored in comma-separated values (CSV) text format files, and 2) a metadata schema to describe the contents, scope, and structure of each data file within the ESS-DIVE repository. File-level metadata is the description of individual files that are part of a larger collection. This metadata is fully consistent with, and augmented, the metadata collected to describe each data collection. Providing file-level metadata helps to provide granular information that will enable data users to search for and locate files that contain specific measured parameters or variables, to understand and compare the contents of each data file within an overarching data collection. File-level metadata Reporting Format and Comma-seperated Values (CSV) Reporting Format Documentation |
  |
Soil respiration data and metadataBen Bond-Lamberty and Stephanie Pennington, PNNL This project designed standards and tools for a soil respiration database. It built upon a prototype continuous soil respiration database (COSORE, see https://github.com/bpbond/cosore), in which individual contributed datasets are mapped to standardized data and metadata fields. This research project aimed to (i) identify and develop a standard format for soil respiration data important to the earth system science community, (ii) define the cross-walk between this standard and other soil respiration standards in use in the community, and (iii) test QA/QC and ingestion tools. |
|
Hydrological Monitoring DataAmy Goldman and Huiying Ren, PNNL The project was build upon existing standards and/or well-established best practices for hydrologic monitoring data including single-point-in-time and logged measurements. Variables included are pressure, river stage, temperature, and conductivity (electrical and specific conductance). The scope was expanded to include other variables such as those measured by aquatic sondes (e.g., dissolved oxygen, pH, oxidation reduction potential). |
|
Storage of leaf level gas exchange data and metadata filesAlistair Rogers and Kim Ely, BNL This research project identified and developed a standard format for storage of leaf level gas exchange data and metadata files. The aim of this work was to develop standardized formats, vocabulary, and metadata requirements for uploading data from leaf-level gas exchange to the ESS-DIVE archive. This included all measurements made with a portable gas exchange system such as survey style measurements of photosynthesis, respiration and stomatal conductance; response curves e.g. light response curves, CO2 response curves, and temperature response curves; curves used to determine stomatal slope and fitting of derived parameters e.g. Vc, max from photosynthetic CO2 response curves. It also included other derivations of key parameters e.g. derivations of Vc max from proxies such as “one-point” measurements or spectroscopy. |
 |
Water quality and soil/sediment samples data and metadata filesKristin Boye, SLAC The aim of this work was to develop standardized formats, vocabulary, and metadata requirements for uploading data from water, soil, and sediment samples to the ESS-DIVE archive. It built upon existing standards and/or well-established best practices for water sample data (e.g. EPA’s WQX) and geological/soil sample data (e.g. National Cooperative Soil Survey and other USDA/USGS recommended/employed formats) to develop ESS-specific standards and templates for data and associated metadata uploads to ESS-DIVE. Water-soil-sediment Chemistry Reporting Format Documentation |
 |
16S amplicon data products standardization and KBase integrationPamela Weisenhorn, ANL The aim of this work was to develop standardized formats, vocabulary, and metadata requirements for uploading 16S amplicon data products (specifically, operational taxonomic unit (OTU) and environmental sequence variant (ESV) tables) to the ESS-DIVE archive. It buildt upon existing standards and/or well-established best practices for this data product, which results from bioinformatic processing of 16S data. Multiple pipelines exist for the generation of this data product and appropriate capture of the metadata associated with the analysis is critical in determining the ability to combine or compare across data sets generated by different researchers for meta-analysis or modeling purposes. This work involved determining best practices for the preservation of data provenance and metadata on bioinformatic processing to enable end-users to determine the appropriateness of the data products for reuse. In addition to consulting broadly with the ESS funded university partners, this data standard was developed in consultation with the JGI, the National Microbiome Data Collaborative (NMDC), KBase, MG RAST, and other relevant stakeholders. |
