Community Webinar

Tuesday, March 18 | 10:00-11:00 PT / 13:00-14:00 ET

Register Now

Join this webinar to learn about ESS-DIVE’s involvement in DOE Biological and Environmental Research (BER) bio-eco data integration efforts. The webinar will include updates on user research and work toward global search with other BER data systems. We will also discuss ESS project use cases linking ESS-DIVE datasets to related resources to improve findability, accessibility, and reusability of our interdisciplinary data. There will be time for participant questions, feedback and discussion on BER bio-eco integration.

During the webinar, we will cover the following topics:

• Plans for global search across BER data systems – Joint Genome Institute (JGI), Environmental Molecular Sciences Laboratory (EMSL), National Microbiome Data Collaborative (NMDC), DOE Systems Biology Knowledgebase KBase), ESS-DIVE;
• Use cases linking ESS-DIVE datasets to associated papers, source data in synthesis studies, other related BER data, workflows, and more; and
• How to cite BER data.

Please encourage anyone from your project who may be interested to attend.

For those interested in microbiome data, there is also an exciting opportunity to apply to become an NMDC Ambassador. Click here for more information and to apply. Application deadline: February 28, 2025.

Joan Damerow, joandamerow@lbl.gov
Community Engagement Lead, Research Scientist
Joan is an environmental scientist with a diverse background in environmental monitoring and remediation, freshwater ecology, and biodiversity informatics. As the Community engagement lead Joan leads engagement activities for ESS-DIVE, including webinars, our community data workshop, and is active in relevant conferences and data working groups (e.g. ESIP, RDA, AGU). She also leads activities related to samples data management and linking related data across BER data repositories. Joan is interested in interdisciplinary data management and tracking, and works with our community of data contributors to identify, develop, and implement practical data standards in ESS-DIVE that support FAIR principles.

Dylan O’Ryan, dporyan@lbl.gov
Environmental Data Research Associate

Dylan is a research associate at Lawrence Berkeley National Lab. As a member of ESS-DIVE’s Community Engagement Team, Dylan leads the dataset publication review process and the adoption and development of community reporting formats. He is also a member of the Watershed Function SFA Data Management and Integration Team, where he focuses on field-data integration. Dylan holds a B.S. in Environmental Studies from Sacramento State and is currently an M.S. student in the Environmental Geosciences program at CSU East Bay.

We are excited to announce the latest ESS-DIVE updates, including:

• Request Community Data Curator Support
• Reminder: File Level Metadata v1.1.0 is now live and available on GitHub and GitBook
• Save the date for our Webinar on BER Integration, March 18, 2025
• Updates to website and features
• Publication service interruption expected Spring 2025
• Metrics Summary

---

Request Community Data Curator Support

One of our new Community Fund Projects is the ESS Community Data Curator. Kim Ely (LBNL) will assist ESS-DIVE data contributors in how to curate high-quality datasets using ESS-DIVE reporting formats, improve data curation workflows, and provide training or advice regarding project data management. 

The ESS Community Data Curator will be available through September 2025 when the project ends. Take advantage of this opportunity for ESS Community Data Curator now while you can.  

To request ESS Community Data Curator support, use the ESS-DIVE Contact Us form and select the “Community Data Curator” category, or email ess-dive-support@lbl.gov directly and provide details about what you need help with.

Reporting Format: File Level Metadata (FLMD) v1.1.0

As a reminder, version 1.1.0 of the File Level Metadata reporting format was released in September 2024, which addresses feedback from early adopters in the community. Importantly, version 1.1.0 is backward compatible with the original version (version 1.0.0), and as such, the Fusion Database can read and parse file level metadata (flmd) and data dictionary (dd) files following both versions.

Save the Date: ESS-DIVE Webinar on BER Integration – March 18

Save the date to attend our webinar on March 18, 10-11 am PT, which will provide the latest on work within ESS-DIVE and other BER data systems toward BER integration. We will send a webinar announcement soon.

Website and Feature Updates

Status Page

• Visit ESS-DIVE’s new status page to see if there are any scheduled service outages and plan your data publication timeline: https://ess-dive.lbl.gov/status/.

Dataset API

We have released enhancements to the Dataset API, ESS-DIVE’s programmatic tool for searching, submitting, and sharing data. Try out the Dataset API with our easy-to-use tutorials that demonstrate its various applications.  

• Bug Fixes:
  • Addressed access issues when programmatically requesting publication
  • Corrected error in dataset citations reported by the Dataset API
  • Fixed DOI and dataset statuses reported for older dataset versions
  • Enables view of dataset permissions for previous dataset versions

New search tool: Deep Dive API!

• The Deep Dive API enhances data discovery in ESS-DIVE by searching within data files. 
• Not all files are searchable using the Deep Dive API. Datasets must use ESS-DIVE reporting formats and be successfully parsed by the Fusion Database. See our documentation for details.
• Try out this search tool with our complete Deep Dive API tutorial
  (hint: launch the tutorial directly in your browser with the Google Collab button)

Publication service interruption expected Spring 2025

ESS-DIVE’s underlying publication management services will be interrupted in Spring 2025 due to the DOE’s Office of Science and Technical Information (OSTI) services being unavailable. ESS-DIVE manages DOI’s through OSTI and we will not be able to reserve DOIs or publish datasets during this service interruption. 

If you need to publish a dataset during Spring 2025, we recommend submitting publication requests and reserving DOIs sooner rather than later to allow time for the complete review cycle. All other ESS-DIVE services will be available. 

Stay tuned for more information on this upcoming publication outage.

Metrics Summary

ESS-DIVE’s Quarterly Metrics Summary captures our user numbers to-date. The percent growth covers the period between January 2024 and January 2025.

We are pleased to share that Deb Agarwal, who served as ESS-DIVE’s Lead PI from its inception in 2017 to July 2023, was named a 2024 AGU Fellow. This award recognizes Deb’s “pioneering user-oriented data infrastructure and value-added products.” Deb now serves as a Senior Advisor to ESS-DIVE.

Deb was awarded at the 2024 AGU Annual Meeting in Washington D.C. on December 11th, 2024. To learn more about Deb’s recognition as a 2024 AGU Fellow, read the article from the Lawrence Berkeley National Laboratory’s Computing Sciences Research Division, including interviews from Deb and Charuleka Varadharajan (ESS-DIVE Co-PI).

Photo Credit: Diane Pizza; Originally shared in Troutman, 2024 (LBNL Computing Sciences Article)

Photo Credit: Originally shared in Troutman, 2024 (LBNL Computing Sciences Article)

Photo Credit: Diane Pizza; Originally shared in Troutman, 2024 (LBNL Computing Sciences Article)

The ESS-DIVE team is looking forward to participating in the 2024 AGU Annual Meeting in Washington D.C. Several members of the ESS-DIVE team will be in-person throughout the meeting and hope to meet some of you there. Come visit us at one of the booths, or presentations listed below!

Booths

LBL Earth and Environmental Sciences Area (EESA) Booth
Location: Walter E. Washington Convention Center, Exhibits Hall
Team Member(s): Joan Damerow, Dylan O’Ryan, & Emily Nagamoto
Date and Time: Wednesday, December 11th; 14:00 – 15:00 ET

Oral Presentations

ESS-DIVE: Building Scalable, Resilient, Interoperable Infrastructure for Data Repositories (IN11A-06)

Presenter: Charuleka Varadharajan
Presentation Type: Oral
Session Date and Time: Monday, December 9th; 09:20 – 09:30 ET
Session Number and Title: IN11A: Collaboratively Advancing Trust in Data Repositories and Integrated Infrastructure to Enable Interdisciplinary Research, Applications, and Uses of Open Data I Oral
Presentation Location: Marquis 3-4 (Marriott Marquis)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Paper/1651870

Abstract

Shreyas Cholia1, Charuleka Varadharajan1, Valerie C. Hendrix1, Joan Damerow1, Emily Robles1, Deborah Agarwal1, Madison Burrus1, Danielle Christianson1, Hesham Elbashandy1, Mario Melara1, Fianna O’Brien1, Dylan O’Ryan1, Sarah Poon1, Shalki Shrivastava1, Karen Whitenack1, Catherine Wong1, Matthew B Jones2, Jing Tao2, Matthew Brooke2, Rushiraj Nenuji2, Jeanette Clark2, (1) Lawrence Berkeley National Laboratory, (2) National Center for Ecological Analysis and Synthesis

The Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) is a repository storing observation, simulation and derived data products generated by research funded by the Department of Energy’s Environmental System Science program. Given the breadth and volumes of DOE research data, ESS-DIVE addresses the increasing demand for robust data management systems that can handle complex environmental data spanning diverse, heterogenous formats and large volumes.

The infrastructure is built to support a wide range of data standards and reporting formats, aligned with the FAIR data principles. ESS-DIVE employs standardized metadata formats and APIs that promote compatibility and ease of use, facilitating data integration across different platforms and research communities. Leveraging these data formats, we developed a novel Fusion Database – a service that extracts, indexes, and serves data from within standardized files, to enable advanced search beyond metadata.

We are pioneering the development of a scalable, resilient, and interoperable infrastructure to support growth in data volumes with a tiered storage model. The primary storage layer manages smaller datasets and all metadata, ensuring verification, data lineage, and redundancy, The Tier 2 storage layer enables long-term archiving of large data (multi-terabyte) for scaling and cost-effectiveness. It provides high performance access to data, directly from the underlying storage volumes through Globus and web interfaces, while the associated metadata are maintained and indexed in the primary tier.

As a long-term archive, resilience is another cornerstone of the ESS-DIVE infrastructure. The system is architected to provide high availability through the use of reproducible, containerized Docker/Kubernetes microservices that can be deployed at multiple sites. We have automated backup and failover processes to facilitate resilient deployments. ESS-DIVE is also a member of the DataONE federation which ensures that our primary metadata and data are automatically replicated to other DataONE sites.

We believe that our approach can serve as a useful model for data repository management, enabling a path towards scalable infrastructure for long-term data preservation.

A scalable and community-centered repository workflow for dataset review in support of FAIR data (IN12A-03)

Presenter: Joan Damerow
Presentation Type: Oral
Session Date and Time: Monday, December 9th; 10:40 – 10:50 ET
Session Number and Title: IN11A: Collaboratively Advancing Trust in Data Repositories and Integrated Infrastructure to Enable Interdisciplinary Research, Applications, and Uses of Open Data I Oral
Presentation Location: Marquis 3-4 (Marriott Marquis)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Paper/1711551

Abstract

Emily Robles1, Dylan O’Ryan1, Deborah A. Agarwal1, Kathleen Beilsmith2, Ben Bond-Lamberty3, Kristin Boye4, Madison Burrus1, Danielle S. Christianson1, Michael Crow5, Robert Crystal-Ornelas1, Joan Damerow1, Hesham Elbashandy1, Kim S. Ely1, Brieanne Forbes3, Amy E. Goldman3, Susan Heinz5, Valerie C Hendrix1, Zarine Kakalia1, Kayla Mathes8, Mario Melara1, Fianna O’Brien1, Drew Paine1, Stephanie C. Pennington3, Sarah Poon1, Lavanya Ramakrishnan1, Alistair Rogers1, Shalki Shrivastava1, Maegen Simmonds1, Terri Velliquette5, Pamela Weisenhorn2, Jessica Nicole Welch5, Karen Whitenack1, Catherine Wong1, Shreyas Cholia1, Charuleka Varadharajan1, (1) Lawrence Berkeley National Laboratory, (2) Argonne National Laboratory, (3) Pacific Northwest National Laboratory, (4) SLAC National Accelerator Laboratory, (5) Oak Ridge National Laboratory, (7) Pacific Northwest National Laboratory, (8) Integrated Life Sciences, Virginia Commonwealth University

Open data are needed to support scientific research broadly; however the quality of data and associated metadata in repositories remain limiting factors for discoverability and usability. Existing standards and guidelines to format data and metadata can be difficult and time intensive to follow, are highly diverse, and often lack clear incentives for data contributors.

With the goal of lowering the barriers to generating Findable, Accessible, Interoperable, Reusable (FAIR) data and metadata, the U.S. Department of Energy’s Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) data repository partnered with researchers to create 12 community standards and templates for a diverse range of Earth science data and metadata types. We developed both high-level (e.g., file-level metadata, CSV guidelines, sample metadata) and data-type specific (e.g., hydrologic monitoring, soil respiration, leaf physiology) reporting formats. In particular, the file-level metadata reporting format is used to describe details of individual files within a dataset, while the CSV reporting format outlines best practices for formatting tabular data to be machine readable, and enable automated validation and parsing.

We have developed automated quality-check tools that validate the completeness of files according to the file-level metadata and CSV reporting format requirements, and parse tabular data files with variables that have been formatted correctly. Automation enables the data publication review process to scale with repository growth. In addition, the adoption of reporting formats has enabled advanced search capabilities and provides the foundation for future data integration tools. Notably, we developed a Deep Dive API which provides advanced search within data files across all ESS-DIVE public datasets that have been parsed and validated by our automated checks. Users can find relevant data points and variables within individual files of a dataset, facilitating more efficient data discovery for scientific workflows than dataset-level search alone. Future development of automated tools to help data contributors standardize their data and metadata will continue to lower the barriers to generating well-curated, rapidly reusable, FAIR data.

Town Halls

Complex Citations: Ensuring Transparency, Reproducibility, and Credit for All Supporting Research Contributions (TH15N)
Moderators: Shelley Stall, Lesley A Wyborn, Martina Stockhause, Joan Damerow, Justin James Henry Buck, James Ayliffe
Date and Time: Monday, December 9th; 18:00 – 19:00 EST
Location: 209 A-C (Convention Center)
Townhall URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/229536 

An ongoing challenge relevant to most research disciplines is the difficulty in citing 100+ digital objects such as datasets, software, samples, and images. Journals require authors to place citations over some set limit into supplemental information, where individual citations are not properly indexed, not linked to the manuscript, nor tracked accurately. Citing these research products is critical to enable transparent and reproducible research and for researchers, institutions, and project managers to trace citation, get appropriate credit, and report impact to funders. 

We propose the term ‘reliquary’ to describe all the datasets, software, or other digital objects that support research findings for a specific paper. A reliquary may contain hundreds to millions of objects, often created by different groups and preserved in different repositories.  We need to develop a scalable implementation strategy to enable researchers to use this type of citation and allow integration into common citation/impact metrics.

This Town Hall is relevant to all Sections of AGU and beyond including researchers, repositories managers, infrastructure builders, journal staff, and indexers. The work is related to the new working group “Complex Citations” of the Research Data Alliance.

A Unified Earth Science Data Infrastructure to Achieve a New Level of Science: Opportunities and Challenges (TH45H)
Moderators: Kerstin Kleese van Dam, Ben P Bond-Lamberty, Giri Prakash, Charuleka Varadharajan, Nicki Linn Hickmon
Date and Time: Thursday, December 12th; 18:00 – 19:00 EST
Location: 209 A-C (Convention Center)
Townhall URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/228926 

Over the past year, scientists across the DOE National Laboratories convened a series of meetings to discuss solutions to a set of science challenges spanning the intersections of genomics, metagenomics, microbial community dynamics, hydrobiogeochemistry, atmosphere-land interactions, and earth system modeling.  The team focused on development of a unified data infrastructure as part of a 10-year vision that integrates science with all-source data derived from, e.g., observations, experiments, simulations, and AI-informed information across existing and future data archives. This town hall will provide a status of progress towards the decadal vision, updates of the science opportunities and early success stories, and barriers that need to be overcome.  Panelists will emphasize the value of linking distributed resources, reproducibility, and interoperability in scientific research. Critical to the discussion is the need to break down the barriers to data access and usability for the broader interdisciplinary community. The session will explore both the opportunities that arise from these unified approaches and the challenges that persist, including data harmonization, privacy, and the scaling of technologies. By bringing together experts and stakeholders, the Town Hall will foster a collaborative environment to refine strategies and accelerate the implementation of effective, community-wide data solutions in the earth sciences.

Convened Sessions

Accelerating the Model-Experiment Cycle Using Artificial Intelligence and Advanced Technologies I Oral (IN21A)

Conveners: Charuleka Varadharajan, Pamela Weisenhorn, Paul E Bayer, Justin Jay Hnilo, Yuan-Heng Wang
Date and Time: Tuesday, December 10th; 08:30 – 10:00 EST
Location: Marquis 12-13 (Marriott Marquis)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/240540 

Accelerating the Model-Experiment Cycle Using Artificial Intelligence and Advanced Technologies II Poster (IN23A)

Conveners: Charuleka Varadharajan, Pamela Weisenhorn, Paul E Bayer, Justin Jay Hnilo, Yuan-Heng Wang
Date and Time: Tuesday, December 10th; 13:40 – 17:30 EST
Location: Hall B-C (Poster Hall) (Convention Center)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/237283  

Of Stations, Sites, Sensors, and Samples: Collaboration, Innovation, and Stewardship of Field-Based Observations I Oral (IN42A)

Conveners: Stephanie Mullins Wingo, Joan Damerow, Natalie Raia, Rorie Edmunds, Lesley A Wyborn
Date and Time: Thursday, December 12th; 10:20 – 11:50 EST
Location: Liberty I-K (Marriott Marquis)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/240616

Of Stations, Sites, Sensors, and Samples: Collaboration, Innovation, and Stewardship of Field-Based Observations II Poster (IN43B)

Conveners: Stephanie Mullins Wingo, Joan Damerow, Natalie Raia, Rorie Edmunds, Lesley A Wyborn
Date and Time: Thursday, December 12th; 13:40 – 17:30 EST
Location: Hall B-C (Poster Hall) (Convention Center)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/228521

We are excited to announce that we have four new Community Fund Projects, led by Environmental System Science (ESS) researchers, to help advance DOE ESS data management and use. To learn more about these projects, join the fully virtual ESS-DIVE Community Data Workshop session next week on “Data Curation to Enable Reuse and Integration with Community Projects,” November 15th, 11:30 am – 1:30 pm PT. Our community partners will discuss their planned work, answer questions, and get your input. Visit the workshop event page for the full workshop agenda, and register here to attend virtually over Zoom.

The 2024 ESS-DIVE Community Fund Projects include:

ESS Community Data Curator

Kim Ely, LBNL

The ESS Community Data Curator project and role will assist ESS-DIVE data contributors in curating high quality datasets using ESS-DIVE reporting formats, improve data curation workflows, advise, and provide training for ESS-DIVE data contributors. To request ESS Community Data Curator support, use the ESS-DIVE Contact Us form and “Request data curation assistance” by selecting the Other option, and provide details for what you need help with.

A Workflow and Reporting Format for Processing Environmental Sensor Data and

Automated Generation of ESS-DIVE Compliant Metadata

Stephanie Pennington and Ben Bond-Lamberty, PNNL

This project will develop an environmental sensor data reporting format, which will include proposing standard guidelines for versioning and processing levels of sensor data, and an automated processing pipeline to handle sensor data and generate ESS-DIVE compliant sensor data and metadata. Reporting format development will involve surveying other major ESS projects using data loggers and environmental sensors to synthesize best practices and commonly used formats. This project will also update the existing Soil Respiration Reporting Format to interact with these sensor data processing new tools.

Harmonization and Increased Usability of ESS-DIVE’s Hydrologic Monitoring and Soil, Sediment, and Water Chemistry Reporting Formats

Amy Goldman and Brieanne Forbes, PNNL

This project will update and harmonize the ESS-DIVE Hydrologic Monitoring and Water-Soil-Sediment Chemistry reporting formats to improve machine readability and interoperability with ESS-DIVE’s features, and develop programmatic tools for data contributors and users to aid in use of the two reporting formats.

Improving Advanced Terrestrial Simulator (ATS) model data managing and archiving standards

Ethan Coon (ORNL) and Zhi Li (PNNL)

This project will prototype standardized procedures for managing and archiving model data for the Advanced Terrestrial Simulator (ATS) model. This project will enhance, optimize, and standardize the existing Model Data Archiving Guidelines reporting format; and improve ATS’s ability to generate standardized Model Data Archives (MDA), including implementing CSV-standard compliant ATS observation files and developing scripts for setting up, automating, and submitting MDAs to ESS-DIVE including the semi-automated generation of File Level Metadata and Data Dictionary files.