We are pleased to share that Deb Agarwal, who served as ESS-DIVE’s Lead PI from its inception in 2017 to July 2023, was named a 2024 AGU Fellow. This award recognizes Deb’s “pioneering user-oriented data infrastructure and value-added products.” Deb now serves as a Senior Advisor to ESS-DIVE.

Deb was awarded at the 2024 AGU Annual Meeting in Washington D.C. on December 11th, 2024. To learn more about Deb’s recognition as a 2024 AGU Fellow, read the article from the Lawrence Berkeley National Laboratory’s Computing Sciences Research Division, including interviews from Deb and Charuleka Varadharajan (ESS-DIVE Co-PI).

Photo Credit: Diane Pizza; Originally shared in Troutman, 2024 (LBNL Computing Sciences Article)

Photo Credit: Originally shared in Troutman, 2024 (LBNL Computing Sciences Article)

Photo Credit: Diane Pizza; Originally shared in Troutman, 2024 (LBNL Computing Sciences Article)

The ESS-DIVE team is looking forward to participating in the 2024 AGU Annual Meeting in Washington D.C. Several members of the ESS-DIVE team will be in-person throughout the meeting and hope to meet some of you there. Come visit us at one of the booths, or presentations listed below!

Booths

LBL Earth and Environmental Sciences Area (EESA) Booth
Location: Walter E. Washington Convention Center, Exhibits Hall
Team Member(s): Joan Damerow, Dylan O’Ryan, & Emily Nagamoto
Date and Time: Wednesday, December 11th; 14:00 – 15:00 ET

Oral Presentations

ESS-DIVE: Building Scalable, Resilient, Interoperable Infrastructure for Data Repositories (IN11A-06)

Presenter: Charuleka Varadharajan
Presentation Type: Oral
Session Date and Time: Monday, December 9th; 09:20 – 09:30 ET
Session Number and Title: IN11A: Collaboratively Advancing Trust in Data Repositories and Integrated Infrastructure to Enable Interdisciplinary Research, Applications, and Uses of Open Data I Oral
Presentation Location: Marquis 3-4 (Marriott Marquis)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Paper/1651870

Abstract

Shreyas Cholia1, Charuleka Varadharajan1, Valerie C. Hendrix1, Joan Damerow1, Emily Robles1, Deborah Agarwal1, Madison Burrus1, Danielle Christianson1, Hesham Elbashandy1, Mario Melara1, Fianna O’Brien1, Dylan O’Ryan1, Sarah Poon1, Shalki Shrivastava1, Karen Whitenack1, Catherine Wong1, Matthew B Jones2, Jing Tao2, Matthew Brooke2, Rushiraj Nenuji2, Jeanette Clark2, (1) Lawrence Berkeley National Laboratory, (2) National Center for Ecological Analysis and Synthesis

The Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) is a repository storing observation, simulation and derived data products generated by research funded by the Department of Energy’s Environmental System Science program. Given the breadth and volumes of DOE research data, ESS-DIVE addresses the increasing demand for robust data management systems that can handle complex environmental data spanning diverse, heterogenous formats and large volumes.

The infrastructure is built to support a wide range of data standards and reporting formats, aligned with the FAIR data principles. ESS-DIVE employs standardized metadata formats and APIs that promote compatibility and ease of use, facilitating data integration across different platforms and research communities. Leveraging these data formats, we developed a novel Fusion Database – a service that extracts, indexes, and serves data from within standardized files, to enable advanced search beyond metadata.

We are pioneering the development of a scalable, resilient, and interoperable infrastructure to support growth in data volumes with a tiered storage model. The primary storage layer manages smaller datasets and all metadata, ensuring verification, data lineage, and redundancy, The Tier 2 storage layer enables long-term archiving of large data (multi-terabyte) for scaling and cost-effectiveness. It provides high performance access to data, directly from the underlying storage volumes through Globus and web interfaces, while the associated metadata are maintained and indexed in the primary tier.

As a long-term archive, resilience is another cornerstone of the ESS-DIVE infrastructure. The system is architected to provide high availability through the use of reproducible, containerized Docker/Kubernetes microservices that can be deployed at multiple sites. We have automated backup and failover processes to facilitate resilient deployments. ESS-DIVE is also a member of the DataONE federation which ensures that our primary metadata and data are automatically replicated to other DataONE sites.

We believe that our approach can serve as a useful model for data repository management, enabling a path towards scalable infrastructure for long-term data preservation.

A scalable and community-centered repository workflow for dataset review in support of FAIR data (IN12A-03)

Presenter: Joan Damerow
Presentation Type: Oral
Session Date and Time: Monday, December 9th; 10:40 – 10:50 ET
Session Number and Title: IN11A: Collaboratively Advancing Trust in Data Repositories and Integrated Infrastructure to Enable Interdisciplinary Research, Applications, and Uses of Open Data I Oral
Presentation Location: Marquis 3-4 (Marriott Marquis)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Paper/1711551

Abstract

Emily Robles1, Dylan O’Ryan1, Deborah A. Agarwal1, Kathleen Beilsmith2, Ben Bond-Lamberty3, Kristin Boye4, Madison Burrus1, Danielle S. Christianson1, Michael Crow5, Robert Crystal-Ornelas1, Joan Damerow1, Hesham Elbashandy1, Kim S. Ely1, Brieanne Forbes3, Amy E. Goldman3, Susan Heinz5, Valerie C Hendrix1, Zarine Kakalia1, Kayla Mathes8, Mario Melara1, Fianna O’Brien1, Drew Paine1, Stephanie C. Pennington3, Sarah Poon1, Lavanya Ramakrishnan1, Alistair Rogers1, Shalki Shrivastava1, Maegen Simmonds1, Terri Velliquette5, Pamela Weisenhorn2, Jessica Nicole Welch5, Karen Whitenack1, Catherine Wong1, Shreyas Cholia1, Charuleka Varadharajan1, (1) Lawrence Berkeley National Laboratory, (2) Argonne National Laboratory, (3) Pacific Northwest National Laboratory, (4) SLAC National Accelerator Laboratory, (5) Oak Ridge National Laboratory, (7) Pacific Northwest National Laboratory, (8) Integrated Life Sciences, Virginia Commonwealth University

Open data are needed to support scientific research broadly; however the quality of data and associated metadata in repositories remain limiting factors for discoverability and usability. Existing standards and guidelines to format data and metadata can be difficult and time intensive to follow, are highly diverse, and often lack clear incentives for data contributors.

With the goal of lowering the barriers to generating Findable, Accessible, Interoperable, Reusable (FAIR) data and metadata, the U.S. Department of Energy’s Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) data repository partnered with researchers to create 12 community standards and templates for a diverse range of Earth science data and metadata types. We developed both high-level (e.g., file-level metadata, CSV guidelines, sample metadata) and data-type specific (e.g., hydrologic monitoring, soil respiration, leaf physiology) reporting formats. In particular, the file-level metadata reporting format is used to describe details of individual files within a dataset, while the CSV reporting format outlines best practices for formatting tabular data to be machine readable, and enable automated validation and parsing.

We have developed automated quality-check tools that validate the completeness of files according to the file-level metadata and CSV reporting format requirements, and parse tabular data files with variables that have been formatted correctly. Automation enables the data publication review process to scale with repository growth. In addition, the adoption of reporting formats has enabled advanced search capabilities and provides the foundation for future data integration tools. Notably, we developed a Deep Dive API which provides advanced search within data files across all ESS-DIVE public datasets that have been parsed and validated by our automated checks. Users can find relevant data points and variables within individual files of a dataset, facilitating more efficient data discovery for scientific workflows than dataset-level search alone. Future development of automated tools to help data contributors standardize their data and metadata will continue to lower the barriers to generating well-curated, rapidly reusable, FAIR data.

Town Halls

Complex Citations: Ensuring Transparency, Reproducibility, and Credit for All Supporting Research Contributions (TH15N)
Moderators: Shelley Stall, Lesley A Wyborn, Martina Stockhause, Joan Damerow, Justin James Henry Buck, James Ayliffe
Date and Time: Monday, December 9th; 18:00 – 19:00 EST
Location: 209 A-C (Convention Center)
Townhall URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/229536 

An ongoing challenge relevant to most research disciplines is the difficulty in citing 100+ digital objects such as datasets, software, samples, and images. Journals require authors to place citations over some set limit into supplemental information, where individual citations are not properly indexed, not linked to the manuscript, nor tracked accurately. Citing these research products is critical to enable transparent and reproducible research and for researchers, institutions, and project managers to trace citation, get appropriate credit, and report impact to funders. 

We propose the term ‘reliquary’ to describe all the datasets, software, or other digital objects that support research findings for a specific paper. A reliquary may contain hundreds to millions of objects, often created by different groups and preserved in different repositories.  We need to develop a scalable implementation strategy to enable researchers to use this type of citation and allow integration into common citation/impact metrics.

This Town Hall is relevant to all Sections of AGU and beyond including researchers, repositories managers, infrastructure builders, journal staff, and indexers. The work is related to the new working group “Complex Citations” of the Research Data Alliance.

A Unified Earth Science Data Infrastructure to Achieve a New Level of Science: Opportunities and Challenges (TH45H)
Moderators: Kerstin Kleese van Dam, Ben P Bond-Lamberty, Giri Prakash, Charuleka Varadharajan, Nicki Linn Hickmon
Date and Time: Thursday, December 12th; 18:00 – 19:00 EST
Location: 209 A-C (Convention Center)
Townhall URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/228926 

Over the past year, scientists across the DOE National Laboratories convened a series of meetings to discuss solutions to a set of science challenges spanning the intersections of genomics, metagenomics, microbial community dynamics, hydrobiogeochemistry, atmosphere-land interactions, and earth system modeling.  The team focused on development of a unified data infrastructure as part of a 10-year vision that integrates science with all-source data derived from, e.g., observations, experiments, simulations, and AI-informed information across existing and future data archives. This town hall will provide a status of progress towards the decadal vision, updates of the science opportunities and early success stories, and barriers that need to be overcome.  Panelists will emphasize the value of linking distributed resources, reproducibility, and interoperability in scientific research. Critical to the discussion is the need to break down the barriers to data access and usability for the broader interdisciplinary community. The session will explore both the opportunities that arise from these unified approaches and the challenges that persist, including data harmonization, privacy, and the scaling of technologies. By bringing together experts and stakeholders, the Town Hall will foster a collaborative environment to refine strategies and accelerate the implementation of effective, community-wide data solutions in the earth sciences.

Convened Sessions

Accelerating the Model-Experiment Cycle Using Artificial Intelligence and Advanced Technologies I Oral (IN21A)

Conveners: Charuleka Varadharajan, Pamela Weisenhorn, Paul E Bayer, Justin Jay Hnilo, Yuan-Heng Wang
Date and Time: Tuesday, December 10th; 08:30 – 10:00 EST
Location: Marquis 12-13 (Marriott Marquis)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/240540 

Accelerating the Model-Experiment Cycle Using Artificial Intelligence and Advanced Technologies II Poster (IN23A)

Conveners: Charuleka Varadharajan, Pamela Weisenhorn, Paul E Bayer, Justin Jay Hnilo, Yuan-Heng Wang
Date and Time: Tuesday, December 10th; 13:40 – 17:30 EST
Location: Hall B-C (Poster Hall) (Convention Center)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/237283  

Of Stations, Sites, Sensors, and Samples: Collaboration, Innovation, and Stewardship of Field-Based Observations I Oral (IN42A)

Conveners: Stephanie Mullins Wingo, Joan Damerow, Natalie Raia, Rorie Edmunds, Lesley A Wyborn
Date and Time: Thursday, December 12th; 10:20 – 11:50 EST
Location: Liberty I-K (Marriott Marquis)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/240616

Of Stations, Sites, Sensors, and Samples: Collaboration, Innovation, and Stewardship of Field-Based Observations II Poster (IN43B)

Conveners: Stephanie Mullins Wingo, Joan Damerow, Natalie Raia, Rorie Edmunds, Lesley A Wyborn
Date and Time: Thursday, December 12th; 13:40 – 17:30 EST
Location: Hall B-C (Poster Hall) (Convention Center)
Session URL: https://agu.confex.com/agu/agu24/meetingapp.cgi/Session/228521

We are excited to announce that we have four new Community Fund Projects, led by Environmental System Science (ESS) researchers, to help advance DOE ESS data management and use. To learn more about these projects, join the fully virtual ESS-DIVE Community Data Workshop session next week on “Data Curation to Enable Reuse and Integration with Community Projects,” November 15th, 11:30 am – 1:30 pm PT. Our community partners will discuss their planned work, answer questions, and get your input. Visit the workshop event page for the full workshop agenda, and register here to attend virtually over Zoom.

The 2024 ESS-DIVE Community Fund Projects include:

ESS Community Data Curator

Kim Ely, LBNL

The ESS Community Data Curator project and role will assist ESS-DIVE data contributors in curating high quality datasets using ESS-DIVE reporting formats, improve data curation workflows, advise, and provide training for ESS-DIVE data contributors. To request ESS Community Data Curator support, use the ESS-DIVE Contact Us form and “Request data curation assistance” by selecting the Other option, and provide details for what you need help with.

A Workflow and Reporting Format for Processing Environmental Sensor Data and

Automated Generation of ESS-DIVE Compliant Metadata

Stephanie Pennington and Ben Bond-Lamberty, PNNL

This project will develop an environmental sensor data reporting format, which will include proposing standard guidelines for versioning and processing levels of sensor data, and an automated processing pipeline to handle sensor data and generate ESS-DIVE compliant sensor data and metadata. Reporting format development will involve surveying other major ESS projects using data loggers and environmental sensors to synthesize best practices and commonly used formats. This project will also update the existing Soil Respiration Reporting Format to interact with these sensor data processing new tools.

Harmonization and Increased Usability of ESS-DIVE’s Hydrologic Monitoring and Soil, Sediment, and Water Chemistry Reporting Formats

Amy Goldman and Brieanne Forbes, PNNL

This project will update and harmonize the ESS-DIVE Hydrologic Monitoring and Water-Soil-Sediment Chemistry reporting formats to improve machine readability and interoperability with ESS-DIVE’s features, and develop programmatic tools for data contributors and users to aid in use of the two reporting formats.

Improving Advanced Terrestrial Simulator (ATS) model data managing and archiving standards

Ethan Coon (ORNL) and Zhi Li (PNNL)

This project will prototype standardized procedures for managing and archiving model data for the Advanced Terrestrial Simulator (ATS) model. This project will enhance, optimize, and standardize the existing Model Data Archiving Guidelines reporting format; and improve ATS’s ability to generate standardized Model Data Archives (MDA), including implementing CSV-standard compliant ATS observation files and developing scripts for setting up, automating, and submitting MDAs to ESS-DIVE including the semi-automated generation of File Level Metadata and Data Dictionary files.

The Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) data archive, based at Lawrence Berkeley National Laboratory, is thrilled to announce the 2024 ESS-DIVE Community Data Workshop on November 14th and 15th, 2024. This free and virtual event is a chance for members of our community of data contributors and data users to join forces, fostering the exchange of valuable insights and expertise on ESS data.

The theme this year will focus on “ESS Data Integration and Use”. The opening session will feature a keynote presentation from Kjiersten Fagnan, Chief Informatics Officer for the DOE Joint Genome Institute (JGI), on DOE’s Biological and Environmental Research (BER) data integration goals. We will then have community presentations from ESS Synthesis projects, who will present on their scientific goals, data management approach, needs and challenges. We will hold discussions and tutorial sessions on data discovery, integration, publication, curation, and use. Whether you are new to the community, or have been engaging with ESS-DIVE for a while, this is your opportunity to learn about the newest features of the ESS-DIVE data repository, and help drive the future of ESS data management, integration, and use.

By attending this workshop, you will gain access to important information and networking opportunities, offering you the following benefits:

• Enhance Data Discovery: Learn about ESS-DIVE tools for enhanced data discovery through search innovations. Learn about the Fusion Database and Deep Dive API for search within dataset files, and the planned search enhancements for data discovery. Participate in discussions led by the ESS Cyberinfrastructure Working Group on the needs for ESS data discovery.
• Advance BER ESS Data Integration: Learn about ongoing data integration efforts across DOE BER, ESS-DIVE, and projects. Hear from Kjiersten Fagnan on the DOE BER plans to enable global search across BER data systems. Participate in discussions on data search and integration needs across ESS-DIVE, JGI, EMSL (Environmental Molecular Sciences Laboratory), NMDC (National Microbiome Data Collaborative), KBase (Systems Biology Knowledgebase), Ameriflux, and ARM (Atmospheric Radiation Measurement). 
• Curate Data Enable Reuse and Integration: Meet the new ESS Community Data Curator and provide input on your project data curation needs. Learn about other new ESS-DIVE community projects, their planned updates and harmonization to existing reporting formats (hydrologic monitoring, water-soil-sediment chemistry, soil respiration), and a new reporting format and tools to handle and process environmental sensor data. 
• Network and Learn from Peers: Participate in a collaborative, interdisciplinary environment where you can learn from fellow scientists with a variety of expertise across data, computational, and domain sciences. Share experiences, challenges, and innovative solutions to advance open ESS data and reuse.

Register Now: Registration for the 2024 ESS-DIVE Community Data Workshop is now open, and encouraged for anyone who is part of an ESS project. While participants are encouraged to attend both days for the full experience, they are also welcome to join sessions of their interest. To secure your spot, simply complete this registration form. Once registered, you will receive a personalized Zoom link, granting you access to the event.

Each session will provide ample opportunities for participants to engage with the ESS-DIVE team, ask questions, share comments, and provide feedback. Before the event, registered attendees will receive quick instructions on preparing for tutorials and discussions to make the most of their experience.

To discover more about the workshop or ESS-DIVE in general, please visit the workshop event page or contact ess-dive-support@lbl.gov.

Don’t miss the chance to be part of this transformative event and take your ESS projects to new heights. Join us at the 2024 ESS-DIVE Community Data Workshop for two days of learning, networking, and innovation to advance ESS data integration and use.

We are excited to announce that ESS-DIVE’s publication services have been upgraded! We’ve been hard at work to make the publication experience on ESS-DIVE easier and improve functionality across the repository over the past year. Additionally, we are proud to announce that the latest File Level Metadata reporting format version is now live.

The latest upgrades are summarized here:

• New interface to request publication, reserve DOIs, and view publication statuses
• File Level Metadata v1.1.0 is now live and available on GitHub
• Dataset API can perform queried searches and bulk publication requests
• Alternate storage and upload methods to support large dataset

---

ESS-DIVE Upgrades

Intuitive new interface for publishing datasets!

Visit our documentation to learn more.

“Manage” permission is required to manage dataset publication. You must be a registered ESS-DIVE Data Contributor.

Transparent Dataset and DOI Status Badges

• Dataset landing pages now display two new icons for your dataset progress and DOI status.
• Statuses provide transparency on where you are in the publication process so you know what your next steps are to publish your dataset.
• See our documentation for a full review of available statuses.

Enhancements 

Removed publish year from header

• The top row of landing page headers no longer list the publication year and instead only include the dataset DOI and version ID.
• This update eliminates confusion between the metadata field and the status of your dataset.

Geographic Coordinate Validation in Editor

• The editor now more thoroughly prevents the entry of invalid geographic coordinates.
• This means no more bounding boxes with inverted latitude and longitude values or those that erroneously span the international dateline or include the poles. It’s a safeguard against common errors that can lead to issues in data interpretation and use.

Bug Fixes 

The following bugs were resolved in the latest release:

• Critical patch for compatibility with latest Chrome browser
• Dataset aggregation metrics displayed in portals have faster response times and more accurate loading messages
• Searching by publish date would raise error in search result window
• Contributor metadata field validation did not capture format issue when required last name was missing

---

Dataset API v1.11

Request publication and reserve DOIs in bulk with the API

• ESS-DIVE’s new publication enhancements are available with the API!
• Data contributors can now add publication endpoints to their workflows to programmatically request publication or reserve DOIs for multiple datasets at once.
• It will still be necessary to be responsive to revision requests via email and complete the publication process.
• Visit our technical documentation for endpoint schema and expected format.

Try out the Dataset API using Jupyter Notebook tutorials

• No coding experience required! Our tutorials are complete, ready-to-run tools that walk through the process of using the Dataset API from start to finish.
• Each tutorial is designed to demonstrate a particular application of the API. Browse our selection of notebooks to see if one fits your needs: https://github.com/ess-dive/essdive-tutorials
• Run the files on Jupyter Notebook locally or click the “launch | binder” to run them directly in your browser.

New R code example to submit many files at once

• ESS-DIVE has a new coding example that demonstrates how to submit a new dataset with metadata and many files at once in R.
• Our example code is intended for data contributors who are building scripts to submit data to ESS-DIVE. Unlike the tutorial notebooks, these have the experienced coder in mind.
• Download the code example from our GitHub repository: submit_metadata_with_multiple_files.r

Enhancements 

Dataset searches with the API are much improved

• There are multiple enhancements in this release that have improved the Dataset API’s search capabilities to make it easier and more informative.

  • Search for datasets by keyword

  • Dataset previews now include the DOI in search results

  • Preview the file sizes (in bytes) before download

  • Enables search by older versions of dataset metadata
    Adds the identifier for the next and previous version of dataset metadata to search results

Search parameter descriptions are easier to use

• ESS-DIVE’s technical documentation for the API is a useful complementary resource to our tutorials.
• The improved search documentation provides more details about available search parameters and format requirements to help you fit the tutorial search examples to specific use cases you are interested in.

---

Reporting Format: File Level Metadata (FLMD) v1.1.0

Version 1.1.0 (v1.1.0) of the File Level Metadata reporting format addresses usability challenges and feedback from early adopters in the community. Importantly, v1.1.0 is backward compatible with the original version (v1.0.0), and as such, the Fusion Database can read and parse file level metadata (flmd) and data dictionary (dd) files following both versions.

If you would like to suggest a change request for the File Level Metadata reporting format, please submit a GitHub issue using one of the templates we provide. If you have any questions, please email ESS-DIVE support at ess-dive-support@lbl.gov.

Standardized variable naming convention to snake_case

• The original FLMD field names used both underscores and capitalizations and did not follow a specific naming convention. In v1.1.0, we have updated the formatting of field names to follow snake_case guidelines to increase consistency and harmonize across the reporting formats. For example:
  • File_Name changed to file_name
  • Column_or_Row_Name changed to column_or_row_name
• No spelling or underscore placement has been revised.
• While templates and instructions have been updated to reflect this convention change, the original capitalization from v1.0.0 will still be accepted as long as spelling and use of underscores are consistent.

New terms for standard field

• The recommended standard terms for ESS-DIVE reporting formats in the file level metadata file can be used to specify a standard or format that a file follows. The standardized terms are important to identify specific files using ESS-DIVE reporting formats.

Removed excess optional fields

• We have removed the following optional fields from the documentation to streamline the creation of the file level metadata file:
  • UTC_offset
  • Contact
  • Date_Start
  • Date_End
  • Latitude
  • Longitude
  • Northwest_Latitude_Coordinate
  • Northwest_Longitude_Coordinate
  • Southeast_Latitude_Coordinate
  • Southeast_Longitude_Coordinate 
• The optional missing_value_code field, which can be used to specify the codes representing missing values, has been moved to the data dictionary file so that it can be used at the variable level.

---

More exciting upgrades! 

Globus: Upload large data with ease

• Do you need faster upload times and more memory to upload large data volumes? You may be interested in using Globus!
• Globus is a free, cloud-based transfer service that anyone can use to move data from your desktop or cloud endpoints to ESS-DIVE.
• We encourage anyone with more than 500GB of data, more than 100 flat files, or anyone experiencing slow local internet speeds to use Globus.
• Get in touch with us if you would like to explore this option (ess-dive-support@lbl.gov).
• Learn more about how ESS-DIVE supports large data in our documentation.