ESS-DIVE

Deep Insight for Earth Science Data

ESS-DIVE at AGU 2023

by

The ESS-DIVE Team is looking forward to participating in the 2023 AGU Fall Meeting in San Francisco, CA. The ESS-DIVE team will be in-person throughout the meeting and we look forward to meeting you in person. Come visit us at one of the help desks, or presentations listed below!

Booths and Help Desks

AGU Open Science & Data Help Desk
Location: Moscone Center, Exhibits Hall
Team Member(s): Joan Damerow
Date and Time: Tuesday, December 12th; 10:00 – 12:00 PST

DOE Biological and Environmental Research (BER) Program Booth
Location: Moscone Center, Exhibits Hall
Team Member(s): Charu Varadharajan
Date and Time: Tuesday, December 12th; 16:00 – 17:00 PST

LBL Earth and Environmental Sciences Area (EESA) Booth
Location: Moscone Center, Exhibits Hall
Team Member(s): Charu Varadharajan, Joan Damerow, & other ESS-DIVE team members
Date and Time: Wednesday, December 13th; 16:00 – 17:00 PST

Oral, eLightning, and Poster Presentations

Open Earth and Environmental Data Advance Scientific Discovery eLightning (H12P-0874)

Presenter: Shreyas Cholia
Presentation Type: Session Chair
Session Date and Time: Monday, 11 December 2023; 16:00 – 17:30 PST
Presentation Location: Moscone Center, South, Hall D; eLightning Theater IV, Hall D – South
Session URL: https://agu.confex.com/agu/fm23/meetingapp.cgi/Session/213275

 

Enabling Integration Across Diverse Environmental Systems Data (IN14B-04)

Presenter: Valerie Hendrix
Presentation Type: eLightning
Session Date and Time: Monday, 11 December 2023; 16:09 – 16:12 PST
Session Number and Title: IN14B: Open Earth and Environmental Data Advance Scientific Discovery eLightning
Presentation Location: Moscone Center, South, Hall D; eLightning Theater IV, Hall D – South
Session URL: https://agu.confex.com/agu/fm23/meetingapp.cgi/Paper/1365123

Abstract

Valerie C Hendrix1, Shreyas Cholia1, Charuleka Varadharajan2, Deb Agarwal3, Joan E Damerow3, Hesham Elbashandy3, Fianna O’Brien3, Emily Robles1, Mario Melara3, Madison Burrus1, Karen Whitenack3, Catherine Wong1, Shalki Shrivastava3, Sarah Poon3, Matthew B. Jones4, Jing Tao5, Matthew Brooke5 and Rushiraj Nenuji5, (1)Lawrence Berkeley National Laboratory, Berkeley, CA, United States, (2)Lawrence Berkeley National Laboratory, Earth and Environmental Sciences Area, Berkeley, CA, United States, (3)Lawrence Berkeley National Laboratory, Berkeley, United States, (4)University of California Santa Barbara, National Center for Ecological Analysis and Synthesis, Santa Barbara, CA, United States, (5)National Center for Ecological Analysis and Synthesis, Santa Barbara, United States

The Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) is a data repository for research sponsored by the U.S. Department of Energy’s Environmental System Science program. ESS-DIVE enables collection, storage, management, and sharing of a variety of observational, experimental, and modeling data in Earth and environmental sciences. The volume, complexity, and diversity of these interdisciplinary data present unique integration challenges. We discuss how ESS-DIVE approaches data integration across these datasets and with other data systems.

ESS-DIVE enables a systematic method for linking datasets from other recognized data providers directly in its metadata. This allows metadata to be searchable in ESS-DIVE, while referencing and linking out to externally managed data products in a standardized manner. In order to track and relate sample data across systems, we encourage the use of common standards for sample data identifiers, such as the International Generic Sample Number (IGSN). ESS-DIVE works closely with community partners to ensure adoption and consistency of these standards to promote interoperability and data integrity. Metadata in ESS-DIVE are published in a number of formats, including the JSON-LD format, which allows the data to be easily ingested and understood by external systems (eg. Google Dataset Search, OSTI, Data.gov etc.). ESS-DIVE offers support for project-specific features allowing researchers to collaborate and share data within their teams efficiently. Also provided are project data portals where related data are collected, cataloged, and easily accessible. ESS-DIVE supports a secondary storage layer to serve very large, hierarchical datasets. This allows users to directly browse and access large volumes of data over the web, as well as through high-performance data transfer mechanisms like Globus which can be used to efficiently move data between sites. Finally, ESS-DIVE is integrated with DataONE, a federation of interoperable data repositories facilitating open science and data discovery. This integration includes replicated data and metadata between ESS-DIVE and DataONE making it easier for users to access the data they need, regardless of where they are stored.

BASIN-3D: Addressing the challenges and opportunity of earth science data synthesis (IN31E-0707)

Presenter: Danielle Christianson
Presentation Type: Poster
Session Date and Time: Wednesday, December 13th; 8:30 AM – 12:50 PM PST
Session Number and Title: IN31E: Empowering Earth Science Data Use and Hydrologic Advancements: Showcasing Innovative Tools and Technologies for Broad User Communities Poster
Presentation Location: Moscone Center, South, Poster Hall A-C (Exhibition Level, South, MC)
Session URL: https://agu.confex.com/agu/fm23/meetingapp.cgi/Paper/1324860

Abstract

Danielle S Christianson, Lawrence Berkeley National Laboratory, Berkeley, United States, Valerie C Hendrix, Lawrence Berkeley National Laboratory, Berkeley, CA, United States and Charuleka Varadharajan, Lawrence Berkeley National Laboratory, Earth and Environmental Sciences Area, Berkeley, CA, United States

By some estimates, researchers spend up to 80% of their time finding, accessing, validating, and/or transforming data for analysis (EC, 2019). BASIN-3D (Broker for Assimilation, Synthesis and Integration of eNvironmental Diverse, Distributed Datasets) is data synthesis software designed to reduce this barrier by synthesizing diverse data from a variety of remote and/or local data sources on demand without the need for additional storage. BASIN-3D is unique in that it fully synthesizes formats, units, and semantics by mapping data source vocabulary and formats to a common data model. Thus, researchers use a single BASIN-3D query language to request data from various sources and receive synthesized results in common formats (e.g., hdf5, Pandas dataframe, json). BASIN-3D is available as a suite of Python Libraries (https://github.com/BASIN-3D) for customizable integration into python-based applications like Jupyter Notebooks and Django web frameworks.

In this presentation, we describe new capabilities and data source connections with lessons learned. For example, we overhauled the vocabulary mapping strategy to enable complex- and multi-mapping between data source terminology and that of BASIN-3D. While this feature enables a greater diversity of data to be synthesized, it also meant researchers wanted more details about the translation to validate appropriateness for their use. Additionally, we demonstrated synthesis of individual datasets from Department of Energy’s ESS-DIVE repository whose data follow a community-developed data reporting format. This demo has shown individual research teams the improved potential to have their data reused by others if their data are formatted in a community-adopted standard. It also enabled us to understand the requirements of building generalized connections to a variety of well-structured data sources.

EC: European Commission, Directorate-General for Research and Innovation, 2019. Cost-benefit analysis for FAIR research data – Cost of not having FAIR research data, Publications Office, https://data.europa.eu/doi/10.2777/02999

Need for Interdisciplinary, Collaborative Data Management on Earth Science Data Repositories (IN41C-0600)

Presenter: Madison Burrus
Presentation Type: Poster
Session Date and Time: Thursday, December 14th; 8:30 AM – 12:50 PM PST
Session Number and Title: IN41C: Fostering Wide-Open Science by Improving Collaboration, Innovation, Attribution, Identity, and Ethics in Data Repositories and Data (Re)publication I Poster
Presentation Location: Moscone Center, South, Poster Hall A-C (Exhibition Level, South, MC)
Session URL: https://agu.confex.com/agu/fm23/meetingapp.cgi/Paper/1325100

Abstract

Madison Burrus1, Fianna O’Brien2, Deb Agarwal2, Joan E Damerow2, Hesham Elbashandy2, Valerie C Hendrix1, Matthew B. Jones3, Mario Melara2, Dylan O’Ryan2, Emily Robles1, Sarah Poon2, Shalki Shrivastava2, Karen Whitenack2, Shreyas Cholia1 and Charuleka Varadharajan1, (1)Lawrence Berkeley National Laboratory, Berkeley, CA, United States, (2)Lawrence Berkeley National Laboratory, Berkeley, United States, (3)University of California Santa Barbara, National Center for Ecological Analysis and Synthesis, Santa Barbara, CA, United States

The US Department of Energy (DOE) sponsors numerous Earth and environmental science research activities through their Earth Systems Science (ESS) program. While the research data produced by this program are vast and diverse, there is a need to centralize resources to curate, archive and access interdisciplinary ESS-funded research data. The Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) data repository was established to be the central repository for ESS data. The goal was to create a space where existing and future research projects funded by ESS could easily publish data and report on progress.

An archival challenge facing ESS-DIVE and the broader environmental science community is that journal requirements and repository infrastructure tend to emphasize data archival at the individual contributor level, yet data products resulting from interdisciplinary scientific research are inherently generated from team contributions. At ESS-DIVE, we are working to accommodate the needs of ESS community projects. Based on community input, we expanded the ESS-DIVE repository to support project collaboration by introducing capabilities that enable team-based data publication, administration, and management.

We introduced several new project-centric data management features, including unique identifiers for projects, the ability to create teams, an administrative project data manager role, and a permission model for sharing dataset access with teams. Additionally, these project specific features were designed to compliment ESS-DIVE’s existing data management services. Projects can use ESS-DIVE’s programmatic web service API to upload and manage datasets in bulk, customize data portal websites to showcase data collections, and share dataset access to individuals as needed. In combination, the tools implemented at ESS-DIVE provide ESS projects a generalized, team-based data archiving experience throughout the publication process.

Here we showcase collaborative data publication tools developed by ESS-DIVE and designed for current and future DOE research projects. We discuss challenges, the need for collaboration to improve the design and development of these tools, and future work.

Linking and Citing Related Resources for Environmental System Science Research (IN41C-0606)

Presenter: Joan Damerow
Presentation Type: Poster
Session Date and Time: Thursday, December 14th; 8:30 AM – 12:50 PM PST
Session Number and Title: IN41C: Fostering Wide-Open Science by Improving Collaboration, Innovation, Attribution, Identity, and Ethics in Data Repositories and Data (Re)publication I Poster
Presentation Location: Moscone Center, South, Poster Hall A-C (Exhibition Level, South, MC)
Session URL: https://agu.confex.com/agu/fm23/meetingapp.cgi/Paper/1294212 

Abstract

Joan E Damerow1, Deb Agarwal1, Madison Burrus2, Shreyas Cholia2, Hesham Elbashandy1, Valerie C Hendrix2, Fianna O’Brien1, Emily Robles1, Mario Melara1, Dylan O’Ryan1 and Charuleka Varadharajan3, (1)Lawrence Berkeley National Laboratory, Berkeley, United States, (2)Lawrence Berkeley National Laboratory, Berkeley, CA, United States, (3)Lawrence Berkeley National Laboratory, Earth and Environmental Sciences Area, Berkeley, CA, United States

The study of natural ecosystems requires multidisciplinary science teams to understand and model multi-scale processes. For example, studies of organic matter cycling through plants and soil involves data and analysis to represent soil biogeochemistry, microbial communities, plant structures, and ecophysiological traits of specific organisms involved. All related data should be clearly linked to represent processes at a given site or across broader scales. When published, however, they are often missing information needed to find, access, integrate, and reuse the data. The U.S. Department of Energy’s Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) data repository archives datasets with related resources that may be distributed across numerous online sources. This presentation will outline ESS-DIVE’s current approach to linking and citations, and plans for tracking additional related identifiers.

Our goal is to enable users to easily identify scientifically useful information related to a dataset, and enable more accurate citation metrics. Here we compare existing approaches to linking related resources, including those from DataCite, Schema.org, and Ecological Metadata Language (EML). We then test use cases for applying related identifiers and connection metadata (e.g. resource type, relationship type) in ESS-DIVE datasets. And explore how any references to datasets and papers are or should be incorporated into citation and usage metrics. ESS-DIVE’s proposed approach to link related resources will make interdisciplinary Environmental System Science data more FAIR. A standard approach for linking identifiers can further facilitate automated transfer of metadata and data across online repositories and data systems. We find that automated tools and coordination is still needed across data systems and publishers to make effective linking practical, and to enable more accurate citation metrics.

Using FAIR-Guided Metadata Requirements for Diverse Data in the ESS-DIVE Data Repository (IN44A-07)

Presenter: Emily Robles
Presentation Type: Oral
Session Date and Time: Thursday, December 14th; 17:00 – 17:10 PST
Session Number and Title: IN44A: Applying Community-Developed Principles and Guidance to Improve Open-Science Capabilities of Scientific Data Repositories and Service Providers II Oral
Presentation Location: Moscone Center, 2014 – West
Session URL: https://agu.confex.com/agu/fm23/meetingapp.cgi/Paper/1451422

Abstract

Emily Robles1, Deb Agarwal1, Madison Burrus2, Joan E Damerow1, Hesham Elbashandy1, Valerie C Hendrix2, Mario Melara1, Fianna O’Brien1, Dylan O’Ryan1, Shalki Shrivastava1, Karen Whitenack1, Shreyas Cholia2 and Charuleka Varadharajan2, (1)Lawrence Berkeley National Laboratory, Berkeley, United States, (2)Lawrence Berkeley National Laboratory, Berkeley, CA, United States

High volumes of environmental science data are needed to support ecological research, including large scale analyses and model development. As open publication practices become more common and increase data availability, data and metadata quality remain limiting factors for discovery and reuse. These issues can begin to be addressed through metadata requirements implemented by data repositories or archives, however the review of metadata remains a complex issue for large data repositories holding diverse data types, both because of the need to prioritize scalability and the need for community adoption. Using guidance from the FAIR principles, which state that data should be finable, accessible, interoperable, and reusable, the U.S. Department of Energy’s Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) data repository has implemented community agreed upon metadata checks at both the data package- and file-level to improve the usability and accessibility of published data.

Package-level metadata, such as complete and descriptive titles and methods information, provides important contextual information needed to reuse data and is reviewed by ESS-DIVE using a system of both automated and manual checks. To address the need for additional formatting guidelines in a field where data are often diverse and multidisciplinary, ESS-DIVE partnered with researchers from its science community to develop 13 reporting formats, covering a broad range of Earth science data and metadata types. Datasets that have adopted reporting format guidelines for file-level metadata now undergo an automated assessment for metadata quality requirements upon submission to ESS-DIVE. Extending metadata review to the file level will not only improve upon the information provided by package-level metadata, but enable future expansion of search and discovery capabilities. Here we discuss the process by which ESS-DIVE used guidance from the FAIR principles, in addition to prioritizing community feedback and collaboration, during the development of dataset requirements, as well as the mechanisms of the metadata review process and future directions for utilizing file-level review.

2023 ESS-DIVE Open Data Workshop (ODW) to Propel Environmental System Science Projects into the Future

by

The Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) data archive, based at Lawrence Berkeley National Laboratory, is thrilled to announce the 2023 ESS-DIVE Open Data Workshop (ODW). This annual event promises to be two days of inspiration, collaboration, and knowledge sharing, bringing together Principal Investigators (PIs) and team members engaged in Environmental System Science (ESS) projects.

This year, the 2023 ESS-DIVE ODW holds special significance as it aligns with the United States government’s celebration of 2023 as the “Year of Open Science.” This multi-agency initiative across the federal government aims to foster open science engagement through events and activities that promote open, equitable, and secure science.

“The 2023 ESS-DIVE Open Data Workshop is a prime opportunity for Environmental System Science researchers to tap into the momentum of the Year of Open Science and bolster their projects with cutting-edge data management practices,” says Charuleka Varadharajan, Principal Investigator with ESS-DIVE.

Building on the success of last year’s Community Data Workshop, this event is tailored to serve both newcomers and those already familiar with ESS-DIVE. The focus will be on new features and resources that facilitate collaboration, and Open data management and publishing to enhance the research process.

By attending this workshop, you will gain access to important information and networking opportunities, offering you the following benefits:

  • Make High-Quality Data Management Easier: Efficient data curation and integration is critical to save time and resources for ESS-related projects. Learn how to use ESS-DIVE tools and resources to make publishing high-quality data and collaborative data management easier. And participate in discussions to shape how we can standardize data practices as a community, work with large and complex data, automate repetitive data tasks, and bridge data silos.
  • Integrate and Use ESS Data: Learn about new ESS-DIVE tools that will make it easier to discover and integrate ESS data. Explore a unique collaborative paper opportunity that aims to compile scientific use cases for data linking and integration across the U.S. DOE’s Biological and Environmental (BER) data systems, including the Joint Genome Institute (JGI), Systems Biology Knowledgebase (KBase), National Microbiome Data Collaborative (NMDC), Environmental Molecular Sciences Laboratory, (EMSL), Atmospheric Radiation Measurement (ARM), Earth System Grid Federation (ESGF), and ESS-DIVE.
  • Network and Learn from Peers: Participate in a collaborative, interdisciplinary environment where you can learn from fellow scientists with a variety of expertise across data, computational, and domain sciences. Share experiences, challenges, and innovative solutions to advance open ESS data and reuse.

Register Now: Registration for the 2023 ESS-DIVE Open Data Workshop is now open, and encouraged for anyone who is part of an ESS project. While participants are encouraged to attend both days for the full experience, they are also welcome to join sessions of their interest. To secure your spot, simply complete this registration form. Once registered, you will receive a personalized Zoom link, granting you access to the event.

Each session will provide ample opportunities for participants to engage with the ESS-DIVE team, ask questions, share comments, and provide feedback. Before the event, registered attendees will receive quick instructions on preparing for tutorials and discussions to make the most of their experience.

To discover more about the workshop or ESS-DIVE in general, please visit the workshop event page or contact ess-dive-support@lbl.gov.

Don’t miss the chance to be part of this transformative event and take your ESS projects to new heights. Join us at the 2023 ESS-DIVE Open Data Workshop for two days of learning, networking, and innovation.

September Webinar on Publishing Large Data on ESS-DIVE’s Tier 2 Storage Service

by

September Community Webinar

Tuesday, September 19 | 10:00-11:00 PT / 13:00-14:00 ET

View Webinar Recording / Link to Webinar Slides

Do you have large volumes of data sitting on a server that you have been wanting to publish? Is the volume of your data a bottleneck to uploading files to ESS-DIVE? Recognizing that many ESS projects generate large data such as remote sensing, model output, and imagery, ESS-DIVE has developed new tools that enable submitting, publishing, and accessing large datasets. 

Join our September webinar for an introduction to Tier 2 storage and Globus transfer services and to learn how they are used to accommodate hierarchical data and large data volumes greater than 500GB.

This webinar is designed to help data contributors, data managers and PIs learn about ESS-DIVE’s new large data support tools and how to incorporate them into data publication workflows. We will cover the following topics:

  • What is considered large data on ESS-DIVE?
  • ESS-DIVE’s data submission tools
    • Web submission form(standard)
    • Dataset API (programmatic)
    • Globus (large data transfer service)
  • Tier 2: large data storage on ESS-DIVE

Please encourage anyone from your project who may be interested to attend.

Fianna O’Brien, flobrien@lbl.gov

Software Engineer
Fianna is a software engineer supporting archival data transitions and data publication for ESS-DIVE. She has experience managing publication data and creating data driven outreach tools for large-scale ESS projects. She holds a B.A. in Computational Linguistics from Hampshire College.

June Webinar on ESS-DIVE Project Management Features

by

June Community Webinar

Monday, June 26 | 10:00-11:00 PT / 13:00-14:00 ET

View Webinar Recording / Link to Webinar Slides

ESS-DIVE has released our first set of features specifically dedicated to supporting projects’ data management needs. These features expand our existing suite of data management tools and provide a more comprehensive way for projects to collaborate on ESS-DIVE. During this webinar, learn about ESS-DIVE features that can be used for managing project data and the necessary steps to set up and leverage these tools. See our recent blog post for a brief description of the new project management features.

If you missed ESS-DIVE’s breakout session at the ESS PI meeting, please come join our June webinar!

This webinar is particularly geared toward helping Principal Investigators (PIs) and Data Managers set up their project data management on ESS-DIVE. We will cover the following topics:

  • Introduction to new project-centric features:
    • ESS-DIVE Project List
    • Project Manager role
    • Project page
    • Teams
    • Sharing Data Permissions
  • How PIs can start setting up project-centric data management on ESS-DIVE
  • Feedback session: tell us what your projects need!

Please encourage anyone from your project who may be interested to attend. This webinar will be presented by Madison Burrus.

Madison Burrus, Computer Systems Engineer

Madison is a computer systems engineer at Lawrence Berkeley National Lab. She supports ESS-DIVE metadata publication, data pipeline workflows, and feature development. Additionally, she is lead support staff and handles community inquiries. She holds a B.S. in Environmental Science from Stony Brook University.

Request for Information from the Biological and Environmental Research Advisory Committee

by

The US Department of Energy’s Office of Science is seeking input on the need and the structure of a unified data framework that links or integrates existing data activities within the Biological and Environmental Research Program (BER).

The aim is to gather current and future science questions that require a more integrated data infrastructure for data access and processing, especially those that span more than one research area. As well as community input regarding what components or technologies are needed to develop a more unified data infrastructure, and what would improve accessibility to these resources for a broad range of communities, perspectives, and research domains.

Information may be used by the BER Advisory Committee (BERAC) to help inform and recommend a strategy for a unified BER data management and analysis framework.

To facilitate collection of responses, several BER programs have coordinated several mechanisms for feedback:

  1. Responses may be submitted via a form: https://bit.ly/BERACUnifiedDataRFI (requires a valid email to submit) with prompts listed in the request for information (RFI): https://www.govinfo.gov/content/pkg/FR-2023-04-17/pdf/2023-08029.pdf

  2. Listening sessions will be held on Zoom and recorded. The recording and transcript will be provided to BER, alongside a summary of points to each prompt.

    Wed Jun 7, 2023 at 9:00 am PT, 12:00 pm ET
    Register in advance for this meeting: bit.ly/BERACUnifiedDataRFIZoom1

    Tue Jun 27, 2023 at 8:00 am PT, 11:00 am ET
    Register in advance for this meeting: bit.ly/BERACUnifiedDataRFIZoom2

If you have any questions about the RFI, please contact Dr. Tristram O. West, (301) 903–5155, Tristram.west@science.doe.gov. If you have any questions about the provided feedback sessions, please contact Joan Damerow at JoanDamerow@lbl.gov.