We are excited to announce that ESS-DIVE’s project management related services have been upgraded! You can now explore new tools and enhancements that make it easier to collaboratively manage project data with your team on ESS-DIVE.

• *NEW* Project List: A list of every project that is approved to store data on ESS-DIVE. Quickly verify that your project name and PI information are up-to-date. 
• *NEW* Project Data Manager role: Anyone given this role will have access to additional project information and features; this role is reserved for PIs and team members they approve.
• *NEW* Project Teams: Create a group of ESS-DIVE users and provide a way to share data permissions with all members of that group; manage team members to control group data access.

If you are interested in working 1:1 with ESS-DIVE to assign data managers for your project or set up teams, you can request to schedule a 30-minute meeting with ESS-DIVE Support. Please reach out to ESS-DIVE Support (ess-dive-support@lbl.gov) to schedule a meeting time.

We will also be presenting a hands-on tutorial on these project management features at the ESS PI Meeting on May 16th at 7:30 AM – 9:30 AM PT / 10:30 AM – 12:30 PM ET. Additionally, we will be holding a webinar on June 26th from 10 AM – 11 AM PT / 1 PM ET – 2 PM ET that will focus on these new project management features. We will send out an email announcement later this month.

Project list and search

• We now have a controlled list of every project approved to store and publish data on ESS-DIVE. 
  • All projects funded by the Department of Energy’s (DOE) Environmental System Sciences (ESS) program are automatically approved, and should be included in this list.
  • If you are an ESS funded project and do not see your project in this list, please email ESS-DIVE Support at ess-dive-support@lbl.gov. 

• After logging in with your ORCID, you can view Principal Investigator (PI) info; you can contact ESS-DIVE Support if any information is out of date.

 

Principal Investigators (PIs) can now assign project data managers

• A Project Data Manager can access additional project information and features via the new “My Projects” page. PIs can be given this role at any time, but must register as a data contributor first.
  

• PIs can assign other project data managers by completing the Add Project Data Managers form. All requested data managers must first register as ESS-DIVE data contributors.
  

• If you’re a PI and became a data contributor before May 5th, you are already a project data manager and can access the “My Projects” page!  
  • PIs can reach out to ESS-DIVE Support at ess-dive-support@lbl.gov if you cannot access the “My Projects” page.
    
• This new role does not affect access to datasets on ESS-DIVE. Project data managers will not receive default permission to access any datasets.

 

Create and share data with project teams

• Project data managers can now create Project Teams on ESS-DIVE. Teams are groups of ESS-DIVE users, and provide a new way of efficiently sharing data permissions with those groups!

• For project data managers: start creating teams for your project to leverage group data sharing in the future!
  • Choose team names that will help data contributors know which team they should share their data with.
  • Add/remove team members at any time to manage who can access datasets or data portals shared with that team.  

• For data contributors: easily share your datasets with a group of people all at once, instead of sharing with one person at a time. Ask your project data managers which teams to share your data with.
  • For a refresher on how to share dataset permissions, visit our Share Data Permissions documentation.
    

• Teams do not receive access to any datasets by default. Work with your project’s data contributors to enable team access to datasets.

The ESS-DIVE Team is looking forward to participating in the Environmental System Science (ESS) Cyberinfrastructure (CI) Working Group Annual Meeting and ESS PI Meeting on May 15th and May 16th-17th, respectively. Deb Agarwal, Charu Varadharahan, Shreyas Cholia, Joan Damerow, and Emily Robles will all be in-person for the ESS CI and ESS PI meetings! Below are some ways that you can engage with ESS-DIVE at both meetings:

ESS CI Working Group Annual Meeting – May 15th, 2023

Project Updates: ESS-DIVE 

Presenter: Charu Varadharajan, ESS-DIVE Co-PI, Lead: Community Engagement

Date and Time: May 15th, 2023; 9:30 AM – 9:40 AM ET / 6:30 AM – 6:40 AM PT

Flash Talk: Interactive Supercomputing With Jupyter

Presenter: Shreyas Cholia, ESS-DIVE Co-PI, Lead: Scaling, Resilience and Integration

Date and Time: May 15th, 2023; 10:45 AM – 12:00 PM ET / 7:45 AM – 9:00 AM PT

Data: Breakout Round Robin

Leads: Joan Damerow, Community Engagement Lead Scientist; Ben Sulman (ORNL)

Date and Time: May 15th, 2023; 3:15 PM – 4:15 PM ET / 12:15 PM – 1:15 PM PT

Data Management supporting integration in ModEx, including 1) model inputs: types of data, data standards and metadata, gap filling, normalization and access, and 2) model outputs: formats, archiving and use in experimental design

Models: Breakout Round Robin

Leads: Charu Varadharajan, ESS-DIVE Co-PI, Lead: Community Engagement; Ethan Coon (ORNL)

Date and Time: May 15th, 2023; 3:15 PM – 4:15 PM ET / 12:15 PM – 1:15 PM PT

Exploring the various types of watershed and ecosystem models, from phenomenological to process-based, their different roles and data needs and how best to use this broad range of models to address different scientific questions at different scales.

ESS PI Meeting – May 16th to 17th, 2023

ESS-DIVE tutorials: Managing project data on ESS-DIVE 

Presenter: Joan Damerow, Community Engagement Lead Scientist

Session Date and Time: May 16th, 2023; 10:30 AM -12:30 PM ET / 7:30 AM – 9:30 AM PT

Presentation Slides

This session is designed for PI’s and data managers who want to learn how to manage project data on ESS-DIVE. We will first provide an overview of how ESS-DIVE can support a project’s data management needs. The tutorial will then walk through how to use ESS-DIVE features for project data management, including: (1) new features to assign project data managers and create teams, (2) sharing data permissions with project members for collaborative editing and review, (3) creating portals to showcase project data collections, and (4) linking ess-dive datasets to other repositories.

ESS-DIVE tutorials: Creating and publishing high-quality datasets

Presenter: Emily Robles, Senior Research Associate 

Session Date and Time: May 16th, 2023; 1:00 PM – 2:30 PM ET / 10:00 AM – 11:30 AM PT

Presentation Slides

Learn how to upload and publish data with ESS-DIVE as a new contributor in this hands-on training session. We will cover all of the basics needed to get started with publishing datasets, from important considerations for organizing data and accompanying metadata, to choosing an upload method for dataset submission, and requesting publication and following the dataset review process. Our hands-on activity walks through creating and submitting a dataset to ESS-DIVE’s sandbox using our online submission form. We will also discuss ways that researchers can improve the quality of their datasets by using ESS-DIVE’s community data and metadata formats.

Poster: ESS-DIVE Overview: A Scalable, User-focused Repository for Earth and Environmental

Presenter: Shreyas Cholia, ESS-DIVE Co-PI, Lead: Scaling, Resilience and Integration 

Poster: Community-developed (meta)data reporting formats to enable data reuse in ESS-DIVE

Presenter: Charu Varadharajan, ESS-DIVE Co-PI, Lead: Community Engagement

February Community Webinar

Monday, February 27 | 10:00-11:00 PT / 13:00-14:00 ET

View Webinar Recording / Link to Webinar Slides

ESS-DIVE is excited to welcome Urban Integrated Field Laboratory (Urban IFL) projects to the community. We invite all who are part of an Urban IFL project and the broader ESS community to an introductory webinar about the ESS-DIVE data repository for Earth and environmental system science data, and associated resources for project data management. We will also have a discussion on data management needs for Urban IFL projects.  

We will cover the following topics:

• Overview of ESS-DIVE and our data;
• ESS-DIVE project data management features; 
• Resources to organize and curate high-quality datasets; and
• Discussion on Urban IFL data management needs.

Please encourage anyone from your project who may be interested to attend. 

The webinar will be presented by Joan Damerow and Emily Robles.

Joan Damerow, Community Engagement Lead Scientist

Joan is an environmental scientist with a diverse background in geoscience sampling, freshwater ecology, and biodiversity informatics. She runs engagement activities for ESS-DIVE, including ESS-DIVE webinars, our community data workshop, and is active in relevant conferences and data working groups (e.g. ESIP, RDA, AGU). Joan is interested in interdisciplinary data management and tracking, and works with our community of data contributors to identify, develop, and implement practical data standards in ESS-DIVE that support FAIR principles.

Emily Robles, Data Management Research Associate

Emily is a research associate and a member of ESS-DIVE’s Community Engagement team. Her work with ESS-DIVE focuses on utilizing community research and feedback to implement a comprehensive data package review workflow for data publication. She has experience with data package organization and quality management as a member of the NGEE Tropics data team and received her BS in Environmental Science from the University of California, Berkeley.

January Community Webinar

Monday, January 30 | 10:00-11:00 PT / 13:00-14:00 ET

View Webinar Video / Link to Webinar Slides

Are you interested in publishing or using data from small Unoccupied Aerials Systems (UAS)? With rapid development of platform and sensor technologies, the use of UASs, also known as UAVs or drones, in Earth sciences is becoming increasingly common. This webinar will review the new UAS data and metadata reporting format that is in development, and provide you an opportunity to give feedback.

We will cover the following topics:

• Benefits of using reporting formats,
• Types of data that this format will cover and associated challenges,
• Metadata variables to describe flight details and equipment, 
• Categorizing data by processing level, and
• Discussion and feedback. 

You can review the draft UAS data and metadata reporting format before the webinar, although this is not required to attend.  This webinar will be presented by Kim Ely and Shawn Serbin from Brookhaven National Laboratory (BNL).

Please encourage anyone from your project who may be interested to attend.

The ESS-DIVE Team is looking forward to participating in the 2022 AGU Fall Meeting. Below are several abstracts that we will be presenting at the meeting!

BASIN-3D: Data Synthesis Software for Earth Science Researchers (H12P-0874)

Presenter: Danielle Christianson
Presentation Type: Poster
Session Date and Time: Monday, 12 December 2022; 9:00 AM – 12:30 PM CST / 7:00 AM – 10:30 AM PST
Session Number and Title: H12P: Hydroinformatics and Data Science: Pathways to Support Reproducible Watershed Modeling II Poster
Session URL: https://agu.confex.com/agu/fm22/webprogrampreliminary/Paper1153073.html

Abstract

Danielle S Christianson, Valerie C Hendrix, Catherine Wong, Charuleka Varadharajan, and Deb Agarwal, Lawrence Berkeley National Laboratory, Berkeley, CA, United States

Integration of diverse data required for earth science research remains a time-consuming task. While approaches such as data warehousing and federation (brokering) have reduced effort, the difficult work of harmonizing formats, units, and semantics remains for the individual researcher. Often the resulting one-off data synthesis products become outdated as data change. In addition, the researcher does not have flexibility to tailor synthesis to specific data needed.

In this presentation, we describe BASIN-3D (Broker for Assimilation, Synthesis and Integration of eNvironmental Diverse, Distributed Datasets), a tool we developed to address this gap. BASIN-3D synthesizes diverse data from a variety of remote and/or local data sources in real-time without the need for additional storage. The software can be deployed as a python package in python-based applications like Jupyter notebooks or as a web service application. Data sources can be configured using a plugin approach that maps their data model and vocabularies to BASIN-3D’s. These mappings enable BASIN-3D to handle translation of query parameters and results into a standardized data model. BASIN-3D then outputs the synthesized results in common data formats (e.g., hdf5, Pandas dataframe, json) specified by the researcher.

We describe the unique features of BASIN-3D using two prototype U.S. Department of Energy use cases. The first is a python-based application that combines USGS National Water Information Systems NWIS and DayMet time series data to support study of disturbances on river water quality with ML techniques. The second is a Django web-based application that integrates time series data from NWIS and project-based databases for access via an online data portal. Finally, we discuss future work and ongoing challenges such as optimization for large data volumes, adding new data sources, and tradeoffs between generalization and customizability.

 

Community-developed (meta)data reporting formats to enable data reuse in environmental repositories (IN41A-06)

Presenter: Charuleka Varadharajan
Presentation Type: Online Poster Discussion
Session Date and Time: Thursday, 15 December 2022; 8:40 AM – 8:48 AM CST / 6:40 AM – 6:48 AM PST
Session Number and Title: IN41A: Adopting Trustworthy Data Repository Stewardship to Enable Reuse of Data Across Disciplines I Online Poster Discussion
Session URL: https://agu.confex.com/agu/fm22/webprogrampreliminary/Paper1163669.html

Abstract

Charuleka Varadharajan¹, Robert Crystal-Ornelas¹, Dylan O’Ryan², Bond-Lamberty Benjamin³, Kathleen Beilsmith⁴, Kristin Boye⁵, Madison Burrus¹, Shreyas Cholia¹, Danielle S Christianson¹, Michael Cameron Crow⁶, Joan E Damerow¹, Kim S Ely⁷, Amy E Goldman⁸, Susan L Heinz⁹, Valerie C Hendrix¹, Zarine Kakalia¹, Kayla Cerise Mathes¹⁰, Fianna O’Brien¹, Stephanie C Pennington^11,12, Emily Robles¹, Alistair Rogers⁷, Maegen Simmonds^1,13, Terri Velliquette⁶, Pamela Weisenhorn¹⁴, Jessica Nicole Welch⁶, Karen Whitenack¹ and Deb Agarwal¹, (1)Lawrence Berkeley National Laboratory, Berkeley, CA, United States, (2)Lawrence Berkeley National Lab, Berkeley, CA, United States, (3)Pacific Northwest National Laboratory, College Park, MD, United States, (4)Argonne National Laboratory, Chicago, United States, (5)SLAC National Acceleratory Laboratory, Stanford Synchrotron Radiation Lightsource, Menlo Park, CA, United States, (6)Oak Ridge National Laboratory, Oak Ridge, TN, United States, (7)Brookhaven National Laboratory, Environmental and Climate Sciences Department, Upton, NY, United States, (8)Pacific Northwest National Laboratory, Biological Sciences, Richland, WA, United States, (9)Oak Ridge National Laboratory, Oak Ridge, United States, (10)Virginia Commonwealth University, Integrative Life Sciences, Richmond, VA, United States, (11)Pacific Northwest National Laboratory, College Park, United States, (12)Joint Global Change Research Institute, College Park, United States, (13)Pivot Bio, Berkeley, United States, (14)Argonne National Laboratory, Argonne, IL, United States

Findable, Accessible, Interoperable, and Reusable (FAIR) principles are intended to enable the reuse of Earth and environmental science data beyond the purpose for which the data were originally collected. One pathway to making data more reusable is for repositories to encourage contributors to organize and publish data that follow established standards and guidelines. However, Earth science data are diverse and multidisciplinary making it difficult for researchers to determine and use the appropriate standards or formats that apply to their data.

Here, we present 11 reporting formats: instructions, templates, and tools for consistently formatting data, for a diverse set of Earth science (meta)data. These formats were developed through a partnership between the U.S. Department of Energy’s Environmental Systems Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) repository and researchers from its science community. They cover a broad range of Earth science (meta)data that includes cross-domain metadata (dataset metadata, location metadata, sample metadata), file-formatting guidelines (file-level metadata, CSV files, terrestrial model data), and domain-specific formats for biological, geochemical, and hydrological data types (amplicon abundance tables, leaf-level gas exchange, soil respiration, water and sediment chemistry, sensor-based hydrologic measurements). We adopted a community consensus process to develop these formats by obtaining extensive input from 247 researchers across 128 institutions. This resulted in a pragmatic set of reporting formats that are based on scientific use cases. We also describe lessons learned from this process and guidelines that communities can use to create new reporting formats that are tailored to their scientific workflows. Such community-developed reporting formats lend themselves to easy adoption, enabling scientific data synthesis and knowledge discovery by making it easier for data contributors to provide (meta)data that are more FAIR.

 

A FAIR Guided and Community-Oriented Approach to Improving Metadata Quality in a Large Scale Data Repository (IN41A-03)

Presenter: Emily Robles
Presentation Type: Online Poster Discussion
Session Date and Time: Thursday, 15 December 2022; 8:16 AM – 8:24 AM CST / 6:16 AM – 6:24 AM PST
Session Number and Title: IN41A: Adopting Trustworthy Data Repository Stewardship to Enable Reuse of Data Across Disciplines I Online Poster Discussion
Session Link: https://agu.confex.com/agu/fm22/webprogrampreliminary/Paper1199893.html

Abstract

Emily Robles¹, Charuleka Varadharajan¹, Madison Burrus¹, Shreyas Cholia¹, Robert Crystal-Ornelas¹, Joan E Damerow¹, Hesham Elbashandy¹, Valerie C Hendrix¹, Christopher S. Jones², Matthew B. Jones³, Zarine Kakalia¹, Mario Melara¹, Fianna O’Brien¹, Peter Slaughter², Karen Whitenack¹ and Deb Agarwal¹, (1) Lawrence Berkeley National Laboratory, Berkeley, CA, United States, (2) National Center for Ecological Analysis and Synthesis, Santa Barbara, CA, United States, (3) National Center for Ecological Analysis and Synthesis, DataONE, Santa Barbara, CA, United States

The Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) data repository stores diverse Earth and environmental science data generated by projects funded by the U.S. Department of Energy (DOE). ESS-DIVE strives to publish datasets that adhere to the FAIR principles, which state that data should be findable, accessible, interoperable, and reusable, and to develop practices that enable data providers to improve the quality of data submissions. However, these strategies must also be able to scale with the growth of the repository, apply to a wide range of data types, and be valuable to both data providers and users.

To address these challenges, we have developed metadata requirements for the ESS-DIVE repository through research into dataset best practices, review of journal metadata requirements, and with community feedback to ensure that they are useful and applicable across data types. We then developed a two-part, semi-automated dataset review workflow that programmatically verifies whether datasets meet metadata quality requirements before publication. The automated component includes FAIR metadata checks developed by the National Center for Ecological Analysis and Synthesis (NCEAS) that were customized to fit ESS-DIVE’s publishing requirements. Results from the automated checks are compiled into a Metadata Assessment Report, providing instant feedback to data providers that identifies where and how their metadata can be improved. ESS-DIVE reviewers then carry out a manual, content-focused metadata review based on FAIR principles. Finally, revision requests are sent by reviewers, who then collaborate 1:1 with data providers until their dataset is eligible for publication.

Since implementation, 401 datasets have been reviewed using the semi-automated dataset review workflow. We have found that incorporating automated metadata validation has reduced review time, allowing the publication workflow to scale as the repository grows and freeing up time for reviewers to interact 1:1 with data providers to improve their publication practices. Finally, by tracking all review results, we are able to make transparent, data based recommendations to our community and continue to improve automation where possible.

 

Enabling proper Citation of Individual Objects Across Large Collections of Datasets (IN42B-0338)

Presenter: Deb Agarwal
Presentation Type: Poster
Session Date and Time: Thursday, 15 December 2022; 9:00 AM – 12:30 PM CST / 7:00 AM – 10:30 AM PST
Session Number and Title: IN42B: Adopting Trustworthy Data Repository Stewardship to Enable Reuse of Data Across Disciplines II Poster
Session Link: https://agu.confex.com/agu/fm22/webprogrampreliminary/Paper1188622.html

Abstract

Deb Agarwal¹, Martina Stockhause², Lesley A Wyborn³, Justin James Henry Buck⁴, James Ayliffe⁴ and Shelley Stall⁵, (1) Lawrence Berkeley National Laboratory, Berkeley, CA, United States, (2) German Climate Computing Centre (DKRZ), Hamburg, Germany, (3) Australian National University, Canberra, ACT, Australia, (4) National Oceanography Center, BODC, Liverpool, United Kingdom, (5) American Geophysical Union, Data Leadership, Washington, DC, United States

 

Sample tracking and synthesis needs for exploring ecosystem response to climate and environmental disturbance (IN55A-03)

Presenter: Joan Damerow
Presentation Type: Oral
Session Date and Time: Friday, 16 December 2022; 3:06 PM – 3:14 PM CST / 1:06 PM – 1:14 PM PST
Session Number and Title: IN55A: Global Community Efforts to Make Samples, Specimens, and Sampling Features (As Well Digital Information About Them) Comply with the FAIR and CARE Principles III Oral
Session Link: https://agu.confex.com/agu/fm22/webprogrampreliminary/Paper1121127.html

Abstract

Joan E Damerow¹, Elisha M Wood-Charlson¹, Charuleka Varadharajan¹, Mikayla Borton², Eoin Brodie³, Richard S. Canon⁴, Shreyas Cholia¹, Paramvir Dehal⁵, Zachary Crockett⁶, Emiley Eloe-Fadrosh⁷, Ricardo J Eloy Alves⁸, Kjiersten Fagnan⁷, Amy E Goldman⁹, David Hays¹⁰, Valerie C Hendrix¹, Lee Ann McCue¹¹, Nancy Shiao-Lynn Merino¹², Marka Miller⁴, Chris Mungall¹³, Supratim Mukherjee¹⁰, T.B.K. Reddy¹⁰, Patrick Sorensen³, Montana L Smith¹⁴, James Stegen¹⁵, Pajau Vangay⁴, Pamela Weisenhorn¹⁶, Steven Wilson¹⁷ and Deb Agarwal¹, (1) Lawrence Berkeley National Laboratory, Berkeley, CA, United States, (2) Colorado State University, Soil and Crop Sciences, Fort Collins, United States, (3) Lawrence Berkeley National Laboratory, Earth and Environmental Sciences Area, Berkeley, CA, United States, (4) Lawrence Berkeley National Laboratory, Berkeley, United States, (5) Lawrence Berkeley National Lab, Berkeley, United States, (6) Oak Ridge National Laboratory, Oak Ridge, United States, (7) Joint Genome Institute, Walnut Creek, CA, United States, (8) Lawrence Berkeley National Laboratory, Climate and Ecosystem Sciences Division, Berkeley, CA, United States, (9) Pacific Northwest National Laboratory, Biological Sciences, Richland, WA, United States, (10) Joint Genome Institute, Berkeley, United States, (11) Pacific Northwest National Lab, Richland, United States, (12) Lawrence Livermore National Laboratory, Livermore, United States, (13) Lawrence Berkeley National Laboratory, Environmental Genomics and Systems Biology, Berkeley, United States, (14) Pacific Northwest National Laboratory, Environmental Molecular Sciences Laboratory, Richland, United States, (15) Pacific Northwest National Laboratory, Richland, United States, (16) Argonne National Laboratory, Argonne, IL, United States, (17)Joint Genome Institute, Berkeley, CA, United States