ESS-DIVE

Deep Insight for Earth Science Data

September Webinar on Publishing Large Data on ESS-DIVE’s Tier 2 Storage Service

by

ESS-DIVE Webinar

Tuesday, September 19 | 10:00-11:00 PT / 13:00-14:00 ET

View Webinar Recording / Link to Webinar Slides

Do you have large volumes of data sitting on a server that you have been wanting to publish? Is the volume of your data a bottleneck to uploading files to ESS-DIVE? Recognizing that many ESS projects generate large data such as remote sensing, model output, and imagery, ESS-DIVE has developed new tools that enable submitting, publishing, and accessing large datasets. 

Join our September webinar for an introduction to Tier 2 storage and Globus transfer services and to learn how they are used to accommodate hierarchical data and large data volumes greater than 500GB.

This webinar is designed to help data contributors, data managers and PIs learn about ESS-DIVE’s new large data support tools and how to incorporate them into data publication workflows. We will cover the following topics:

  • What is considered large data on ESS-DIVE?
  • ESS-DIVE’s data submission tools
    • Web submission form(standard)
    • Dataset API (programmatic)
    • Globus (large data transfer service)
  • Tier 2: large data storage on ESS-DIVE

Please encourage anyone from your project who may be interested to attend.

Fianna O’Brien, flobrien@lbl.gov

Software Engineer
Fianna is a software engineer supporting archival data transitions and data publication for ESS-DIVE. She has experience managing publication data and creating data driven outreach tools for large-scale ESS projects. She holds a B.A. in Computational Linguistics from Hampshire College.

June Webinar on ESS-DIVE Project Management Features

by

ESS-DIVE Webinar

Monday, June 26 | 10:00-11:00 PT / 13:00-14:00 ET

View Webinar Recording / Link to Webinar Slides

ESS-DIVE has released our first set of features specifically dedicated to supporting projects’ data management needs. These features expand our existing suite of data management tools and provide a more comprehensive way for projects to collaborate on ESS-DIVE. During this webinar, learn about ESS-DIVE features that can be used for managing project data and the necessary steps to set up and leverage these tools. See our recent blog post for a brief description of the new project management features.

If you missed ESS-DIVE’s breakout session at the ESS PI meeting, please come join our June webinar!

This webinar is particularly geared toward helping Principal Investigators (PIs) and Data Managers set up their project data management on ESS-DIVE. We will cover the following topics:

  • Introduction to new project-centric features:
    • ESS-DIVE Project List
    • Project Manager role
    • Project page
    • Teams
    • Sharing Data Permissions
  • How PIs can start setting up project-centric data management on ESS-DIVE
  • Feedback session: tell us what your projects need!

Please encourage anyone from your project who may be interested to attend. This webinar will be presented by Madison Burrus.

Madison Burrus, Computer Systems Engineer

Madison is a computer systems engineer at Lawrence Berkeley National Lab. She supports ESS-DIVE metadata publication, data pipeline workflows, and feature development. Additionally, she is lead support staff and handles support inquiries. She holds a B.S. in Environmental Science from Stony Brook University.

Request for Information from the Biological and Environmental Research Advisory Committee

by

The US Department of Energy’s Office of Science is seeking input on the need and the structure of a unified data framework that links or integrates existing data activities within the Biological and Environmental Research Program (BER).

The aim is to gather current and future science questions that require a more integrated data infrastructure for data access and processing, especially those that span more than one research area. As well as DOE ESS input regarding what components or technologies are needed to develop a more unified data infrastructure, and what would improve accessibility to these resources for a broad range of communities, perspectives, and research domains.

Information may be used by the BER Advisory Committee (BERAC) to help inform and recommend a strategy for a unified BER data management and analysis framework.

To facilitate collection of responses, several BER programs have coordinated several mechanisms for feedback:

  1. Responses may be submitted via a form: https://bit.ly/BERACUnifiedDataRFI (requires a valid email to submit) with prompts listed in the request for information (RFI): https://www.govinfo.gov/content/pkg/FR-2023-04-17/pdf/2023-08029.pdf

  2. Listening sessions will be held on Zoom and recorded. The recording and transcript will be provided to BER, alongside a summary of points to each prompt.

    Wed Jun 7, 2023 at 9:00 am PT, 12:00 pm ET
    Register in advance for this meeting: bit.ly/BERACUnifiedDataRFIZoom1

    Tue Jun 27, 2023 at 8:00 am PT, 11:00 am ET
    Register in advance for this meeting: bit.ly/BERACUnifiedDataRFIZoom2

If you have any questions about the RFI, please contact Dr. Tristram O. West, (301) 903–5155, Tristram.west@science.doe.gov. If you have any questions about the provided feedback sessions, please contact Joan Damerow at JoanDamerow@lbl.gov.

ESS-DIVE Has Released New Project Management Features

by

We are excited to announce that ESS-DIVE’s project management related services have been upgraded! You can now explore new tools and enhancements that make it easier to collaboratively manage project data with your team on ESS-DIVE.

  • *NEW* Project List: A list of every project that is approved to store data on ESS-DIVE. Quickly verify that your project name and PI information are up-to-date. 
  • *NEW* Project Data Manager role: Anyone given this role will have access to additional project information and features; this role is reserved for PIs and team members they approve.
  • *NEW* Project Teams: Create a group of ESS-DIVE users and provide a way to share data permissions with all members of that group; manage team members to control group data access.

If you are interested in working 1:1 with ESS-DIVE to assign data managers for your project or set up teams, you can request to schedule a 30-minute meeting with ESS-DIVE Support. Please reach out to ESS-DIVE Support (ess-dive-support@lbl.gov) to schedule a meeting time.

We will also be presenting a hands-on tutorial on these project management features at the ESS PI Meeting on May 16th at 7:30 AM – 9:30 AM PT / 10:30 AM – 12:30 PM ET. Additionally, we will be holding a webinar on June 26th from 10 AM – 11 AM PT / 1 PM ET – 2 PM ET that will focus on these new project management features. We will send out an email announcement later this month.

Project list and search

  • We now have a controlled list of every project approved to store and publish data on ESS-DIVE. 
  • After logging in with your ORCID, you can view Principal Investigator (PI) info; you can contact ESS-DIVE Support if any information is out of date.

 

Principal Investigators (PIs) can now assign project data managers

  • A Project Data Manager can access additional project information and features via the new “My Projects” page. PIs can be given this role at any time, but must register as a data contributor first.
  • PIs can assign other project data managers by completing the Add Project Data Managers form. All requested data managers must first register as ESS-DIVE data contributors.
  • If you’re a PI and became a data contributor before May 5th, you are already a project data manager and can access the “My Projects” page!  
  • This new role does not affect access to datasets on ESS-DIVE. Project data managers will not receive default permission to access any datasets.

 

Create and share data with project teams

  • Project data managers can now create Project Teams on ESS-DIVE. Teams are groups of ESS-DIVE users, and provide a new way of efficiently sharing data permissions with those groups!

  • For project data managers: start creating teams for your project to leverage group data sharing in the future!
    • Choose team names that will help data contributors know which team they should share their data with.
    • Add/remove team members at any time to manage who can access datasets or data portals shared with that team.  
  • For data contributors: easily share your datasets with a group of people all at once, instead of sharing with one person at a time. Ask your project data managers which teams to share your data with.
  • Teams do not receive access to any datasets by default. Work with your project’s data contributors to enable team access to datasets.

ESS-DIVE at ESS CI Annual Meeting and ESS PI Meeting

by

The ESS-DIVE Team is looking forward to participating in the Environmental System Science (ESS) Cyberinfrastructure (CI) Working Group Annual Meeting and ESS PI Meeting on May 15th and May 16th-17th, respectively. Deb Agarwal, Charu Varadharahan, Shreyas Cholia, Joan Damerow, and Emily Robles will all be in-person for the ESS CI and ESS PI meetings! Below are some ways that you can engage with ESS-DIVE at both meetings:

ESS CI Working Group Annual Meeting – May 15th, 2023

Project Updates: ESS-DIVE 

Presenter: Charu Varadharajan, ESS-DIVE Co-PI

Date and Time: May 15th, 2023; 9:30 AM – 9:40 AM ET / 6:30 AM – 6:40 AM PT

Flash Talk: Interactive Supercomputing With Jupyter

Presenter: Shreyas Cholia, ESS-DIVE Co-PI, Lead: Scaling, Resilience and Integration

Date and Time: May 15th, 2023; 10:45 AM – 12:00 PM ET / 7:45 AM – 9:00 AM PT

Data: Breakout Round Robin

Leads: Joan Damerow, Lead Scientist; Ben Sulman (ORNL)

Date and Time: May 15th, 2023; 3:15 PM – 4:15 PM ET / 12:15 PM – 1:15 PM PT

Data Management supporting integration in ModEx, including 1) model inputs: types of data, data standards and metadata, gap filling, normalization and access, and 2) model outputs: formats, archiving and use in experimental design

Models: Breakout Round Robin

Leads: Charu Varadharajan, ESS-DIVE Co-PI; Ethan Coon (ORNL)

Date and Time: May 15th, 2023; 3:15 PM – 4:15 PM ET / 12:15 PM – 1:15 PM PT

Exploring the various types of watershed and ecosystem models, from phenomenological to process-based, their different roles and data needs and how best to use this broad range of models to address different scientific questions at different scales.

ESS PI Meeting – May 16th to 17th, 2023

ESS-DIVE tutorials: Managing project data on ESS-DIVE 

Presenter: Joan Damerow, Lead Scientist

Session Date and Time: May 16th, 2023; 10:30 AM -12:30 PM ET / 7:30 AM – 9:30 AM PT

Presentation Slides

This session is designed for PI’s and data managers who want to learn how to manage project data on ESS-DIVE. We will first provide an overview of how ESS-DIVE can support a project’s data management needs. The tutorial will then walk through how to use ESS-DIVE features for project data management, including: (1) new features to assign project data managers and create teams, (2) sharing data permissions with project members for collaborative editing and review, (3) creating portals to showcase project data collections, and (4) linking ess-dive datasets to other repositories.

ESS-DIVE tutorials: Creating and publishing high-quality datasets

Presenter: Emily Robles, Senior Research Associate 

Session Date and Time: May 16th, 2023; 1:00 PM – 2:30 PM ET / 10:00 AM – 11:30 AM PT

Presentation Slides

Learn how to upload and publish data with ESS-DIVE as a new contributor in this hands-on training session. We will cover all of the basics needed to get started with publishing datasets, from important considerations for organizing data and accompanying metadata, to choosing an upload method for dataset submission, and requesting publication and following the dataset review process. Our hands-on activity walks through creating and submitting a dataset to ESS-DIVE’s sandbox using our online submission form. We will also discuss ways that researchers can improve the quality of their datasets by using ESS-DIVE’s data and metadata formats.

Poster: ESS-DIVE Overview: A Scalable, User-focused Repository for Earth and Environmental

Presenter: Shreyas Cholia, ESS-DIVE Co-PI, Lead: Scaling, Resilience and Integration 

Poster: (Meta)data reporting formats to enable data reuse in ESS-DIVE

Presenter: Charu Varadharajan, ESS-DIVE Co-PI