MOODA WaterFrame: an abstraction for enriched data harmonization and transport from data ingestion to data analytics

Abstract

MOODA (Module for Ocean Observatory Data Analysis) is an open-source python-based framework that allows easy management of data from different scientific instrumentation, platforms, and formats. It was initially developed within the EMSODEV European founded project for data quality control and plotting; however, it has evolved as part of a comprehensive data management ecosystem adopted by the European Multidisciplinary Seafloor and water column Observatory (EMSO ERIC). MOODA is built upon the WaterFrame data abstraction, which is an extension of the Pandas DataFrame structure and contains means for enriching the data with embedded metadata (e.g., data quality processes). This abstraction is used to harmonize data from different and heterogeneous sources using the OceanSites data format, which is based on the NetCDF Climate and Forecast Metadata Convention, instead of a project-specific standard. In addition to harmonization via extensible plug-ins, it also offers a powerful abstraction for data transport and a medium for implementing tools for data quality control, data management and analytics of data from multiple and heterogeneous sources, including visual analytics (e.g., Jupyter environments and dashboards). These are key aspects of the roadmap of the ENVRI-FAIR Eurpean H2020 project, which aims at implementing effective wide access to data and services from different research infrastructure across Europe according to FAIR principles. We use MOODA for harmonizing and enabling data analytics with a proof-of-concept study using essential ocean variables from data sets across EMSO ERIC and SeaDataNet research infrastructures. This approach is a pragmatic view to identify gaps and potential solutions for improving interoperability. The conclusions of our experiences include the need for developing curated catalogs of tools that can be accessed as a service and easily integrated within data workflows crossing the boundaries of single research infrastructures