On the optimal data processing of the Soil Moisture and Ocean Salinity measurements

Abstract

The amount of data that must be processed in satellite missions is increasing over time, directly affecting the hardware resources and time required to carry out this processing. With more than 11 years in orbit, the SMOS mission has a lot of over-sampled data, which implies a more intensive use of the CPU and greater use of disk space if the processing is done without any type of data management. For this reason, it is increasingly necessary to optimize the resources involved in the processing of large volumes of data. Such optimizations include minimizing the processing time, achieving maximum efficiency of computational resources, and doing a good management of the generated data, both to make it more accessible and to optimize the disk space it demands. This work presents different techniques that can be applied when designing software architectures for the particular case of the SMOS Sea Surface Salinity data processing. A study is made on how the data can be aggregated and ordered in the first stages of processing to reduce the processing time of the following stages and the disk usage of intermediate products. The SMOS measurements can be easily divided into smaller independent processing units (such as a half-orbit or a snapshot, which is even smaller and still independent of the other snapshots). The granularity of the data allows the processing to be divided into very small pieces that can be executed in parallel, making an optimal use of CPU resources and reducing the total processing time. Disk operations, such as reading and writing files are also a big part of the processing time. Data has been arranged in a way in which disk operations are minimized (avoiding multiple reads of the same file) . Preliminary results show an improvement of the 20% of computational time and a reduction of the 40% of the required disk space with respect to the current implementation of the Barcelona Expert Center internal data processing chain