Sea surface suspended concentrations prediction based on pre-trained transformer.

Sea surface suspended sediments (SSSC) monitoring is an important branch of marine engineering, which plays an important role in port transportation, marine habitat ecology, and the environment. However, understanding and monitoring the dynamics of suspended sediments remain challenging due to complex natural and anthropogenic interactions. Existing approaches, such as using suit data or satellite data to construct numerical models, face limitations in sampling irregular spaces or lacking generalization capabilities. Besides, unlike natural language processing (NLP) and computer vision (CV), the amount of suit data or satellite data is small, which make it difficult for some data-driven deep models to achieve excellent performance. To address these challenges, we propose a pre-trained transformer model that leverages an NLP pre-trained model trained on billions of tokens for SSSC reconstruction. Specifically, we design input embedding layers to project remote sensing image data into dimensions required by a specific pre-trained model. During the fine-tuning phase, we freeze the self-attention and feed-forward layers of the residual block in the pre-trained model, allowing knowledge transfer to the SSSC task. Experimental results demonstrate that our proposed pre-trained transformer models achieve state-of-the-art performance on the SSSC task. This result may indicate that parameters trained on massive data in large models have the potential to enhance performance on other tasks.