In recent years, there has been significant progress in the development of fully data-driven global numerical weather prediction models. These machine learning weather prediction models have their strength, notably accuracy and low computational requirements, but also their weakness: they struggle to represent fundamental dynamical balances, and they are far from being suitable for data assimilation experiments. Hybrid modelling emerges as a promising approach to address these limitations. Hybrid models integrate a physics-based core component with a statistical component, typically a neural network, to enhance prediction capabilities. In this article, we propose to develop a model error correction for the operational Integrated Forecasting System (IFS) of the European Centre for Medium-Range Weather Forecasts using a neural network. The neural network is initially pre-trained offline using a large dataset of operational analyses and analysis increments. Subsequently, the trained network is integrated into the IFS within the Object-Oriented Prediction System (OOPS) so as to be used in data assimilation and forecast experiments. It is then further trained online using a recently developed variant of weak-constraint 4D-Var. The results show that the pre-trained neural network already provides a reliable model error correction, which translates into reduced forecast errors in many conditions and that the online training further improves the accuracy of the hybrid model in many conditions.

A data-driven model (DDM) suitable for regional weather forecasting applications is presented. The model extends the Artificial Intelligence Forecasting System by introducing a stretched-grid architecture that dedicates higher resolution over a regional area of interest and maintains a lower resolution elsewhere on the globe. The model is based on graph neural networks, which naturally affords arbitrary multi-resolution grid configurations. The model is applied to short-range weather prediction for the Nordics, producing forecasts at 2.5 km spatial and 6 h temporal resolution. The model is pre-trained on 43 years of global ERA5 data at 31 km resolution and is further refined using 3.3 years of 2.5 km resolution operational analyses from the MetCoOp Ensemble Prediction System (MEPS). The performance of the model is evaluated using surface observations from measurement stations across Norway and is compared to short-range weather forecasts from MEPS. The DDM outperforms both the control run and the ensemble mean of MEPS for 2 m temperature. The model also produces competitive precipitation and wind speed forecasts, but is shown to underestimate extreme events.

One of the guiding principles for designing AI-based weather forecasting systems is to embed physical constraints as inductive priors in the neural network architecture. A popular prior is locality, where the atmospheric data is processed with local neural interactions, like 3D convolutions or 3D local attention windows as in Pangu-Weather. On the other hand, some works have shown great success in weather forecasting without this locality principle, at the cost of a much higher parameter count. In this paper, we show that the 3D local processing in Pangu-Weather is computationally sub-optimal. We design ArchesWeather, a transformer model that combines 2D attention with a column-wise attention-based feature interaction module, and demonstrate that this design improves forecasting skill. ArchesWeather is trained at 1.5{\deg} resolution and 24h lead time, with a training budget of a few GPU-days and a lower inference cost than competing methods. An ensemble of four of our models shows better RMSE scores than the IFS HRES and is competitive with the 1.4{\deg} 50-members NeuralGCM ensemble for one to three days ahead forecasting. Our code and models are publicly available at https://github.com/gcouairon/ArchesWeather.

Post-processing ensemble prediction systems can improve the reliability of weather forecasting, especially for extreme event prediction. In recent years, different machine learning models have been developed to improve the quality of weather post-processing. However, these models require a comprehensive dataset of weather simulations to produce high-accuracy results, which comes at a high computational cost to generate. This paper introduces the ENS-10 dataset, consisting of ten ensemble members spanning 20 years (1998-2017). The ensemble members are generated by perturbing numerical weather simulations to capture the chaotic behavior of the Earth. To represent the three-dimensional state of the atmosphere, ENS-10 provides the most relevant atmospheric variables at 11 distinct pressure levels and the surface at 0.5-degree resolution for forecast lead times T=0, 24, and 48 hours (two data points per week). We propose the ENS-10 prediction correction task for improving the forecast quality at a 48-hour lead time through ensemble post-processing. We provide a set of baselines and compare their skill at correcting the predictions of three important atmospheric variables. Moreover, we measure the baselines' skill at improving predictions of extreme weather events using our dataset. The ENS-10 dataset is available under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

OpenCUBE aims to develop an open-source full software stack for Cloud computing blueprint deployed on EPI hardware, adaptable to emerging workloads across the computing continuum. OpenCUBE prioritizes energy awareness and utilizes open APIs, Open Source components, advanced SiPearl Rhea processors, and RISC-V accelerator. The project leverages representative workloads, such as cloud-native workloads and workflows of weather forecast data management, molecular docking, and space weather, for evaluation and validation.

This document is one of the deliverable reports created for the ESCAPE project. ESCAPE stands for Energy-efficient Scalable Algorithms for Weather Prediction at Exascale. The project develops world-class, extreme-scale computing capabilities for European operational numerical weather prediction and future climate models. This is done by identifying Weather & Climate dwarfs which are key patterns in terms of computation and communication (in the spirit of the Berkeley dwarfs). These dwarfs are then optimised for different hardware architectures (single and multi-node) and alternative algorithms are explored. Performance portability is addressed through the use of domain specific languages. Atlas has been presented in deliverable D1.3. With this deliverable D2.3, a first version of the Atlas software libraries is publicly released with a permissive open-source license. The software is freely available for download, and contains a user guide and installation instructions. The Atlas libraries have been carefully designed with the user's perspective in mind. Even though Atlas is mainly coded in C++, an equivalent Fortran interface is presented without additional runtime overhead. The Fortran interfaces are provided to accommodate existing NWP and climate models that typically consist of Fortran subroutines. The mixed Fortran/C++ design enhances interoperability between NWP and climate models and novel data management techniques. Atlas provides interoperability with accelerator hardware, and can serve as foundation to support higher level abstractions as used in domain specific languages.

We introduce Devito, a new domain-specific language for implementing high-performance finite difference partial differential equation solvers. The motivating application is exploration seismology where methods such as Full-Waveform Inversion and Reverse-Time Migration are used to invert terabytes of seismic data to create images of the earth's subsurface. Even using modern supercomputers, it can take weeks to process a single seismic survey and create a useful subsurface image. The computational cost is dominated by the numerical solution of wave equations and their corresponding adjoints. Therefore, a great deal of effort is invested in aggressively optimizing the performance of these wave-equation propagators for different computer architectures. Additionally, the actual set of partial differential equations being solved and their numerical discretization is under constant innovation as increasingly realistic representations of the physics are developed, further ratcheting up the cost of practical solvers. By embedding a domain-specific language within Python and making heavy use of SymPy, a symbolic mathematics library, we make it possible to develop finite difference simulators quickly using a syntax that strongly resembles the mathematics. The Devito compiler reads this code and applies a wide range of analysis to generate highly optimized and parallel code. This approach can reduce the development time of a verified and optimized solver from months to days.

Data assimilation of atmospheric observations traditionally relies on variational and Kalman filter methods. Here, an alternative neural-network data assimilation (NNDA) with variational autoencoder (VAE) is proposed. The three-dimensional variational (3D-Var) data assimilation cost function is utilised to determine the analysis that optimally fuses simulated observations and the encoded short-range persistence forecast (background), accounting for their errors. The minimisation is performed in the reduced-order latent space, discovered by the VAE. The variational problem is auto-differentiable, simplifying the computation of the cost function gradient necessary for efficient minimisation. We demonstrate that the background-error covariance ($\mathbf{B}$) matrix measured and represented in the latent space is quasi-diagonal. The background-error covariances in the grid-point space are flow-dependent, evolving seasonally and depending on the current state of the atmosphere. Data assimilation experiments with a single temperature observation in the lower troposphere indicate that the $\mathbf{B}$-matrix simultaneously describes both tropical and extratropical background-error covariances.

This document is one of the deliverable reports created for the ESCAPE project. ESCAPE stands for Energy-efficient Scalable Algorithms for Weather Prediction at Exascale. The project develops world-class, extreme-scale computing capabilities for European operational numerical weather prediction and future climate models. This is done by identifying weather & climate dwarfs which are key patterns in terms of computation and communication (in the spirit of the Berkeley dwarfs). These dwarfs are then optimised for different hardware architectures (single and multi-node) and alternative algorithms are explored. Performance portability is addressed through the use of domain specific languages. This deliverable contains the description of the characteristics of a second set of so-called numerical weather & climate prediction dwarfs that form key functional components of prediction models in terms of the science that they encapsulate and in terms of computational cost they impose on the forecast production. The ESCAPE work flow between work packages centres on these dwarfs and hence their selection, their performance assessment, code adaptation and optimisation is crucial for the success of the project. These new dwarfs have been chosen with the purpose of extending the range of computational characteristic represented by the dwarfs previously selected in batch 1 (see Deliverable D1.1). The dwarfs have been made, their documentation has been compiled and the software has been made available on the software exchange platform. The dwarfs in this deliverable include a multigrid elliptic solver, a novel advection scheme for unstructured meshes, an advection scheme for structured meshes and a radiation scheme. This deliverable includes their scientific description and the guidance for installation, execution and testing.

This document is one of the deliverable reports created for the ESCAPE project. ESCAPE stands for Energy-efficient Scalable Algorithms for Weather Prediction at Exascale. The project develops world-class, extreme-scale computing capabilities for European operational numerical weather prediction and future climate models. This is done by identifying weather & climate dwarfs which are key patterns in terms of computation and communication (in the spirit of the Berkeley dwarfs). These dwarfs are then optimised for different hardware architectures (single and multi-node) and alternative algorithms are explored. Performance portability is addressed through the use of domain specific languages. This deliverable contains the description of the characteristics of the weather & climate dwarfs that form key functional components of prediction models in terms of the science that they encapsulate and in terms of computational cost they impose on the forecast production. The ESCAPE work flow between work packages centres on these dwarfs and hence their selection, their performance assessment, code adaptation and optimization is crucial for the success of the project. At this stage of ESCAPE, a selection of established and new dwarfs has been made, their documentation been compiled and the software been made available on the software exchange platform. The selection of dwarfs will be extended throughout the course of the project (see Deliverable D1.2). The current selection includes the spectral transforms, the cloud microphysics scheme, two and three-dimensional elliptic solvers, a bi-Fourier spectral transform, an interpolation needed for the semi-Lagrangian advection scheme and a first version of the semi-Lagrangian advection scheme itself. This deliverable includes their scientific description and the guidance for installation, execution and testing.

Long-range ensemble forecasts are typically verified as anomalies with respect to a lead-time dependent climatological mean to remove the influence of systematic biases. However, common methods for calculating anomalies result in statistical inconsistencies between forecast and verification anomalies, even for a perfectly reliable ensemble. It is important to account for these systematic effects when evaluating ensemble forecast systems, particularly when tuning a model to improve the reliability of forecast anomalies or when comparing spread-error diagnostics between systems with different reforecast periods. Here, we show that unbiased variances and spread-error ratios can be recovered by deriving estimators that are consistent with the values that would be achieved when calculating anomalies relative to the true, but unknown, climatological mean. An elegant alternative is to construct forecast climatologies separately for each member, which ensures that forecast and verification anomalies are defined relative to reference climatological means with the same sampling uncertainty. This alternative approach has no impact on forecast ensemble means but systematically modifies the total variance and ensemble spread of forecast anomalies in such a way that anomaly-based spread-error ratios are unbiased without any explicit correction for climatology sample size. Furthermore, the improved statistical consistency of forecast and verification anomalies means that probabilistic forecast skill is optimised when the underlying forecast is also perfectly reliable. Alternative methods for anomaly calculation can thus impact probabilistic forecast skill, especially when anomalies are defined relative to climatologies with a small sample size. Finally, we demonstrate the equivalence of anomalies calculated using different methods after applying an unbiased statistical calibration.

Recent studies have suggested the existence of a `signal-to-noise paradox' (SNP) in ensemble forecasts that manifests as situations where the correlation between the forecast ensemble mean and the observed truth is larger than the correlation between the forecast ensemble mean and individual forecast members. A perfectly reliable ensemble, in which forecast members and observations are drawn from the same underlying probability distribution, will not exhibit an SNP if sample statistics can be evaluated using a sufficiently large ensemble size ($N$) over a sufficiently large number of independent cases ($M$). However, when $M$ is finite, an apparent SNP will sometimes occur as a natural consequence of sampling uncertainty, even in a perfectly reliable ensemble with many members. In this study, we evaluate the forecast skill, reliability characteristics, and signal-to-noise properties of three large-scale atmospheric circulation indices in 100-member subseasonal reforecasts. Consistent with recent studies, this reforecast dataset exhibits an apparent SNP in the North Atlantic Oscillation (NAO) at subseasonal lead times. However, based on several lines of evidence, we conclude that the apparent paradox in this dataset is a consequence of large observational sampling uncertainties that are insensitive to ensemble size and common to all model comparisons over the same period. Furthermore, we demonstrate that this apparent SNP can be eliminated by application of an unbiased reliability calibration. However, this is achieved through overfitting such that sample statistics from calibrated forecasts inherit the large sampling uncertainties present in the observations and thus exhibit unphysical variations with lead time. Finally, we make several recommendations for the robust and unbiased evaluation of reliability and signal-to-noise properties in the presence of large sampling uncertainties.

Implicit time-stepping for advection is applied locally in space and time where Courant numbers are large, but standard explicit time-stepping is used for the remaining solution which is typically the majority. This adaptively implicit advection scheme facilitates efficient and robust integrations with long time-steps while having negligible impact on the overall accuracy, and achieving monotonicity and local conservation on general meshes. A novel and important aspect for the efficiency of the approach is that only one linear solver iteration is needed for each advection solve. The implementation in this paper uses a second-order Runge-Kutta implicit/explicit time-stepping in combination with a second/third-order finite volume spatial discretisation. We demonstrate the adaptively implicit advection in the context of deformational flow advection on the sphere and a fully compressible model for atmospheric flows. Tracers are advected over the poles of latitude-longitude grids with very large Courant numbers and through hexagonal and cubed-sphere meshes with the same algorithm. Buoyant flow simulations with strong local updrafts also benefit from adaptively implicit advection. Stably stratified flow simulations require a stable combination of implicit treatment of gravity and acoustic waves as well as advection in order to achieve long stable time-steps.

Recent studies have shown that it is possible to combine machine learning methods with data assimilation to reconstruct a dynamical system using only sparse and noisy observations of that system. The same approach can be used to correct the error of a knowledge-based model. The resulting surrogate model is hybrid, with a statistical part supplementing a physical part. In practice, the correction can be added as an integrated term (i.e. in the model resolvent) or directly inside the tendencies of the physical model. The resolvent correction is easy to implement. The tendency correction is more technical, in particular it requires the adjoint of the physical model, but also more flexible. We use the two-scale Lorenz model to compare the two methods. The accuracy in long-range forecast experiments is somewhat similar between the surrogate models using the resolvent correction and the tendency correction. By contrast, the surrogate models using the tendency correction significantly outperform the surrogate models using the resolvent correction in data assimilation experiments. Finally, we show that the tendency correction opens the possibility to make online model error correction, i.e. improving the model progressively as new observations become available. The resulting algorithm can be seen as a new formulation of weak-constraint 4D-Var. We compare online and offline learning using the same framework with the two-scale Lorenz system, and show that with online learning, it is possible to extract all the information from sparse and noisy observations.

An Ensemble of Data Assimilations (EDA) can provide valuable information on the analysis and short-range forecast uncertainties. The present ECMWF operational ocean analysis and reanalysis system, called ORAS5, produces an ensemble but does not exploit it for the specification of the background-error covariance matrix $\mathbf{B}$, a key component of the data assimilation system. In this article, we describe EDA developments for the ocean, which take advantage of the short-range forecast ensemble for specifying, in two distinct ways, parameters of a covariance model representation of $\mathbf{B}$. First, we generate a climatological ensemble over an extended period to produce seasonally varying climatological estimates of background-error variances and horizontal correlation length-scales. Second, on each assimilation cycle, we diagnose flow-dependent variances from the ensemble and blend them with the climatological estimates to form hybrid variances. We also use the ensemble to diagnose flow-dependent vertical correlation length-scales. We demonstrate for the Argo-rich period that this new, hybrid formulation of $\mathbf{B}$ results in a significant reduction of background errors compared to the parameterized formulation of $\mathbf{B}$ used in ORAS5. The new ocean EDA system will be employed in ORAS6, ECMWF's next generation ocean reanalysis system.