Machine learning is becoming an increasingly important component of weather forecasting. At ECMWF, a team of three people has been established to coordinate related activities. They expect machine learning to continue to play a growing role in weather forecasting in the years to come.
The team has taken over from Peter Düben, who previously coordinated machine learning work at ECMWF. It includes members of three key departments: Christine Kitchen, deputy director of computing; Victoria Bennett, the head of the user services section in the forecasting department; and Matthew Chantry, a scientist in the research department.
“Our role is to oversee machine learning activities and to drive forward strategic issues and progress,” said Bennett.
Kitchen added, “We represent a range of skills and backgrounds as machine learning may be applied across different parts of the weather forecasting process.”
The only active machine learning scientist of the three is Chantry. “I’ve been working on machine learning for about five years, so the three of us have quite different sets of skills that we can bring to this role,” he said.
What is machine learning?
Machine learning is a concept that lies between the more general idea of artificial intelligence and the narrower subfield of deep learning. It refers to building models based on sample data to make predictions or take decisions. In numerical weather prediction, this contrasts with the traditional approach of using the laws of physics to make predictions.
This does not mean that machine learning would entail a loss of control over the weather prediction process.
“In applying machine learning, we don’t allow a computer to go away and learn with very little control over it. Rather, we have trained an algorithm to do a specific task. We know how it was trained to do something and what it was trained to do,” said Chantry.
Machine learning in weather forecasting
At ECMWF, machine learning has a role to play in many parts of the process of arriving at a numerical weather forecast and beyond. Examples are the quality control of weather observations, the learning of model error within data assimilation and the emulation of model components.
In the area of observations, for example, ECMWF has developed a machine learning tool for the detection and classification of observation anomalies. This will improve the classification of events by severity and cause and will help to monitor satellite data from a growing number of platforms. It is still being tested, but it represents an application that is close to operational use.
Data assimilation is a field that has some similarities with machine learning, in that observational data is used to correct model output. In addition, in data assimilation attempts are being made to combine a physical model with machine learning methods to estimate and correct model error.
In the field of numerical weather forecasts, progress has been made in replacing traditional tools that are computationally very costly with machine learning tools, for example in the areas of the radiation scheme and gravity wave drag parametrization. But very high demands in terms of stability and accuracy have to be met before deploying these methods.
The current growth in machine learning in the area of weather prediction is partly the result of more data being available.
“Some of the concepts behind machine learning have been around for a long time, but over the last few years the availability of data has witnessed rapid growth, and technology has evolved to the point of allowing some of the deeper levels of computation to be accomplished,” said Kitchen.
The growing role of machine learning in numerical weather and climate predictions is reflected in a Massive Open Online Course (MOOC) on the topic, which ECMWF is running until April this year.
What the future may hold
Opinions vary as to what extent machine learning may replace more traditional methods in the future. At a recent ECMWF-European Space Agency workshop on machine learning for Earth observation and prediction, some participants suggested that it will eventually replace today’s methods in numerical weather prediction.
This belief is fueled by the availability of high-quality historical data sets, such as ECMWF’s ERA5 reanalysis of the global weather and climate. ERA5 covers many decades and has been produced using traditional methods.
Most of those taking part in the conference foresaw a more complementary coexistence of traditional and machine learning techniques in weather prediction: in aspects of data assimilation, forecasting and post-processing.
“Nevertheless, I wouldn’t rule out machine learning being a big component of the forecasting system in ten years’ time if progress continues at the current rate,” said Chantry.
He points out that the computational resources required to make a prediction using machine learning are much smaller than in traditional methods. This would make it possible, for example, to use a much bigger number of ensemble members than in current forecasts.
Meanwhile, ECMWF’s machine learning coordinators are keeping a close eye on developments both internally and externally.
“We aim to make sure the right kind of people are talking to each other internally, but I also see us as visible contact points for external parties in this field to establish connections to the center,” said Bennett. “We also try to ensure that more machine learning work can be carried through to operations.”
This article was originally posted on the ECMWF website, and can be viewed here.
Opening paragraph adapted to suit the Meteorological Technology International website. Content shared under Creative Commons — Attribution 4.0 International — CC BY 4.0
