MIT researchers have developed an approach for assessing predictions with a spatial dimension, like forecasting weather or mapping air pollution.
The paper was written by lead author and MIT postdoc David R Burt, EECS graduate student Yunyi Shen and Tamara Broderick, who is an associate professor in MIT’s department of electrical engineering and computer science (EECS), a member of the Laboratory for Information and Decision Systems and the Institute for Data, Systems, and Society, and an affiliate of the Computer Science and Artificial Intelligence Laboratory (CSAIL).
The technique assesses prediction-validation methods and uses them to prove that two classical methods can be substantively wrong on spatial problems. The researchers then determined why these methods can fail and created a new method designed to handle the types of data used for spatial predictions.
In experiments with real and simulated data, their new method provided more accurate validations than the two most common techniques. The researchers evaluated each method using realistic spatial problems, including predicting the wind speed at Chicago O’Hare Airport and forecasting the air temperature at five US metro locations.
Their validation method could be applied to a range of problems, from helping climate scientists predict sea surface temperatures to aiding epidemiologists in estimating the effects of air pollution on certain diseases.
“Hopefully, this will lead to more reliable evaluations when people are coming up with new predictive methods and a better understanding of how well methods are performing,” said Broderick.
Evaluating validations
The group has recently collaborated with oceanographers and atmospheric scientists to develop machine-learning (ML) prediction models that can be used for problems with a strong spatial component.
Through this work, they noticed that traditional validation methods can be inaccurate in spatial settings. These methods hold out a small amount of training data, called validation data, and use it to assess the accuracy of the predictor.
To find the root of the problem, they conducted a thorough analysis and determined that traditional methods make assumptions that are inappropriate for spatial data. Evaluation methods rely on assumptions about how validation data and the data one wants to predict, called test data, are related.
Traditional methods assume that validation data and test data are independent and identically distributed, which implies that the value of any data point does not depend on the other data points. However, in a spatial application, this is often not the case.
For instance, a scientist may be using validation data from EPA air pollution sensors to test the accuracy of a method that predicts air pollution in conservation areas. However, the EPA sensors are not independent – they were positioned based on the location of other sensors.
In addition, perhaps the validation data is from EPA sensors near cities, while the conservation sites are in rural areas. Because this data is from different locations, it likely has different statistical properties, so it is not identically distributed.
“Our experiments showed that you get some really wrong answers in the spatial case when these assumptions made by the validation method break down,” Broderick commented.
This led the researchers to come up with a new assumption.
Specifically spatial
Thinking specifically about a spatial context, where data is gathered from different locations, they designed a method that assumes validation data and test data vary smoothly in space. For instance, air pollution levels are unlikely to change dramatically between two neighboring houses.
“This regularity assumption is appropriate for many spatial processes, and it allows us to create a way to evaluate spatial predictors in the spatial domain. To the best of our knowledge, no one has done a systematic theoretical evaluation of what went wrong, to come up with a better approach,” said Broderick.
To use the new evaluation technique, someone would input their predictor, the locations they want to predict, and their validation data, then the technique automatically does the rest. In the end, it estimates how accurate the predictor’s forecast will be for the location in question. However, effectively assessing the validation technique proved to be a challenge.
“We are not evaluating a method. Instead, we are evaluating an evaluation. So, we had to step back, think carefully and get creative about the appropriate experiments we could use,” Broderick explained.
First, the researchers designed several tests using simulated data, which had unrealistic aspects but enabled them to carefully control key parameters. Then, they created more realistic, semi-simulated data by modifying real data. Finally, they used real data for several experiments.
Using three types of data from realistic problems, like predicting the price of an apartment in England based on its location and forecasting wind speed, enabled them to conduct a comprehensive evaluation. In most experiments, their technique was more accurate than either traditional method they compared it with.
In the future, the researchers plan to apply these techniques to improve uncertainty quantification in spatial settings. They also want to find other areas where the regularity assumption could improve the performance of predictors, such as with time-series data.
In related news, the Government of Fiji, in partnership with the UN Development Programme (UNDP) Pacific Office in Fiji, recently successfully concluded the final validation workshop for the country’s first-ever Green Climate Fund (GCF) Early Warnings for All (EW4ALL) Project proposal. Click here to read the full story.
