Evaluation

On each day, the submitted values will be compared to the ground truth (i.e. pollution concentration in the real world) via Symmetric mean absolute percentage error:

If actual value and forecast value are both 0, we will set SMAPE score 0, too.

Participants are requested to predict the PM2.5, PM10, and O3 concentration levels over the coming 48 hours for every measurement station in Beijing (another city will be disclosed on 3/31). Predictions can be made on the daily basis, over the course of one month. To evaluate the predictions, on each day, SMAPE scores will be calculated for each station, each hour of the day (48 hours overall), and each pollutant (PM2.5, PM10, and O3). The daily SMAPE score will then be the average of all the individual SMAPE scores.

The final score of each team in this competition is the average of 25 lowest daily SMAPE scores. If a team miss a day, the score of this team on that day will be imputed by the baseline score from the model published by the organizer. The top 10 teams will be declared as the winners of the competition.

If any station fails to return a result on a given stamp, the score on this point won't be calculated.

Evaluation Code

Our evaluation code (in python3) is as following:

import numpy as np

def smape(actual, predicted):
    dividend= np.abs(np.array(actual) - np.array(predicted))
    denominator = np.array(actual) + np.array(predicted)
    
    return 2 * np.mean(np.divide(dividend, denominator, out=np.zeros_like(dividend), where=denominator!=0, casting='unsafe'))

Please note:

1. For all the NA values in submission files, we will replace them by 0.

2. All data points that have NA values or negative values in PM2.5 or PM10 (or O3 in Beijing) by the time we evaluate, will be considered as invalid data and dropped from the truth file. For example, the data [2957631,CT3,2018-04-13 00:00:00,24.0,,15.7,,,] has no PM2.5 data, so it will be dropped from scoring matrix, even though it has PM10 data.

3, The submission file shall not contain any negative value.

Tiebreaker

In case of potential ties in scores, here are the steps in a sequence we will take to break a tie:

1) The team with more submissions in the final stage is ranked higher.

2) If there is still a tie, the team with earlier submissions in average in the final stage will be ranked higher. In each day, the time difference (delta) between the submission with the lowest SMAPE score in that day and the submission deadline will be considered.

Here,

The average delta among the submission days will be used to break the tie. Since submission should always be before the submission deadline, delta should always be positive. Submission after a deadline will be automatically considered as the submission for the next deadline.

Please note that the smaller score values indicate better performance.