Does anyone succeed to reproduce LB SMAPE score? - Biendata

^En/_中
Host a Competition
Login
Sign Up

Login
Sign Up

Discussion

Home Discussion Board KDD CUP of Fresh Air Does anyone succeed to reproduce LB SMAPE score?

akiyama

Does anyone succeed to reproduce LB SMAPE score?

posted in KDD CUP of Fresh Air

2018-04-12 10:37

23 comments

akiyama

2018-04-12 11:59

Reply

0

akiyama reply akiyama

2018-04-12 12:02

Reply

0

<pre><code class="lang-python">import pandas as pd import numpy as np def calc_smape(labels, forecasts): # arrange both indices forecasts = forecasts.loc[labels.index].copy() # calc symmetric absolute percentage error (not SMAPE) sape_df = ((labels-forecasts).abs()/((labels+forecasts)/2)) # set sape value 0 if actual value and forecast value are both 0 sape_df[(labels==0)&(forecasts==0)] = 0 # form the dataframe in line, drop nan, and calc SMAPE sape_array = sape_df.values.reshape([-1]) smape = sape_array[~np.isnan(sape_array)].mean() return smape # labels and forecasts have same format labels = pd.read_csv('labels_path', index_col=['test_id']) forecasts = pd.read_csv('forecasts_path', index_col=['test_id']) SMAPE = calc_smape(labels, forecasts) </code></pre>

akiyama reply akiyama

2018-04-12 12:03

Reply

0

<p>Sorry, I can not copy and paste well. </p>

Ren

2018-04-12 12:34

Reply

1

sunfanyun

2018-04-12 15:22

Reply

0

sunfanyun reply sunfanyun

2018-04-12 15:54

Reply

0

<p>or is it a consistent gap?</p>

Ren reply sunfanyun

2018-04-12 17:01

Reply

0

<p>For me, it is not consistant at all. </p> <p>Lets say I have two models A and B. On one day I have model A scored lower with my local script than the website, and mdoel B scored higher locally. Which is already frastrative. But, whats worse is that, the next day the direction can change, ie, model A score higher locally while B score lower locally.</p>

Ren reply sunfanyun

2018-04-12 17:03

Reply

0

<p>and the gap for me can range from -0.07 to +0.07, which is huge…</p>

akiyama reply sunfanyun

2018-04-13 01:52

Reply

0

<p>In my case also the same phenomenon occurs as Ren. But the gap is larger, range from -0.2 to +0.2. </p>

akiyama reply Ren

2018-04-13 01:56

Reply

0

<p>Thanks for sharing code. I can not judge whether my labels is incorrect or my calculation is incorrect, so I really want official scoring script. </p>

Ren reply sunfanyun

2018-04-13 02:58

Reply

0

<p>I am starting to see bigger gaps as april 10th results came out. I got + 0.13 difference from what they calculated and what I calculated.. </p>

/

/

/