Duplicate Entries Found in Training and Validation Datasets

Discussion

Home Discussion Board AQA-KDD-2024 Duplicate Entries Found in Training and Validation Datasets

misogawa

Duplicate Entries Found in Training and Validation Datasets

posted in AQA-KDD-2024

2024-05-27 06:50

23 comments

OAG-Challenge Team

Fanjin Zhang reply nothing

2024-05-29 09:09

Reply

0

Hi, we are studying and working on this.

nothing reply Fanjin Zhang

2024-05-28 11:15

Reply

0

Hi,Could you update the current leaderboard by excluding those samples? The private leaderboard is still several days away

OAG-Challenge Team

Fanjin Zhang

2024-05-28 06:48

NISH01

2024-05-28 05:29

Hi there.

I picked a specific example.

`qa_train.txt` contains the line:
```json
{"question": "What is the difference between problem solving and intelligence?", "body": "What is the difference between problem solving and intelligence?\n\nAll I know is that the tower of hanoi is thought of as a problem solving test, perhaps due to its lack of automation, I don't know. I'd assume that \"intelligence\" is broader, measured by iq tests and reflected in skills etc..\n\nIs it just that intelligence is <a href=\"https://psychology.stackexchange.com/questions/145/what-is-the-difference-between-solving-a-problem-and-acquiring-a-skill\">learnt</a>?\n", "pids": ["56d82022dabfae2eeeb68805"]}
```

And `qa_valid_wo_ans.txt` also has
```json
{"question": "What is the difference between problem solving and intelligence?", "body": "What is the difference between problem solving and intelligence?\n\nAll I know is that the tower of hanoi is thought of as a problem solving test, perhaps due to its lack of automation, I don't know. I'd assume that \"intelligence\" is broader, measured by iq tests and reflected in skills etc..\n\nIs it just that intelligence is <a href=\"https://psychology.stackexchange.com/questions/145/what-is-the-difference-between-solving-a-problem-and-acquiring-a-skill\">learnt</a>?\n"}
```

The question and body are exactly the same.
In this case, submitting `56d82022dabfae2eeeb68805` from the training data seems like the obvious choice and increases the leaderboard score, but it’s essentially meaningless.

There are 655 similar examples out of 2919.

This issue seems quite critical for this competition.

misogawa

2024-05-28 05:03

OAG-Challenge Team

Fanjin Zhang

2024-05-28 04:01