18

Stack Exchange recently started a collaboration project with the University of Melbourne, in an attempt to improve the automatic detection of duplicate questions.

More information on the project and a call for help from the community can be found here: Project Reduplication of Deduplication Has Begun!

In it, Tim Post mentions that Doris Hoogeveen (me) will post a message on the different meta sites to let you know that we're ready for help. This is it!

As a brief summary, the project is about finding ways to automatically determine if two questions are duplicates, and also to detect the opposite: falsely tagged duplicates. To validate our methods, we need manual judgements from people who really know the data. This is where we need your help. It is impossible to manually annotate all the data from Stack Exchange, and so we have applied several filters to make sure we only show you questions pairs with a high likelihood of being duplicates.

Participation is completely optional, and there's no minimum time requirement. There's a reputation requirement however: only people who have duplicate question tagging rights on the site can participate. That is people with at least 3000 reputation points, or a gold badge. This is to ensure the annotations are compatible with StackExchange's quality standard.

If you are interested in participating in the project, please head over to the annotation interface, which can be found here: http://hum.csse.unimelb.edu.au/se-annotate/

You can log in with your Stack Exchange account, so no sign up is required. The site also contains some more information on the project, which might answer some questions you may have. And I'm also happy to answer them here.

Thank you!

5
  • Gee, I'd better get those remaining 91 rep points fast!
    – Dmitry Grigoryev
    Commented Nov 21, 2016 at 17:21
  • @DmitryGrigoryev, that would be great! Thanks a lot for wanting to help out.
    – Monozygotic
    Commented Nov 21, 2016 at 22:15
  • I'm not a big helper actually. I've once missed free T-shirts on one of the SE sites by a few rep points, that's why I react like that ;)
    – Dmitry Grigoryev
    Commented Nov 22, 2016 at 17:56
  • Is there an 'end' date for this project?
    – Jeff Schaller Mod
    Commented Nov 23, 2016 at 15:12
  • @JeffSchaller, we haven't determined when exactly the end date will be, but it will be at least a month from now. I'll let you know when we've settled on an exact date.
    – Monozygotic
    Commented Nov 25, 2016 at 3:53

2 Answers 2

Reset to default
4

From what I have seen so far, your dataset contains very few duplicates. Now, I do understand that this is something normal: your goal is to improve the algorithm, so in the beginning it's expected to have somewhat low signal-to-noise ratio. But I'm afraid you risk getting biased data because of this: if someone has to click on "NO DUPLICATES" 10 times before seeing a pair of somewhat related questions, the urge to click on "DUPLICATE" button is strong.

Have you considered to artificially add some known duplicates in the mix, at least in the beginning?

5
  • Interesting idea. I have not considered that. I was actually worried I would get comments about questions already having been tagged as duplicates on the site, but not in my dataset yet, because the latest dump is always a little bit behind on the actual status. I'm basically worried about annoying people if I show them known duplicates. You do have a point though, and this is definitely something I can discuss with the other researchers.
    – Monozygotic
    Commented Nov 22, 2016 at 22:17
  • 1
    @Monozygotic nah, SE already does that to us via review audits, and most users are OK with it. People who want to help with this should be mostly fine with it too.
    – muru
    Commented Nov 23, 2016 at 2:00
  • @muru, ha, I didn't know that! Great, that's encouraging. I'll speak to the other researchers tomorrow, so will bring this point up.
    – Monozygotic
    Commented Nov 23, 2016 at 4:43
  • 1
    @muru and Dmitry, I had a discussion with the other researchers, but we've decided not to add known duplicates. The main reason is that it's virtually impossible to eliminate all bias. Apparently a lot of research has been done on this by psychologists. So instead, we'll try to get as many annotations as possible, to make the received labels as trustworthy as possible. Different sites have different numbers of duplicates, so we might be able to use this info to identify cases where there's a lot of bias too. We'll have to experiment when we have all the data.
    – Monozygotic
    Commented Nov 25, 2016 at 3:55
  • @Monozygotic Ok, thanks for letting me know and good luck!
    – Dmitry Grigoryev
    Commented Nov 25, 2016 at 8:18
2

The project is drawing to a close, and I am planning to take down the annotation interface this weekend.

Thank you very much to everyone who helped out, and thank you for all the valuable feedback and suggestions. Hopefully the collected annotations will lead to the development of better duplicate question detection systems, especially for StackExchange.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .