DSL Shared Task 2014 - Results

by Marcos Zampieri, Liling Tan (Saarland University), Nikola Ljubešić (University of Zagreb) and Jörg Tiedemann (Uppsala University)

We are happy to report that 22 teams registered for the shared task and 8 teams submitted systems,
many of which participated for all language/varieties groups for close submissions:

Evaluation

Evaluations is based on overall accuracy and per-group accuracy for every language/variety group.
The best performing system(s) are decided by the: where:

overall accuracy = sum(TP) / #sents
TP = True Positive for all languages/varieties
#sents = total number of documents in evaluation dataset (i.e. 13*1000 for Group A to E and 2*800 for Group F)

per-group accuracy = sum(TPgrp) / #sentsgrp
TPgrp = True Positive for languages/varieties in a group
#sentsgrp = total number of documents in each language/variety group (i.e. 1000 for Group A to E and 800 for Group F)

Results

For reproducibility, the code use to generate the results can be found here
Results for individual teams can be downloaded here
Detailed results can be viewed here

Overall accuracy for Group A to E (close submissions):

TeamMacro-avg F-scoreOverall Accuracy
NRC-CNRC0.95710.9571
RAE0.94660.9468
UMich0.93210.9321
UniMelb-NLP0.91760.9176
QMUL0.92510.9061
LIRA0.72080.7664
UDE0.65750.6815
CLCG0.40520.4533


Per-group best accuracy for Group A to F (close submissions):

GroupTeamAccuracy
ANRC-CNRC0.9360
BNRC-CNRC0.9955
CLIRA1.0000
 NRC-CNRC1.0000
 RAE1.0000
 UniMelb-NLP1.0000
DNRC-CNRC0.9560
ENRC-CNRC0.9095
FUMich0.6394


Overall accuracy for Group A-E (open submissions):

TeamMacro-avg F-scoreOverall Accuracy
UniMelb-NLP0.87830.8800
UMich0.85800.8589


Per-group best accuracy for Group F (open submissions):

TeamMacro-avg F-scoreOverall Accuracy
UMich0.63940.6394
UniMelb-NLP0.58380.5812

Organizers

DSL webpage: http://corporavm.uni-koeln.de/vardial/sharedtask.htm
  • Marcos Zampieri, Saarland University
  • Liling Tan, Saarland University
  • Nikola Ljubešić, University of Zagreb
  • Jörg Tiedemann, Uppsala University