CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing

Text | General | Response

CrisisBench is general-disaster dataset for informativeness detection and categorization of humanitarian tasks post-disaster. It contains 166,098 tweets for informativeness and 141,533 tweets for humanitarian classification. It is introduced for binary and multiclass classification.

  • ML task type: Binary classification, multiclass classification
  • Data Source: Social Media (Twitter)
  • Size: 166,098 Tweets for Informativeness; 141,533 Tweets for Humanitarian Classification
  • Timespan: 2010 - 2017
  • Geographical Coverage: Global
  • Baseline Information
  • Evaluated on: CNN, fastText, and pre-trained transformer models
  • Metrics used: Weighted average precision, recall, F1,
  • Results for Informativeness as reported in original paper: Precision: 0.879 (RoBERTa), Recall: 0.879 (RoBERTa) , F1: 0.878 (RoBERTa)
  • Results for Humanitarian classification as reported in original paper: Precision: 0.871 (RoBERTa), Recall: 0.870 (RoBERTa) , F1: 0.870 (RoBERTa);

Firoj Alam, Hassan Sajjad, Muhammad Imran, and Ferda Ofli. CrisisBench: Benchmarking crisis-related social media datasets for humanitarian information processing. In International Conference on Web and Social Media, pages 923–932, 2021.