Saved datasets
Last updated
Download format
Usage rights
License from data provider
Please review the applicable license to make sure your contemplated use is permitted.
Topic
Provider
Free
Cost to access
Described as free to access or have a license that allows redistribution.
2 datasets found
  1. Webis Query Segmentation Corpus 2010 (Webis-QSeC-10)

    • zenodo.org
    zip
    Updated Jan 24, 2020
    + more versions
  2. Webis Query Spelling Corpus 2017 (Webis-QSpell-17)

    • zenodo.org
    zip
    Updated Jan 5, 2022
    + more versions
  3. Not seeing a result you expected?
    Learn how you can add new datasets to our index.

Share
FacebookFacebook
TwitterTwitter
Email
Click to copy link
Link copied
Close
Cite
Matthias Hagen; Matthias Hagen; Martin Potthast; Martin Potthast; Benno Stein; Benno Stein; Christof Bräutigam; Anna Beyer; Christof Bräutigam; Anna Beyer (2020). Webis Query Segmentation Corpus 2010 (Webis-QSeC-10) [Dataset]. http://doi.org/10.5281/zenodo.3256198
Organization logo

Webis Query Segmentation Corpus 2010 (Webis-QSeC-10)

Explore at:
zipAvailable download formats
Dataset updated
Jan 24, 2020
Dataset provided by
Zenodohttp://zenodo.org/
Authors
Matthias Hagen; Matthias Hagen; Martin Potthast; Martin Potthast; Benno Stein; Benno Stein; Christof Bräutigam; Anna Beyer; Christof Bräutigam; Anna Beyer
License

Attribution 4.0 (CC BY 4.0)https://creativecommons.org/licenses/by/4.0/
License information was derived automatically

Description

The Webis Query Segmentation Corpus 2010 (Webis-QSeC-10) contains segmentations for 53,437 web queries obtained from Mechanical Turk crowdsourcing (4,850 used for training in our CIKM 2012 paper). For each query, at least 10 MTurk workers were asked to segment the query. The corpus represents the distribution of their decisions.

We provide the training and test sets as single folders in Zip archives containing several files. The files "...-queries.txt" contain the query strings and a unique ID for each query. The files "...-segmentations-crowdsourced.txt" contain the crowdsourced segmentations with their number of votes per query ID (see below for an example). The "data" folders contain all the data (n-gram frequencies, PMI values, POS tags, etc.) needed to replicate the evaluation results of our proposed segmentation algorithms. For convenience reasons, the folder "segmentations-of-algorithms" contain the segmentations that our proposed algorithms compute.

The original queries were extracted from the AOL query log, and range from 3 to 10 keywords in length. For each query at least 10 MTurk workers were asked to segment the query and their decisions are accumulated in the corpus. The examples below demonstrate two different cases.

Sample queries with internal ID (as in "Webis-QSeC-10-training-set-queries.txt"):

  • 2315313155 harvard community credit union
  • 1858084875 women's cycling tops

Sample segmentations (as in "webis-qsec-10-training-set-segmentations-crowdsourced.txt"):

  • 2315313155 [(6, 'harvard community credit union'), (2, 'harvard community|credit union'), (1, 'harvard|community|credit union'), (1, 'harvard|community credit union')]
  • 1858084875 [(5, "women's|cycling tops"), (2, "women's|cycling|tops"), (2, "women's cycling|tops"), (1, "women's cycling tops")]

Each query has a unique internal ID (e.g., 2315313155 in the first example) and the segmentations file contains at least 10 different decisions the MTurk workers made for that query. In the first example, 6 workers have all 4 keywords in one segment, 2 workers decided to break after the second word (denoted by a |) etc. Note that apostrophe in the second example (query ID 1858084875) is escaped by double quotes around the segmentation strings.

Search
Clear search
Close search
Google apps
Main menu