Injecting user models and time into precision via Markov chains

Marco Ferrante, Nicola Ferro, Maria Maistro

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

18 Citations (Scopus)

Abstract

We propose a family of new evaluation measures, called Markov Precision (MP), which exploits continuous-time and discrete-time Markov chains in order to inject user models into precision. Continuous-time MP behaves like timecalibrated measures, bringing the time spent by the user into the evaluation of a system; discrete-time MP behaves like traditional evaluation measures. Being part of the same Markovian framework, the time-based and rank-based versions of MP produce values that are directly comparable. We show that it is possible to re-create average precision using specific user models and this helps in providing an explanation of Average Precision (AP) in terms of user models more realistic than the ones currently used to justify it. We also propose several alternative models that take into account different possible behaviors in scanning a ranked result list. Finally, we conduct a thorough experimental evaluation of MP on standard TREC collections in order to show that MP is as reliable as other measures and we provide an example of calibration of its time parameters based on click logs from Yandex.

Original languageEnglish
Title of host publicationSIGIR 2014 - Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval
Number of pages10
PublisherASSOCIATION FOR COMPUTING MACHINERY. JOU
Publication date1 Jan 2014
Pages597-606
ISBN (Print)9781450322591
DOIs
Publication statusPublished - 1 Jan 2014
Externally publishedYes
Event37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2014 - Gold Coast, QLD, Australia
Duration: 6 Jul 201411 Jul 2014

Conference

Conference37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2014
Country/TerritoryAustralia
CityGold Coast, QLD
Period06/07/201411/07/2014
SponsorBaidu, et al., Google, Microsoft Research, Special Interest Group on Information Retrieval (ACM SIGIR), Tourism and Events Queensland

Keywords

  • Evaluation
  • Markov precision
  • Time
  • User model

Cite this