ir_measures

New: Explore IR measures using our demo at demo.ir-measur.es!

Check out our documentation website: ir-measur.es

Provides a common interface to many IR measure tools.

Provided by the Terrier Team @ Glasgow. Find us at terrierteam/ir_measures.

Getting Started

Install via pip

pip install ir-measures

Python API

import ir_measures
from ir_measures import * # imports all supported measures, e.g., AP, nDCG, RR, P

qrels = {
    'Q0': {"D0": 0, "D1": 1},
    "Q1": {"D0": 0, "D3": 2}
}
run = {
    'Q0': {"D0": 1.2, "D1": 1.0},
    "Q1": {"D0": 2.4, "D3": 3.6}
}

# aggregated results
ir_measures.calc_aggregate([AP, nDCG, RR, nDCG@10, P(rel=2)@10], qrels, run)
# {AP: 0.75, nDCG: 0.8154648767857288, RR: 0.75, nDCG@10: 0.8154648767857288, P(rel=2)@10: 0.05}

# by query
for m in ir_measures.iter_calc([AP, nDCG, RR, nDCG@10, P(rel=2)@10], qrels, run):
    print(m)
# Metric(query_id='Q0', measure=AP, value=0.5)
# Metric(query_id='Q0', measure=RR, value=0.5)
# Metric(query_id='Q0', measure=nDCG, value=0.6309297535714575)
# Metric(query_id='Q0', measure=nDCG@10, value=0.6309297535714575)
# Metric(query_id='Q1', measure=AP, value=1.0)
# Metric(query_id='Q1', measure=RR, value=1.0)
# Metric(query_id='Q1', measure=nDCG, value=1.0)
# Metric(query_id='Q1', measure=nDCG@10, value=1.0)
# Metric(query_id='Q0', measure=P(rel=2)@10, value=0.0)
# Metric(query_id='Q1', measure=P(rel=2)@10, value=0.1)

Qrels can be provided in the following formats:

# dict of dict
qrels = {
    'Q0': {
        "D0": 0,
        "D1": 1,
    },
    "Q1": {
        "D0": 0,
        "D3": 2
    }
}

# dataframe
import pandas as pd
qrels = pd.DataFrame([
    {'query_id': "Q0", 'doc_id': "D0", 'relevance': 0},
    {'query_id': "Q0", 'doc_id': "D1", 'relevance': 1},
    {'query_id': "Q1", 'doc_id': "D0", 'relevance': 0},
    {'query_id': "Q1", 'doc_id': "D3", 'relevance': 2},
])

# any iterable of namedtuples (e.g., list, generator, etc)
qrels = [
    ir_measures.Qrel("Q0", "D0", 0),
    ir_measures.Qrel("Q0", "D1", 1),
    ir_measures.Qrel("Q1", "D0", 0),
    ir_measures.Qrel("Q1", "D3", 2),
]

# TREC-formatted qrels file
qrels = ir_measures.read_trec_qrels('qrels.txt')

# qrels from the ir_datasets package (https://ir-datasets.com/)
import ir_datasets
qrels = ir_datasets.load('trec-robust04').qrels_iter()

Runs can be provided in the following formats:

# dict of dict
run = {
    'Q0': {
        "D0": 1.2,
        "D1": 1.0,
    },
    "Q1": {
        "D0": 2.4,
        "D3": 3.6
    }
}

# dataframe
import pandas as pd
run = pd.DataFrame([
    {'query_id': "Q0", 'doc_id': "D0", 'score': 1.2},
    {'query_id': "Q0", 'doc_id': "D1", 'score': 1.0},
    {'query_id': "Q1", 'doc_id': "D0", 'score': 2.4},
    {'query_id': "Q1", 'doc_id': "D3", 'score': 3.6},
])

# any iterable of namedtuples (e.g., list, generator, etc)
run = [
    ir_measures.ScoredDoc("Q0", "D0", 1.2),
    ir_measures.ScoredDoc("Q0", "D1", 1.0),
    ir_measures.ScoredDoc("Q1", "D0", 2.4),
    ir_measures.ScoredDoc("Q1", "D3", 3.6),
]

Command Line Interface

ir_measures also functions as a command line interface, with syntax similar to trec_eval.

Example:

ir_measures /path/to/qrels /path/to/run P@10 'P(rel=2)@5 nDCG@15 Judged@10' NumQ NumRel NumRet NumRelRet
P@10    0.4382
P(rel=2)@5  0.0827
nDCG@15 0.4357
Judged@10   0.9812
NumQ    249.0000
NumRel  17412.0000
NumRet  241339.0000
NumRet(rel=1)   10272.0000

Syntax:

ir_measures qrels run measures... [-q] [-n] [-p 4]

• qrels: a TREC-formatted QRELS file
• run: a TREC-formatted results file
• measures: one or more measure name strings. Note that in bash, () must be in single quotes. For simplicity, you can provide multiple meaures in a single quotation, which are split on whitespace.
• -q: provide results for each query individually
• -n: when used with -q, skips summary statistics
• -p: number of decimal places to report results (default: 4)

PyTerrier API

PyTerrier uses ir_measures:

from pyterrier.measures import *
pt.Experiment(
    [bm25],
    topics,
    qrels,
    measures=[P@10, P(rel=2)@5, nDCG@15]

Documentation

• Measures
• Providers

Credits

• Sean MacAvaney, University of Glasgow
• Craig Macdonald, University of Glasgow
• Charlie Clarke, University of Waterloo
• Benjamin Piwowarski, CNRS
• Harry Scells, Leipzig University

If you use this package, be sure to cite:

@inproceedings{DBLP:conf/ecir/MacAvaneyMO22a,
  author       = {Sean MacAvaney and
                  Craig Macdonald and
                  Iadh Ounis},
  title        = {Streamlining Evaluation with ir-measures},
  booktitle    = {Advances in Information Retrieval - 44th European Conference on {IR}
                  Research, {ECIR} 2022, Stavanger, Norway, April 10-14, 2022, Proceedings,
                  Part {II}},
  series       = {Lecture Notes in Computer Science},
  volume       = {13186},
  pages        = {305--310},
  publisher    = {Springer},
  year         = {2022},
  url          = {https://doi.org/10.1007/978-3-030-99739-7\_38},
  doi          = {10.1007/978-3-030-99739-7\_38}
}