Metrics¶

All metrics allow you to get both individual scores as well as mean scores:

`rankereval.metrics.Metric.score`	Individual scores for each ranking.
`rankereval.metrics.Metric.mean`	Mean score over all ranking after handling NaN values.

The code below shows how ReciprocalRank@3 scores as well as mean ReciprocalRank@3 (MRR@3) can be computed:

from rankereval import BinaryLabels, Rankings, ReciprocalRank

# define some input data
y_true = BinaryLabels.from_positive_indices([[0, 5],[1,2,3]])
y_pred = Rankings.from_ranked_indices([[3,2,1], [1,2]])

# score rankings
metric = ReciprocalRank(3)
rr_scores = metric.score(y_true, y_pred)
mrr = metric.mean(y_true, y_pred)

You can also obtain bootstrapped confidence intervals when passing conf_interval=True:

mrr = metric.mean(y_true, y_pred, conf_interval=True)
print(mrr["conf_interval"])

Note

rankereval was designed so that metrics return NaN values in ill-specified cases. If you observe a NaN value, please consult the documentation below and make sure you understand why it occurred. You can then either remove the ranking(s) from your input data, or handle them via the nan_handling option.

Base class¶

class rankereval.metrics.Metric[source]¶

mean(y_true, y_pred, nan_handling='propagate', conf_interval=False, n_bootstrap_samples=1000, confidence=0.95)[source]¶

Mean score over all ranking after handling NaN values.

Parameters:	y_true (`Labels`) – Ground truth labels, see also above. y_pred (`Rankings`) – Rankings to be evaluated. nan_handling ({'propagate', 'drop', 'zerofill'}, optional) – ‘propagate’ (default): Return NaN if any value is NaN ’drop’ : Ignore NaN values ’zerofill’ : Replace NaN values with zero conv_interval (bool, optional) – If True, then return bootstrapped confidence intervals of mean, otherwise interval is None. Defaults to False. n_bootstrap_samples (int, optional) – Number of bootstrap samples to draw. confidence (float, optional) – Indicates width of confidence interval. Default is 0.95 (95%).
Returns:	mean – Dictionary with `mean["score"]` and `mean["conf_interval"]` for the confidence interval tuple (lower CI, upper CI).
Return type:	float mean if conv_interval is false otherwise
Raises:	`TypeError` – if y_true or y_pred are of incorrect type. `ValueError` – if n_bootstrap_samples, confidence or nan_handling contain invalid values.

score(y_true, y_pred)[source]¶

Individual scores for each ranking.

Parameters:	y_true (`Labels`) – Ground truth labels. y_pred (`Rankings`) – Rankings to be evaluated.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type. `ValueError` – if n_bootstrap_samples, confidence or nan_handling contain invalid values.

Precision@k¶

class rankereval.metrics.Precision(k, truncated=False)[source]¶

Parameters:	k (int) – specifies number of top results k of each ranking to be evaluated. truncated (bool) – if true, k gets clipped at length of input ranking.
Raises:	`ValueError` – if k is not integer > 0.

score(y_true, y_pred)[source]¶

Computes Precision@k [MN] of each ranking y in y_pred as

\[\mathrm{Precision}@k(y) = \frac{\sum_{i=1}^{k'} \mathrm{rel}(y_i)}{k'},\]

where \(\mathrm{rel}(y_i)\) is the relevance label of the item at rank i in the ranking y. If truncated then \(k' = \min(k,|y|)\), i.e., k’ can never exceed the length of the input ranking; otherwise \(k' = k\) (default).

Parameters:	y_true (`BinaryLabels`) – Ground truth labels, must be binary. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

1. Ranking to be evaluated is empty, i.e., \(|y|=0\). Per definition above, \(\mathrm{Precision}@k(y) = 0\).

2. There are no relevant items in y_true: \(\mathrm{Precision}@k(y) =\) NaN. This marks invalid instances explicitly and is consistent with Recall.

Examples

>>> from rankereval import BinaryLabels, Rankings, Precision
>>> # use separate labels for each ranking
>>> y_true = BinaryLabels.from_positive_indices([[0, 5],[1,2,3]])
>>> y_pred = Rankings.from_ranked_indices([[3,2,1], [1,2]])
>>> Precision(3).score(y_true, y_pred)
array([0.        , 0.66666667])

Recall@k¶

class rankereval.metrics.Recall(k, truncated=False)[source]¶

Parameters:	k (int) – specifies number of top results k of each ranking to be evaluated. truncated (bool) – if true, number of relevant results gets clipped at k.
Raises:	`ValueError` – if k is not integer > 0.

score(y_true, y_pred)[source]¶

Computes Recall@k [MN] as the fraction of relevant results in y_true that were in the top k results of y_pred. More formally, the recall of each ranking y in y_pred with labels y_true is defined as

\[\mathrm{Recall}@k(y) = \frac{\sum_{i=1}^{k'} \mathrm{rel}(y_i)}{n_{rel}},\]

where \(\mathrm{rel}(y_i)\) is the relevance label of the item at rank i in the ranking y and \(n_{rel} = \left\| y_{true} \right\| _1\) if truncated is false (default) and \(n_{rel} = \max(\left\| y_{true} \right\| _1, k)\) otherwise.

Parameters:	y_true (`BinaryLabels`) – Ground truth labels, must be binary. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

1. Ranking to be evaluated is empty, i.e., \(|y|=0\). Per definition above, \(\mathrm{Recall}@k(y) = 0\).

2. There are no relevant items in y_true: \(\mathrm{Recall}@k(y) =\) NaN. This marks invalid instances explicitly.

Examples

>>> from rankereval import BinaryLabels, Rankings, Recall
>>> # use separate labels for each ranking
>>> y_true = BinaryLabels.from_positive_indices([[0, 5],[1]])
>>> y_pred = Rankings.from_ranked_indices([[3,2,1], [1,2]])
>>> Recall(3).score(y_true, y_pred) # doctest: +NORMALIZE_WHITESPACE
array([0., 1.])

F1@k¶

class rankereval.metrics.F1(k, truncated=False)[source]¶

score(y_true, y_pred)[source]¶

Computes F1 [MN] as harmonic mean of Precision@k and Recall@k. More formally, the F1 score of each ranking y in y_pred is defined as

\[\mathrm{F1}@k(y) = \frac{2*\big(\mathrm{Precision}@k(y) * \mathrm{Recall}@k(y)\big)}{\mathrm{Precision}@k(y) + \mathrm{Recall}@k(y)}.\]

Parameters:	y_true (`BinaryLabels`) – Ground truth labels, must be binary. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

1. Ranking to be evaluated is empty, i.e., \(|y|=0\). In this case, \(\mathrm{Recall}@k(y) = \mathrm{Precision}@k(y) = 0\) .

2. If \(\mathrm{Recall}@k(y) = \mathrm{Precision}@k(y) = 0\), we define \(\mathrm{F1}@k(y) = 0\).

There are no relevant items in y_true: \(\mathrm{F1}@k(y) =\) NaN.

Examples

>>> from rankereval import BinaryLabels, Rankings, F1
>>> # use separate labels for each ranking
>>> y_true = BinaryLabels.from_positive_indices([[0, 5],[1]])
>>> y_pred = Rankings.from_ranked_indices([[3,2,1], [1,2]])
>>> F1(3).score(y_true, y_pred)  # doctest: +NORMALIZE_WHITESPACE
array([0. , 0.5])

HitRate@k¶

class rankereval.metrics.HitRate(k, truncated=False)[source]¶

Parameters:	k (int) – specifies number of top results k of each ranking to be evaluated.
Raises:	`ValueError` – if k is not integer > 0.

score(y_true, y_pred)[source]¶

Computes HitRate@k [MN] as the whether a relevant item occurs in k results of y_pred. Differs from Recall@k in that y_true has to contain exactly one element per row. More formally, the HitRate of each ranking y in y_pred with labels y_true is defined as

\[ \begin{align}\begin{aligned}\mathrm{HitRate}@k(y) &= \sum_{i=1}^{k'} \mathrm{rel}(y_i),\\k' &= \min(k,|y|);\end{aligned}\end{align} \]

where \(\mathrm{rel}(y_i)\) is the relevance label of the item at rank i in the ranking y.

Parameters:	y_true (`BinaryLabels`) – Ground truth labels, must be binary and exactly one relevant item per row. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

1. Ranking to be evaluated is empty, i.e., \(|y|=0\). Per definition above, \(\mathrm{HitRate}@k(y) = 0\).

2. There is not exactly one relevant item in y_true: \(\mathrm{HitRate}@k(y) =\) NaN.

Examples

>>> from rankereval import BinaryLabels, Rankings, HitRate
>>> # use separate labels for each ranking
>>> y_true = BinaryLabels.from_positive_indices([[0],[1]])
>>> y_pred = Rankings.from_ranked_indices([[3,2,1], [1,2]])
>>> HitRate(3).score(y_true, y_pred) # doctest: +NORMALIZE_WHITESPACE
array([0., 1.])

ReciprocalRank@k¶

class rankereval.metrics.ReciprocalRank(k)[source]¶

Parameters:	k (int) – specifies number of top results k of each ranking to be evaluated.
Raises:	`ValueError` – if k is not integer > 0.

score(y_true, y_pred)[source]¶

Computes ReciprocalRank@k [NC] as the rank where the first relevant item occurs in the top k results of y_pred. More formally, the ReciprocalRank of each ranking y in y_pred is defined as

\[ \begin{align}\begin{aligned}\mathrm{ReciprocalRank}@k(y) &= \max_{i=1,\ldots,k'} \frac{\mathrm{rel}(y_i)}{i},\\k' &= \min(k,|y|);\end{aligned}\end{align} \]

where \(\mathrm{rel}(y_i)\) is the relevance label of the item at rank i in the ranking y.

Parameters:	y_true (`BinaryLabels`) – Ground truth labels, must be binary. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

1. Ranking to be evaluated is empty or first relevant item appears beyond rank k. Per definition above, \(\mathrm{ReciprocalRank}@k(y) = 0\).

There is no relevant item in y_true: \(\mathrm{ReciprocalRank}@k(y) =\) NaN.

Examples

>>> from rankereval import BinaryLabels, Rankings, ReciprocalRank
>>> # use separate labels for each ranking
>>> y_true = BinaryLabels.from_positive_indices([[0, 5],[1]])
>>> y_pred = Rankings.from_ranked_indices([[3,0,1], [1,2]])
>>> ReciprocalRank(3).score(y_true, y_pred) # doctest: +NORMALIZE_WHITESPACE
array([0.5, 1. ])

AveragePrecision@k¶

class rankereval.metrics.AP(k)[source]¶

Parameters:	k (int) – specifies number of top results k of each ranking to be evaluated.
Raises:	`ValueError` – if k is not integer > 0.

score(y_true, y_pred)[source]¶

Computes AveragePrecision@k [MN], an approximation to the the area under the precision-recall curve, of each ranking y in y_pred as

\[ \begin{align}\begin{aligned}\mathrm{AveragePrecision}@k(y) &= \frac{1}{Z} \sum_{i=1}^{k'} \mathrm{rel}(y_i)\cdot \mathrm{Precision}@i(y),\\k' &= \min(k,|y|),\\Z &= \min \big(k, \left\| y_{true} \right\| _1\big);\end{aligned}\end{align} \]

where \(\mathrm{rel}(y_i)\) is the relevance label of the item at rank i in the ranking y.

Note

Sometimes the denominator Z is defined with respect to only the retrieved or recommended set of items y_pred. This is not desirable as AP could be artificially inflated, e.g., by returning only one relevant item at the top and then filling up the ranking with and irrelevant items.

Parameters:	y_true (`BinaryLabels`) – Ground truth labels, must be binary. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

1. Ranking to be evaluated is empty, i.e., \(|y|=0\). Per definition above, \(\mathrm{AveragePrecision}@k(y) = 0\).

2. There are no relevant items in y_true: \(\mathrm{AveragePrecision}@k(y) =\) NaN to make it consistent with other metrics.

3. There are no relevant items in y_pred up to k: Per definition above, \(\mathrm{AveragePrecision}@k(y) = 0\).

Examples

>>> from rankereval import BinaryLabels, Rankings, AP
>>> # use separate labels for each ranking
>>> y_true = BinaryLabels.from_positive_indices([[0, 5],[1,2,3]])
>>> y_pred = Rankings.from_ranked_indices([[3,2,1], [1,2]])
>>> AP(3).score(y_true, y_pred) # doctest: +NORMALIZE_WHITESPACE
array([0. , 0.66666667])

MeanRanks¶

class rankereval.metrics.MeanRanks[source]¶

Used for evaluating permutations of y_true. Does not accept k as it scores permutations.

score(y_true, y_pred)[source]¶

Computes MeanRanks as the mean of ranks of relevant items y_pred. More formally, it is defined for each ranking y in y_pred as

\[\mathrm{MeanRanks}(y) = \frac{\sum_{i=1}^{|y|} i\cdot\mathrm{rel}(y_i)}{\sum_{i=1}^{|y|} \mathrm{rel}(y_i)},\]

where \(\mathrm{rel}(y_i)\) is the relevance label of the item at rank i in the ranking y.

Parameters:	y_true (`BinaryLabels`) – Ground truth labels, must be binary. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

There is no relevant item in y_true: \(\mathrm{MeanRanks}(y) =\) NaN.

Examples

>>> from rankereval import BinaryLabels, Rankings, MeanRanks
>>> # use separate labels for each ranking
>>> y_true = BinaryLabels.from_positive_indices([[0, 5],[1]])
>>> y_pred = Rankings.from_ranked_indices([[3,0,5], [1,2]])
>>> MeanRanks().score(y_true, y_pred) # doctest: +NORMALIZE_WHITESPACE
array([2.5, 1. ])

FirstRelevantRank¶

class rankereval.metrics.FirstRelevantRank[source]¶

Used for evaluating permutations of y_true. Does not accept k as it scores permutations.

score(y_true, y_pred)[source]¶

Computes FirstRelevantRank as the mean of ranks of relevant items y_pred. More formally, it is defined for each ranking y in y_pred as

\[\mathrm{FirstRelevantRank}(y) = \min \{i \mid \mathrm{rel}(y_i) = 1\},\]

where \(\mathrm{rel}(y_i)\) is the relevance label of the item at rank i in the ranking y.

Parameters:	y_true (`BinaryLabels`) – Ground truth labels, must be binary. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

There is no relevant item in y_true: \(\mathrm{FirstRelevantRank}(y) =\) NaN.

Examples

>>> from rankereval import BinaryLabels, Rankings, FirstRelevantRank
>>> # use separate labels for each ranking
>>> y_true = BinaryLabels.from_positive_indices([[0, 5],[1]])
>>> y_pred = Rankings.from_ranked_indices([[3,0,5], [1,2]])
>>> FirstRelevantRank().score(y_true, y_pred) # doctest: +NORMALIZE_WHITESPACE
array([2., 1. ])

DCG@k¶

class rankereval.metrics.DCG(k=None, relevance_scaling='identity', log_base='e')[source]¶

Parameters:

k (int) – specifies number of top results k of each ranking to be evaluated.
relevance_scaling (str, ['binary' (default), 'power']) –
Determines are relevance labels are transformed:

’identity’: (default)

\(f(\mathrm{rel}(y_i)) = \mathrm{rel}(y_i)\)

’power’:

\(f(\mathrm{rel}(y_i)) = 2^{\mathrm{rel}(y_i)} - 1\)
log_base (str, ['e' (default), '2']) –
Determines what log base is used in denominator. The smaller this value, the heavier emphasis on top-ranked documents.

’e’ (default):

Natural logarithm \(\ln\)

’2’:

\(\log_2\)

Notes

The original definition of (n)DCG [KJ] uses ‘identity’ for relevance_scaling, but leaves the choice of log_base open.

Raises:	`ValueError` – if k is not integer > 0 or relevance_scaling or log_base are invalid.

score(y_true, y_pred)[source]¶

Computes Discounted Cumulative Gain at k (DCG@k) [KJ] as a weighted sum of relevance labels of top k results of y_pred. More formally, the recall of each ranking y in y_pred with labels y_true is defined as

\[ \begin{align}\begin{aligned}\mathrm{DCG}@k(y) &= \sum_{i=1}^{k'} \frac{f(\mathrm{rel}(y_i))}{\log_b(i + 1)},\\k' &= \min(k,|y|);\end{aligned}\end{align} \]

where \(\mathrm{rel}(y_i)\) is the relevance label of the item at rank i in the ranking y. f is the relevance_scaling function and b the log_base parameters defined earlier.

Parameters:	y_true (`Labels`) – Ground truth labels, binary or numeric. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

1. Ranking to be evaluated is empty, i.e., \(|y|=0\). Per definition above, \(\mathrm{DCG}@k(y) = 0\).

2. There are no items with relevance > 0 in y_true: \(\mathrm{DCG}@k(y) =\) NaN to make it consistent with other metrics.

Examples

>>> from rankereval import NumericLabels, Rankings, DCG
>>> # use separate labels for each ranking
>>> y_true = NumericLabels.from_matrix([[1, 2, 3], [4, 5]])
>>> y_pred = Rankings.from_ranked_indices([[0,2,1], [1,0]])
>>> DCG(3).score(y_true, y_pred) # doctest: +NORMALIZE_WHITESPACE
array([ 5.61610776, 10.85443211])

NDCG@k¶

class rankereval.metrics.NDCG(k=None, relevance_scaling='identity', log_base='e')[source]¶

For a description of the arguments, see DCG.

score(y_true, y_pred)[source]¶

Computes the normalized Discounted Cumulative Gain at k (nDCG@k) [KJ] as a weighted sum of relevance labels of top k results of y_pred, normalized to the range [0, 1]. More formally, the nDCG of each ranking y in y_pred with labels y_true is defined as

\[\begin{split}\mathrm{nDCG}@k(y) = \begin{cases} \frac{\mathrm{DCG}@k(y)} {\mathrm{IDCG}@k(y)} &\mbox{if } \mathrm{IDCG}@k(y) > 0 \\ 0 & \mbox{otherwise } \end{cases},\end{split}\]

where \(\mathrm{IDCG}@k(y)\) is the maximum DCG@k value that can be achieved on all relevance labels (i.e., DCG@k of the sorted relevance labels).

Note

Sometimes IDCG is defined with respect to only the retrieved or recommended set of items y. This is not desirable as nDCG could be artificially inflated, e.g., by returning only one relevant item at the top and then filling up the ranking with and irrelevant items.

Parameters:	y_true (`Labels`) – Ground truth labels, binary or numeric. y_pred (`Rankings`) – Rankings to be evaluated. If y_true only contains one row, the labels in this row will be used for every ranking in y_pred. Otherwise, each row i in y_pred uses label row i in y_true.
Returns:	computed_metric – Computed metric for each ranking.
Return type:	ndarray, shape (n_rankings, )
Raises:	`TypeError` – if y_true or y_pred are of incorrect type.

Notes

Edge cases:

y is empty, i.e., \(|y|=0\). Per definition above, \(\mathrm{DCG}@k(y) = 0\).

2. There are no items with relevance > 0 in y_true: \(\mathrm{DCG}@k(y) =\) NaN to make it consistent with other metrics.

There are no items with relevance > 0 in y up to k: \(\mathrm{nDCG}@k(y) = 0\).

Examples

>>> from rankereval import NumericLabels, Rankings, NDCG
>>> # use separate labels for each ranking
>>> y_true = NumericLabels.from_matrix([[1, 2, 3], [4, 5]])
>>> y_pred = Rankings.from_ranked_indices([[0,2,1], [1,0]])
>>> NDCG(3).score(y_true, y_pred) # doctest: +NORMALIZE_WHITESPACE
array([0.81749351, 1.        ])

References¶

[MN]	(1, 2, 3, 4, 5) C. D. Manning, P. Raghavan and H. Schütze: “Introduction to Information Retrieval”. Cambridge University Press (2008).

[NC]	Craswell: Mean Reciprocal Rank in: “Encyclopedia of Database Systems”. Springer (2009).

[KJ]	(1, 2, 3) Kalervo Järvelin, Jaana Kekäläinen: “Cumulated gain-based evaluation of IR techniques”. ACM Transactions on Information Systems 20(4), 422–446 (2002).