Data Formats

rankereval distinguishes explicitly between ground truth labels and predictions. This avoids accidental swapping of arguments and allows for additional type and consistency checking.

rankereval.data.BinaryLabels	Represents binary ground truth data (e.g., 1 indicating relevance).
rankereval.data.NumericLabels	Represents numeric ground truth data (e.g., relevance labels from 1-5).
rankereval.data.Rankings	Represents (predicted) rankings to be evaluated.

Ground truth labels

class rankereval.data.BinaryLabels

Represents binary ground truth data (e.g., 1 indicating relevance).

classmethod from_matrix(labels)

Construct a binary labels instance from dense or sparse matrix where each item’s label is specified.

Parameters:labels (1D or 2D array, one row per context (e.g., user or query)) – Contains binary labels for each item. Labels must be in {0, 1}. Raises:ValueError – if labels is of invalid shape, type or non-binary.

Examples

>>> BinaryLabels.from_matrix([[0, 1, 1], [0, 0, 1]]) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
<rankereval.data.BinaryLabels...>

classmethod from_positive_indices(indices)

Construct a binary labels instance from sparse data where only positive items are specified.

Parameters:indices (array_like, one row per context (e.g., user or query)) – Specifies positive indices for each sample. Must be 1D or 2D, but row lengths can differ. Raises:ValueError – if indices is of invalid shape, type or contains duplicate, negative or non-integer indices.

Examples

>>> BinaryLabels.from_positive_indices([[1,2], [2]]) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
<rankereval.data.BinaryLabels...>

class rankereval.data.NumericLabels

Represents numeric ground truth data (e.g., relevance labels from 1-5).

classmethod from_matrix(labels)

Construct a binary labels instance from dense or sparse matrix where each item’s label is specified.

Parameters:labels (1D or 2D array, one row per context (e.g., user or query)) – Contains binary labels for each item. Labels must be in {0, 1}. Raises:ValueError – if labels is of invalid shape, type or non-binary.

Examples

>>> BinaryLabels.from_matrix([[0, 1, 1], [0, 0, 1]]) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
<rankereval.data.BinaryLabels...>

classmethod from_positive_indices(indices)

Construct a binary labels instance from sparse data where only positive items are specified.

Parameters:indices (array_like, one row per context (e.g., user or query)) – Specifies positive indices for each sample. Must be 1D or 2D, but row lengths can differ. Raises:ValueError – if indices is of invalid shape, type or contains duplicate, negative or non-integer indices.

Examples

>>> BinaryLabels.from_positive_indices([[1,2], [2]]) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
<rankereval.data.BinaryLabels...>

Predicted rankings

class rankereval.data.Rankings(indices, valid_items=None, invalid_items=None, warn_empty=True)

Represents (predicted) rankings to be evaluated.

classmethod from_ranked_indices(indices, valid_items=None, invalid_items=None)

Construct a rankings instance from data where item indices are specified in ranked order.

Parameters:

• indices (array_like, one row per ranking) – Indices of items after ranking. Must be 1D or 2D, but row lengths can differ.
• valid_items (array_like, one row per ranking) – Indices of valid items (e.g., candidate set). Invalid items will be discarded from ranking.

Raises:

ValueError – if indices or valid_items of invalid shape or type.

Examples

>>> Rankings.from_ranked_indices([[5, 2], [4, 3, 1]]) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
<rankereval.data.Rankings...>

classmethod from_scores(raw_scores, valid_items=None, invalid_items=None, warn_empty=True)

Construct a rankings instance from raw scores where each item’s score is specified. Items will be ranked in descending order (higher scores meaning better).

Parameters:

• raw_scores (array_like, one row per ranking) – Contains raw scores for each item. Must be 1D or 2D, but row lengths can differ.
• valid_items (array_like, one row per ranking) – Indices of valid items (e.g., candidate set). Invalid items will be discarded from ranking.

Raises:

ValueError – if raw_scores or valid_items of invalid shape or type.

Warns:

InvalidValuesWarning – if raw_scores contains non-finite values.

Examples

>>> Rankings.from_scores([[0.1, 0.5, 0.2], [0.4, 0.2, 0.5]]) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
<rankereval.data.Rankings...>