recognite.eval package
- recognite.eval.accuracy(scores: Tensor, gallery_labels: Tensor, query_labels: Tensor) Tensor[source]
Computes the top-1 accuracy.
This is computed as the percentage of queries where the most similar gallery item has the same label as the query.
- Parameters:
scores – The scores for each query (rows) and each gallery item (columns).
query_labels – The true label of each query (rows of
scores).gallery_labels – The labels of the items in the gallery (columns of
scores).
- Returns:
The top-1 accuracy.
- recognite.eval.hard_pos_neg_scores(scores: Tensor, gallery_labels: Tensor, query_labels: Tensor) Dict[str, Tensor][source]
Computes the similarity scores between each query and the hardest negative and hardest positive in the gallery.
- Parameters:
scores – The scores for each query (rows) and each gallery item (columns).
query_labels – The true label of each query (rows of
scores).gallery_labels – The labels of the items in the gallery (columns of
scores).
- Returns:
A dictionary with the following items
'HardPosScores': The score of the hardest positive of each query.'HardNegScores': The score of the hardest negative of each query.
- recognite.eval.knn(scores: Tensor, gallery_labels: Tensor, k: int) Tensor[source]
Classifies the queries with k-Nearest Neighbours.
For each query, we do a majority voting among the labels of the k gallery items with the highest similarity score.
- Parameters:
scores – The scores for each query (rows) and each gallery item (columns).
gallery_labels – The labels of the items in the gallery (columns of
scores).k – The number of nearest neighbours to consider.
- Returns:
The result of the k-NN classification for each query.
- recognite.eval.pr_metrics(scores: Tensor, gallery_labels: Tensor, query_labels: Tensor) Dict[str, List[Tensor] | Tensor][source]
Computes the precision-recall curves and related metrics.
- Parameters:
scores – The scores for each query (rows) and each gallery item (columns).
gallery_labels – The labels of the items in the gallery (columns of
scores).query_labels – The true label of each query (rows of
scores).
- Returns:
A dictionary containing the metrics. The dict contains the following keys:
'Precisions': The precision values of the PR-curve of each query.'Recalls': The recall values of the PR-curve of each query.'Thresholds': The thresholds that correspond to each point on the PR curve.'AP': The average precision of each query.'mAP': The mean of the average precisions of all queries.'P@maxF1': The precision at the point on the PR curve with the maximum F1 score for each query.'R@maxF1': The recall at the point on the PR curve with the maximum F1 score for each query.'T@maxF1': The threshold at the point on the PR curve with the maximum F1 score for each query.'maxF1': The maximum F1 score for each query.
- recognite.eval.score_matrix(model: Module, dl_gal: DataLoader, dl_quer: DataLoader, metric='inner', device: device | None = None, agg_gal_fn: Callable | None = None, get_embeddings_fn: Callable | None = None) Tuple[Tensor, Tensor, Tensor][source]
Computes the score matrix for a given model, queries and gallery.
The rows in the score matrix correspond to the queries, the columns correspond to the gallery.
We first iterate over the gallery batches to compute the gallery embeddings and compose a gallery. Then we compute the embedding of each query image. Finally, we compute the score matrix as the dot product between each pair of gallery and query embeddings.
The model should return a batch of embeddings when calling it with a batch of (transformed) images. You can also pass in a custom function
get_embeddings_fn(model, imgs)that takes the model and the batch of images as input and returns the embeddings.Optionally, the embeddings of the gallery can be aggregated (e.g. averaged per class) with a custom
agg_gal_fn(gal_embeddings, gal_labels)which takes the gallery embeddings and labels and returns their aggregated versions.- Parameters:
model – The model used for embedding extraction.
dl_gal – The data loader for the gallery data. It should yield a tuple
(imgs, labels)containing the batch of images and labels when iterating over it.dl_quer – The data loader for the query data. It should yield a tuple
(imgs, labels)containing the batch of images and labels when iterating over it.metric – The metric to use for computing the scores. Can be
'inner'` (inner prosuct), ``'cosine'(cosine similarity),'euclid'(Euclidean distance),'sq_euclid'(squared Euclidean distance). Distances will be negated so that higher score means higher similarity.device – The device on which to perform the computations. If
None, use CUDA if it is available.agg_gal_fn – A function that aggregates the gallery embeddings and labels. It should take two arguments: the gallery embeddings and the gallery labels.
get_embeddings_fn – A custom function to use for embedding extraction. It should take two arguments: the model and the image batch.
- Returns:
The score matrix, the labels of the gallery items (columns) and the labels of the queries (rows).
- recognite.eval.sort_scores(scores: Tensor, gallery_labels: Tensor) Tuple[Tensor, Tensor][source]
Sorts the scores and labels according to descending score.
- Parameters:
scores – The scores for each query (rows) and each gallery item (columns).
gallery_labels – The labels of the items in the gallery (columns of
scores).
- Returns:
A tuple with the sorted scores and labels.
- recognite.eval.top_k(scores: Tensor, gallery_labels: Tensor, k: int) Tuple[Tensor, Tensor][source]
Returns the top k scores and corresponding labels.
- Parameters:
scores – The scores for each query (rows) and each gallery item (columns).
gallery_labels – The labels of the items in the gallery (columns of
scores).k – The number of nearest neighbours to consider.
- Returns:
A tuple with the scores and labels of the k highest similarities.
- recognite.eval.top_k_accuracy(scores: Tensor, gallery_labels: Tensor, query_labels: Tensor, k: int)[source]
Computes the top-k accuracy.
This is computed as the percentage of queries where any of the k most similar gallery items has the same label as the query.
- Parameters:
scores – The scores for each query (rows) and each gallery item (columns).
query_labels – The true label of each query (rows of
scores).gallery_labels – The labels of the items in the gallery (columns of
scores).
- Returns:
The top-k accuracy.