ā¹ļø Skipped - page is already crawled
| Filter | Status | Condition | Details |
|---|---|---|---|
| HTTP status | PASS | download_http_code = 200 | HTTP 200 |
| Age cutoff | PASS | download_stamp > now() - 6 MONTH | 0 months ago |
| History drop | PASS | isNull(history_drop_reason) | No drop reason |
| Spam/ban | PASS | fh_dont_index != 1 AND ml_spam_score = 0 | ml_spam_score=0 |
| Canonical | PASS | meta_canonical IS NULL OR = '' OR = src_unparsed | Not set |
| Property | Value |
|---|---|
| URL | https://scikit-learn.org/stable/modules/model_evaluation.html |
| Last Crawled | 2026-04-17 16:50:03 (55 minutes ago) |
| First Indexed | 2018-08-27 21:23:53 (7 years ago) |
| HTTP Status Code | 200 |
| Meta Title | 3.4. Metrics and scoring: quantifying the quality of predictions ā scikit-learn 1.8.0 documentation |
| Meta Description | Which scoring function should I use?: Before we take a closer look into the details of the many scores and evaluation metrics, we want to give some guidance, inspired by statistical decision theory... |
| Meta Canonical | null |
| Boilerpipe Text | 3.4.1.
Which scoring function should I use?
#
Before we take a closer look into the details of the many scores and
evaluation metrics
, we want to give some guidance, inspired by statistical
decision theory, on the choice of
scoring functions
for
supervised learning
,
see
[Gneiting2009]
:
Which scoring function should I use?
Which scoring function is a good one for my task?
In a nutshell, if the scoring function is given, e.g. in a kaggle competition
or in a business context, use that one.
If you are free to choose, it starts by considering the ultimate goal and application
of the prediction. It is useful to distinguish two steps:
Predicting
Decision making
Predicting:
Usually, the response variable
Y
is a random variable, in the sense that there
is
no deterministic
function
Y
=
g
(
X
)
of the features
X
.
Instead, there is a probability distribution
F
of
Y
.
One can aim to predict the whole distribution, known as
probabilistic prediction
,
orāmore the focus of scikit-learnāissue a
point prediction
(or point forecast)
by choosing a property or functional of that distribution
F
.
Typical examples are the mean (expected value), the median or a quantile of the
response variable
Y
(conditionally on
X
).
Once that is settled, use a
strictly consistent
scoring function for that
(target) functional, see
[Gneiting2009]
.
This means using a scoring function that is aligned with
measuring the distance
between predictions
y_pred
and the true target functional using observations of
Y
, i.e.
y_true
.
For classification
strictly proper scoring rules
, see
Wikipedia entry for Scoring rule
and
[Gneiting2007]
, coincide with strictly consistent scoring functions.
The table further below provides examples.
One could say that consistent scoring functions act as
truth serum
in that
they guarantee
āthat truth telling [ā¦] is an optimal strategy in
expectationā
[Gneiting2014]
.
Once a strictly consistent scoring function is chosen, it is best used for both: as
loss function for model training and as metric/score in model evaluation and model
comparison.
Note that for regressors, the prediction is done with
predict
while for
classifiers it is usually
predict_proba
.
Decision Making:
The most common decisions are done on binary classification tasks, where the result of
predict_proba
is turned into a single outcome, e.g., from the predicted
probability of rain a decision is made on how to act (whether to take mitigating
measures like an umbrella or not).
For classifiers, this is what
predict
returns.
See also
Tuning the decision threshold for class prediction
.
There are many scoring functions which measure different aspects of such a
decision, most of them are covered with or derived from the
metrics.confusion_matrix
.
List of strictly consistent scoring functions:
Here, we list some of the most relevant statistical functionals and corresponding
strictly consistent scoring functions for tasks in practice. Note that the list is not
complete and that there are more of them.
For further criteria on how to select a specific one, see
[Fissler2022]
.
functional
scoring or loss function
response
y
prediction
Classification
mean
Brier score
1
multi-class
predict_proba
mean
log loss
multi-class
predict_proba
mode
zero-one loss
2
multi-class
predict
, categorical
Regression
mean
squared error
3
all reals
predict
, all reals
mean
Poisson deviance
non-negative
predict
, strictly positive
mean
Gamma deviance
strictly positive
predict
, strictly positive
mean
Tweedie deviance
depends on
power
predict
, depends on
power
median
absolute error
all reals
predict
, all reals
quantile
pinball loss
all reals
predict
, all reals
mode
no consistent one exists
reals
1
The Brier score is just a different name for the squared error in case of
classification with one-hot encoded targets.
2
The zero-one loss is only consistent but not strictly consistent for the mode.
The zero-one loss is equivalent to one minus the accuracy score, meaning it gives
different score values but the same ranking.
3
RĀ² gives the same ranking as squared error.
Fictitious Example:
Letās make the above arguments more tangible. Consider a setting in network reliability
engineering, such as maintaining stable internet or Wi-Fi connections.
As provider of the network, you have access to the dataset of log entries of network
connections containing network load over time and many interesting features.
Your goal is to improve the reliability of the connections.
In fact, you promise your customers that on at least 99% of all days there are no
connection discontinuities larger than 1 minute.
Therefore, you are interested in a prediction of the 99% quantile (of longest
connection interruption duration per day) in order to know in advance when to add
more bandwidth and thereby satisfy your customers. So the
target functional
is the
99% quantile. From the table above, you choose the pinball loss as scoring function
(fair enough, not much choice given), for model training (e.g.
HistGradientBoostingRegressor(loss="quantile",
quantile=0.99)
) as well as model
evaluation (
mean_pinball_loss(...,
alpha=0.99)
- we apologize for the different
argument names,
quantile
and
alpha
) be it in grid search for finding
hyperparameters or in comparing to other models like
QuantileRegressor(quantile=0.99)
.
References
[
Gneiting2014
]
T. Gneiting and M. Katzfuss.
Probabilistic Forecasting
. In: Annual Review of Statistics and Its Application 1.1 (2014), pp. 125ā151.
3.4.2.
Scoring API overview
#
There are 3 different APIs for evaluating the quality of a modelās
predictions:
Estimator score method
: Estimators have a
score
method providing a
default evaluation criterion for the problem they are designed to solve.
Most commonly this is
accuracy
for classifiers and the
coefficient of determination
(
R
2
) for regressors.
Details for each estimator can be found in its documentation.
Scoring parameter
: Model-evaluation tools that use
cross-validation
(such as
model_selection.GridSearchCV
,
model_selection.validation_curve
and
linear_model.LogisticRegressionCV
) rely on an internal
scoring
strategy.
This can be specified using the
scoring
parameter of that tool and is discussed
in the section
The scoring parameter: defining model evaluation rules
.
Metric functions
: The
sklearn.metrics
module implements functions
assessing prediction error for specific purposes. These metrics are detailed
in sections on
Classification metrics
,
Multilabel ranking metrics
,
Regression metrics
and
Clustering metrics
.
Finally,
Dummy estimators
are useful to get a baseline
value of those metrics for random predictions.
3.4.3.
The
scoring
parameter: defining model evaluation rules
#
Model selection and evaluation tools that internally use
cross-validation
(such as
model_selection.GridSearchCV
,
model_selection.validation_curve
and
linear_model.LogisticRegressionCV
) take a
scoring
parameter that
controls what metric they apply to the estimators evaluated.
They can be specified in several ways:
None
: the estimatorās default evaluation criterion (i.e., the metric used in the
estimatorās
score
method) is used.
String name
: common metrics can be passed via a string
name.
Callable
: more complex metrics can be passed via a custom
metric callable (e.g., function).
Some tools do also accept multiple metric evaluation. See
Using multiple metric evaluation
for details.
3.4.3.1.
String name scorers
#
For the most common use cases, you can designate a scorer object with the
scoring
parameter via a string name; the table below shows all possible values.
All scorer objects follow the convention that
higher return values are better
than lower return values
. Thus metrics which measure the distance between
the model and the data, like
metrics.mean_squared_error
, are
available as āneg_mean_squared_errorā which return the negated value
of the metric.
Scoring string name
Function
Comment
Classification
āaccuracyā
metrics.accuracy_score
ābalanced_accuracyā
metrics.balanced_accuracy_score
ātop_k_accuracyā
metrics.top_k_accuracy_score
āaverage_precisionā
metrics.average_precision_score
āneg_brier_scoreā
metrics.brier_score_loss
requires
predict_proba
support
āf1ā
metrics.f1_score
for binary targets
āf1_microā
metrics.f1_score
micro-averaged
āf1_macroā
metrics.f1_score
macro-averaged
āf1_weightedā
metrics.f1_score
weighted average
āf1_samplesā
metrics.f1_score
by multilabel sample
āneg_log_lossā
metrics.log_loss
requires
predict_proba
support
āprecisionā etc.
metrics.precision_score
suffixes apply as with āf1ā
ārecallā etc.
metrics.recall_score
suffixes apply as with āf1ā
ājaccardā etc.
metrics.jaccard_score
suffixes apply as with āf1ā
āroc_aucā
metrics.roc_auc_score
āroc_auc_ovrā
metrics.roc_auc_score
āroc_auc_ovoā
metrics.roc_auc_score
āroc_auc_ovr_weightedā
metrics.roc_auc_score
āroc_auc_ovo_weightedā
metrics.roc_auc_score
ād2_log_loss_scoreā
metrics.d2_log_loss_score
requires
predict_proba
support
ād2_brier_scoreā
metrics.d2_brier_score
requires
predict_proba
support
Clustering
āadjusted_mutual_info_scoreā
metrics.adjusted_mutual_info_score
āadjusted_rand_scoreā
metrics.adjusted_rand_score
ācompleteness_scoreā
metrics.completeness_score
āfowlkes_mallows_scoreā
metrics.fowlkes_mallows_score
āhomogeneity_scoreā
metrics.homogeneity_score
āmutual_info_scoreā
metrics.mutual_info_score
ānormalized_mutual_info_scoreā
metrics.normalized_mutual_info_score
ārand_scoreā
metrics.rand_score
āv_measure_scoreā
metrics.v_measure_score
Regression
āexplained_varianceā
metrics.explained_variance_score
āneg_max_errorā
metrics.max_error
āneg_mean_absolute_errorā
metrics.mean_absolute_error
āneg_mean_squared_errorā
metrics.mean_squared_error
āneg_root_mean_squared_errorā
metrics.root_mean_squared_error
āneg_mean_squared_log_errorā
metrics.mean_squared_log_error
āneg_root_mean_squared_log_errorā
metrics.root_mean_squared_log_error
āneg_median_absolute_errorā
metrics.median_absolute_error
ār2ā
metrics.r2_score
āneg_mean_poisson_devianceā
metrics.mean_poisson_deviance
āneg_mean_gamma_devianceā
metrics.mean_gamma_deviance
āneg_mean_absolute_percentage_errorā
metrics.mean_absolute_percentage_error
ād2_absolute_error_scoreā
metrics.d2_absolute_error_score
Usage examples:
>>>
from
sklearn
import
svm
,
datasets
>>>
from
sklearn.model_selection
import
cross_val_score
>>>
X
,
y
=
datasets
.
load_iris
(
return_X_y
=
True
)
>>>
clf
=
svm
.
SVC
(
random_state
=
0
)
>>>
cross_val_score
(
clf
,
X
,
y
,
cv
=
5
,
scoring
=
'recall_macro'
)
array([0.96, 0.96, 0.96, 0.93, 1. ])
Note
If a wrong scoring name is passed, an
InvalidParameterError
is raised.
You can retrieve the names of all available scorers by calling
get_scorer_names
.
3.4.3.2.
Callable scorers
#
For more complex use cases and more flexibility, you can pass a callable to
the
scoring
parameter. This can be done by:
Adapting predefined metrics via make_scorer
Creating a custom scorer object
(most flexible)
3.4.3.2.1.
Adapting predefined metrics via
make_scorer
#
The following metric functions are not implemented as named scorers,
sometimes because they require additional parameters, such as
fbeta_score
. They cannot be passed to the
scoring
parameters; instead their callable needs to be passed to
make_scorer
together with the value of the user-settable
parameters.
Function
Parameter
Example usage
Classification
metrics.fbeta_score
beta
make_scorer(fbeta_score,
beta=2)
Regression
metrics.mean_tweedie_deviance
power
make_scorer(mean_tweedie_deviance,
power=1.5)
metrics.mean_pinball_loss
alpha
make_scorer(mean_pinball_loss,
alpha=0.95)
metrics.d2_tweedie_score
power
make_scorer(d2_tweedie_score,
power=1.5)
metrics.d2_pinball_score
alpha
make_scorer(d2_pinball_score,
alpha=0.95)
One typical use case is to wrap an existing metric function from the library
with non-default values for its parameters, such as the
beta
parameter for
the
fbeta_score
function:
>>>
from
sklearn.metrics
import
fbeta_score
,
make_scorer
>>>
ftwo_scorer
=
make_scorer
(
fbeta_score
,
beta
=
2
)
>>>
from
sklearn.model_selection
import
GridSearchCV
>>>
from
sklearn.svm
import
LinearSVC
>>>
grid
=
GridSearchCV
(
LinearSVC
(),
param_grid
=
{
'C'
:
[
1
,
10
]},
...
scoring
=
ftwo_scorer
,
cv
=
5
)
The module
sklearn.metrics
also exposes a set of simple functions
measuring a prediction error given ground truth and prediction:
functions ending with
_score
return a value to
maximize, the higher the better.
functions ending with
_error
,
_loss
, or
_deviance
return a
value to minimize, the lower the better. When converting
into a scorer object using
make_scorer
, set
the
greater_is_better
parameter to
False
(
True
by default; see the
parameter description below).
3.4.3.2.2.
Creating a custom scorer object
#
You can create your own custom scorer object using
make_scorer
.
You can build a completely custom scorer object
from a simple python function using
make_scorer
, which can
take several parameters:
the python function you want to use (
my_custom_loss_func
in the example below)
whether the python function returns a score (
greater_is_better=True
,
the default) or a loss (
greater_is_better=False
). If a loss, the output
of the python function is negated by the scorer object, conforming to
the cross validation convention that scorers return higher values for better models.
for classification metrics only: whether the python function you provided requires
continuous decision certainties. If the scoring function only accepts probability
estimates (e.g.
metrics.log_loss
), then one needs to set the parameter
response_method="predict_proba"
. Some scoring
functions do not necessarily require probability estimates but rather non-thresholded
decision values (e.g.
metrics.roc_auc_score
). In this case, one can provide a
list (e.g.,
response_method=["decision_function",
"predict_proba"]
),
and scorer will use the first available method, in the order given in the list,
to compute the scores.
any additional parameters of the scoring function, such as
beta
or
labels
.
Here is an example of building custom scorers, and of using the
greater_is_better
parameter:
>>>
import
numpy
as
np
>>>
def
my_custom_loss_func
(
y_true
,
y_pred
):
...
diff
=
np
.
abs
(
y_true
-
y_pred
)
.
max
()
...
return
float
(
np
.
log1p
(
diff
))
...
>>>
# score will negate the return value of my_custom_loss_func,
>>>
# which will be np.log(2), 0.693, given the values for X
>>>
# and y defined below.
>>>
score
=
make_scorer
(
my_custom_loss_func
,
greater_is_better
=
False
)
>>>
X
=
[[
1
],
[
1
]]
>>>
y
=
[
0
,
1
]
>>>
from
sklearn.dummy
import
DummyClassifier
>>>
clf
=
DummyClassifier
(
strategy
=
'most_frequent'
,
random_state
=
0
)
>>>
clf
=
clf
.
fit
(
X
,
y
)
>>>
my_custom_loss_func
(
y
,
clf
.
predict
(
X
))
0.69
>>>
score
(
clf
,
X
,
y
)
-0.69
While defining the custom scoring function alongside the calling function
should work out of the box with the default joblib backend (loky),
importing it from another module will be a more robust approach and work
independently of the joblib backend.
For example, to use
n_jobs
greater than 1 in the example below,
custom_scoring_function
function is saved in a user-created module
(
custom_scorer_module.py
) and imported:
>>>
from
custom_scorer_module
import
custom_scoring_function
>>>
cross_val_score
(
model
,
...
X_train
,
...
y_train
,
...
scoring
=
make_scorer
(
custom_scoring_function
,
greater_is_better
=
False
),
...
cv
=
5
,
...
n_jobs
=-
1
)
3.4.3.3.
Using multiple metric evaluation
#
Scikit-learn also permits evaluation of multiple metrics in
GridSearchCV
,
RandomizedSearchCV
and
cross_validate
.
There are three ways to specify multiple scoring metrics for the
scoring
parameter:
As an iterable of string metrics:
>>>
scoring
=
[
'accuracy'
,
'precision'
]
As a
dict
mapping the scorer name to the scoring function:
>>>
from
sklearn.metrics
import
accuracy_score
>>>
from
sklearn.metrics
import
make_scorer
>>>
scoring
=
{
'accuracy'
:
make_scorer
(
accuracy_score
),
...
'prec'
:
'precision'
}
Note that the dict values can either be scorer functions or one of the
predefined metric strings.
As a callable that returns a dictionary of scores:
>>>
from
sklearn.model_selection
import
cross_validate
>>>
from
sklearn.metrics
import
confusion_matrix
>>>
# A sample toy binary classification dataset
>>>
X
,
y
=
datasets
.
make_classification
(
n_classes
=
2
,
random_state
=
0
)
>>>
svm
=
LinearSVC
(
random_state
=
0
)
>>>
def
confusion_matrix_scorer
(
clf
,
X
,
y
):
...
y_pred
=
clf
.
predict
(
X
)
...
cm
=
confusion_matrix
(
y
,
y_pred
)
...
return
{
'tn'
:
cm
[
0
,
0
],
'fp'
:
cm
[
0
,
1
],
...
'fn'
:
cm
[
1
,
0
],
'tp'
:
cm
[
1
,
1
]}
>>>
cv_results
=
cross_validate
(
svm
,
X
,
y
,
cv
=
5
,
...
scoring
=
confusion_matrix_scorer
)
>>>
# Getting the test set true positive scores
>>>
print
(
cv_results
[
'test_tp'
])
[10 9 8 7 8]
>>>
# Getting the test set false negative scores
>>>
print
(
cv_results
[
'test_fn'
])
[0 1 2 3 2]
3.4.4.
Classification metrics
#
The
sklearn.metrics
module implements several loss, score, and utility
functions to measure classification performance.
Some metrics might require probability estimates of the positive class,
confidence values, or binary decisions values.
Most implementations allow each sample to provide a weighted contribution
to the overall score, through the
sample_weight
parameter.
Some of these are restricted to the binary classification case:
Others also work in the multiclass case:
balanced_accuracy_score
(y_true,Ā y_pred,Ā *[,Ā ...])
Compute the balanced accuracy.
cohen_kappa_score
(y1,Ā y2,Ā *[,Ā labels,Ā ...])
Compute Cohen's kappa: a statistic that measures inter-annotator agreement.
confusion_matrix
(y_true,Ā y_pred,Ā *[,Ā ...])
Compute confusion matrix to evaluate the accuracy of a classification.
hinge_loss
(y_true,Ā pred_decision,Ā *[,Ā ...])
Average hinge loss (non-regularized).
matthews_corrcoef
(y_true,Ā y_pred,Ā *[,Ā ...])
Compute the Matthews correlation coefficient (MCC).
roc_auc_score
(y_true,Ā y_score,Ā *[,Ā average,Ā ...])
Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.
top_k_accuracy_score
(y_true,Ā y_score,Ā *[,Ā ...])
Top-k Accuracy classification score.
Some also work in the multilabel case:
accuracy_score
(y_true,Ā y_pred,Ā *[,Ā ...])
Accuracy classification score.
classification_report
(y_true,Ā y_pred,Ā *[,Ā ...])
Build a text report showing the main classification metrics.
f1_score
(y_true,Ā y_pred,Ā *[,Ā labels,Ā ...])
Compute the F1 score, also known as balanced F-score or F-measure.
fbeta_score
(y_true,Ā y_pred,Ā *,Ā beta[,Ā ...])
Compute the F-beta score.
hamming_loss
(y_true,Ā y_pred,Ā *[,Ā sample_weight])
Compute the average Hamming loss.
jaccard_score
(y_true,Ā y_pred,Ā *[,Ā labels,Ā ...])
Jaccard similarity coefficient score.
log_loss
(y_true,Ā y_pred,Ā *[,Ā normalize,Ā ...])
Log loss, aka logistic loss or cross-entropy loss.
multilabel_confusion_matrix
(y_true,Ā y_pred,Ā *)
Compute a confusion matrix for each class or sample.
precision_recall_fscore_support
(y_true,Ā ...)
Compute precision, recall, F-measure and support for each class.
precision_score
(y_true,Ā y_pred,Ā *[,Ā labels,Ā ...])
Compute the precision.
recall_score
(y_true,Ā y_pred,Ā *[,Ā labels,Ā ...])
Compute the recall.
roc_auc_score
(y_true,Ā y_score,Ā *[,Ā average,Ā ...])
Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.
zero_one_loss
(y_true,Ā y_pred,Ā *[,Ā ...])
Zero-one classification loss.
d2_log_loss_score
(y_true,Ā y_pred,Ā *[,Ā ...])
D
2
score function, fraction of log loss explained.
And some work with binary and multilabel (but not multiclass) problems:
In the following sub-sections, we will describe each of those functions,
preceded by some notes on common API and metric definition.
3.4.4.1.
From binary to multiclass and multilabel
#
Some metrics are essentially defined for binary classification tasks (e.g.
f1_score
,
roc_auc_score
). In these cases, by default
only the positive label is evaluated, assuming by default that the positive
class is labelled
1
(though this may be configurable through the
pos_label
parameter).
In extending a binary metric to multiclass or multilabel problems, the data
is treated as a collection of binary problems, one for each class.
There are then a number of ways to average binary metric calculations across
the set of classes, each of which may be useful in some scenario.
Where available, you should select among these using the
average
parameter.
"macro"
simply calculates the mean of the binary metrics,
giving equal weight to each class. In problems where infrequent classes
are nonetheless important, macro-averaging may be a means of highlighting
their performance. On the other hand, the assumption that all classes are
equally important is often untrue, such that macro-averaging will
over-emphasize the typically low performance on an infrequent class.
"weighted"
accounts for class imbalance by computing the average of
binary metrics in which each classās score is weighted by its presence in the
true data sample.
"micro"
gives each sample-class pair an equal contribution to the overall
metric (except as a result of sample-weight). Rather than summing the
metric per class, this sums the dividends and divisors that make up the
per-class metrics to calculate an overall quotient.
Micro-averaging may be preferred in multilabel settings, including
multiclass classification where a majority class is to be ignored.
"samples"
applies only to multilabel problems. It does not calculate a
per-class measure, instead calculating the metric over the true and predicted
classes for each sample in the evaluation data, and returning their
(
sample_weight
-weighted) average.
Selecting
average=None
will return an array with the score for each
class.
While multiclass data is provided to the metric, like binary targets, as an
array of class labels, multilabel data is specified as an indicator matrix,
in which cell
[i,
j]
has value 1 if sample
i
has label
j
and value
0 otherwise.
3.4.4.2.
Accuracy score
#
The
accuracy_score
function computes the
accuracy
, either the fraction
(default) or the count (normalize=False) of correct predictions.
In multilabel classification, the function returns the subset accuracy. If
the entire set of predicted labels for a sample strictly match with the true
set of labels, then the subset accuracy is 1.0; otherwise it is 0.0.
If
y
^
i
is the predicted value of
the
i
-th sample and
y
i
is the corresponding true value,
then the fraction of correct predictions over
n
samples
is
defined as
accuracy
(
y
,
y
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
1
(
y
^
i
=
y
i
)
where
1
(
x
)
is the
indicator function
.
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
accuracy_score
>>>
y_pred
=
[
0
,
2
,
1
,
3
]
>>>
y_true
=
[
0
,
1
,
2
,
3
]
>>>
accuracy_score
(
y_true
,
y_pred
)
0.5
>>>
accuracy_score
(
y_true
,
y_pred
,
normalize
=
False
)
2.0
In the multilabel case with binary label indicators:
>>>
accuracy_score
(
np
.
array
([[
0
,
1
],
[
1
,
1
]]),
np
.
ones
((
2
,
2
)))
0.5
Examples
See
Test with permutations the significance of a classification score
for an example of accuracy score usage using permutations of
the dataset.
3.4.4.3.
Top-k accuracy score
#
The
top_k_accuracy_score
function is a generalization of
accuracy_score
. The difference is that a prediction is considered
correct as long as the true label is associated with one of the
k
highest
predicted scores.
accuracy_score
is the special case of
k
=
1
.
The function covers the binary and multiclass classification cases but not the
multilabel case.
If
f
^
i
,
j
is the predicted class for the
i
-th sample
corresponding to the
j
-th largest predicted score and
y
i
is the
corresponding true value, then the fraction of correct predictions over
n
samples
is defined as
top-k accuracy
(
y
,
f
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
ā
j
=
1
k
1
(
f
^
i
,
j
=
y
i
)
where
k
is the number of guesses allowed and
1
(
x
)
is the
indicator function
.
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
top_k_accuracy_score
>>>
y_true
=
np
.
array
([
0
,
1
,
2
,
2
])
>>>
y_score
=
np
.
array
([[
0.5
,
0.2
,
0.2
],
...
[
0.3
,
0.4
,
0.2
],
...
[
0.2
,
0.4
,
0.3
],
...
[
0.7
,
0.2
,
0.1
]])
>>>
top_k_accuracy_score
(
y_true
,
y_score
,
k
=
2
)
0.75
>>>
# Not normalizing gives the number of "correctly" classified samples
>>>
top_k_accuracy_score
(
y_true
,
y_score
,
k
=
2
,
normalize
=
False
)
3.0
3.4.4.4.
Balanced accuracy score
#
The
balanced_accuracy_score
function computes the
balanced accuracy
, which avoids inflated
performance estimates on imbalanced datasets. It is the macro-average of recall
scores per class or, equivalently, raw accuracy where each sample is weighted
according to the inverse prevalence of its true class.
Thus for balanced datasets, the score is equal to accuracy.
In the binary case, balanced accuracy is equal to the arithmetic mean of
sensitivity
(true positive rate) and
specificity
(true negative
rate), or the area under the ROC curve with binary predictions rather than
scores:
balanced-accuracy
=
1
2
(
T
P
T
P
+
F
N
+
T
N
T
N
+
F
P
)
If the classifier performs equally well on either class, this term reduces to
the conventional accuracy (i.e., the number of correct predictions divided by
the total number of predictions).
In contrast, if the conventional accuracy is above chance only because the
classifier takes advantage of an imbalanced test set, then the balanced
accuracy, as appropriate, will drop to
1
n
_
c
l
a
s
s
e
s
.
The score ranges from 0 to 1, or when
adjusted=True
is used, it is rescaled to
the range
1
1
ā
n
_
c
l
a
s
s
e
s
to 1, inclusive, with
performance at random scoring 0.
If
y
i
is the true value of the
i
-th sample, and
w
i
is the corresponding sample weight, then we adjust the sample weight to:
w
^
i
=
w
i
ā
j
1
(
y
j
=
y
i
)
w
j
where
1
(
x
)
is the
indicator function
.
Given predicted
y
^
i
for sample
i
, balanced accuracy is
defined as:
balanced-accuracy
(
y
,
y
^
,
w
)
=
1
ā
w
^
i
ā
i
1
(
y
^
i
=
y
i
)
w
^
i
With
adjusted=True
, balanced accuracy reports the relative increase from
balanced-accuracy
(
y
,
0
,
w
)
=
1
n
_
c
l
a
s
s
e
s
. In the binary case, this is also known as
Youdenās J statistic
,
or
informedness
.
Note
The multiclass definition here seems the most reasonable extension of the
metric used in binary classification, though there is no certain consensus
in the literature:
Our definition:
[Mosley2013]
,
[Kelleher2015]
and
[Guyon2015]
, where
[Guyon2015]
adopt the adjusted version to ensure that random predictions
have a score of
0
and perfect predictions have a score of
1
.
Class balanced accuracy as described in
[Mosley2013]
: the minimum between the precision
and the recall for each class is computed. Those values are then averaged over the total
number of classes to get the balanced accuracy.
Balanced Accuracy as described in
[Urbanowicz2015]
: the average of sensitivity and specificity
is computed for each class and then averaged over total number of classes.
References
[
Guyon2015
]
(
1
,
2
)
I. Guyon, K. Bennett, G. Cawley, H.J. Escalante, S. Escalera, T.K. Ho, N. MaciĆ ,
B. Ray, M. Saeed, A.R. Statnikov, E. Viegas,
Design of the 2015 ChaLearn AutoML Challenge
, IJCNN 2015.
3.4.4.5.
Cohenās kappa
#
The function
cohen_kappa_score
computes
Cohenās kappa
statistic.
This measure is intended to compare labelings by different human annotators,
not a classifier versus a ground truth.
The kappa score is a number between -1 and 1.
Scores above .8 are generally considered good agreement;
zero or lower means no agreement (practically random labels).
Kappa scores can be computed for binary or multiclass problems,
but not for multilabel problems (except by manually computing a per-label score)
and not for more than two annotators.
>>>
from
sklearn.metrics
import
cohen_kappa_score
>>>
labeling1
=
[
2
,
0
,
2
,
2
,
0
,
1
]
>>>
labeling2
=
[
0
,
0
,
2
,
2
,
0
,
2
]
>>>
cohen_kappa_score
(
labeling1
,
labeling2
)
0.4285714285714286
3.4.4.6.
Confusion matrix
#
The
confusion_matrix
function evaluates
classification accuracy by computing the
confusion matrix
with each row corresponding
to the true class (Wikipedia and other references may use different convention
for axes).
By definition, entry
i
,
j
in a confusion matrix is
the number of observations actually in group
i
, but
predicted to be in group
j
. Here is an example:
>>>
from
sklearn.metrics
import
confusion_matrix
>>>
y_true
=
[
2
,
0
,
2
,
2
,
0
,
1
]
>>>
y_pred
=
[
0
,
0
,
2
,
2
,
0
,
2
]
>>>
confusion_matrix
(
y_true
,
y_pred
)
array([[2, 0, 0],
[0, 0, 1],
[1, 0, 2]])
ConfusionMatrixDisplay
can be used to visually represent a confusion
matrix as shown in the
Evaluate the performance of a classifier with Confusion Matrix
example, which creates the following figure:
The parameter
normalize
allows to report ratios instead of counts. The
confusion matrix can be normalized in 3 different ways:
'pred'
,
'true'
,
and
'all'
which will divide the counts by the sum of each columns, rows, or
the entire matrix, respectively.
>>>
y_true
=
[
0
,
0
,
0
,
1
,
1
,
1
,
1
,
1
]
>>>
y_pred
=
[
0
,
1
,
0
,
1
,
0
,
1
,
0
,
1
]
>>>
confusion_matrix
(
y_true
,
y_pred
,
normalize
=
'all'
)
array([[0.25 , 0.125],
[0.25 , 0.375]])
For binary problems, we can get counts of true negatives, false positives,
false negatives and true positives as follows:
>>>
y_true
=
[
0
,
0
,
0
,
1
,
1
,
1
,
1
,
1
]
>>>
y_pred
=
[
0
,
1
,
0
,
1
,
0
,
1
,
0
,
1
]
>>>
tn
,
fp
,
fn
,
tp
=
confusion_matrix
(
y_true
,
y_pred
)
.
ravel
()
.
tolist
()
>>>
tn
,
fp
,
fn
,
tp
(2, 1, 2, 3)
With
confusion_matrix_at_thresholds
we can get true negatives, false positives,
false negatives and true positives for different thresholds:
>>>
from
sklearn.metrics
import
confusion_matrix_at_thresholds
>>>
y_true
=
np
.
array
([
0.
,
0.
,
1.
,
1.
])
>>>
y_score
=
np
.
array
([
0.1
,
0.4
,
0.35
,
0.8
])
>>>
tns
,
fps
,
fns
,
tps
,
thresholds
=
confusion_matrix_at_thresholds
(
y_true
,
y_score
)
>>>
tns
array([2., 1., 1., 0.])
>>>
fps
array([0., 1., 1., 2.])
>>>
fns
array([1., 1., 0., 0.])
>>>
tps
array([1., 1., 2., 2.])
>>>
thresholds
array([0.8, 0.4, 0.35, 0.1])
Note that the thresholds consist of distinct
y_score
values, in decreasing order.
Examples
See
Evaluate the performance of a classifier with Confusion Matrix
for an example of using a confusion matrix to evaluate classifier output
quality.
See
Recognizing hand-written digits
for an example of using a confusion matrix to classify
hand-written digits.
See
Classification of text documents using sparse features
for an example of using a confusion matrix to classify text
documents.
3.4.4.7.
Classification report
#
The
classification_report
function builds a text report showing the
main classification metrics. Here is a small example with custom
target_names
and inferred labels:
>>>
from
sklearn.metrics
import
classification_report
>>>
y_true
=
[
0
,
1
,
2
,
2
,
0
]
>>>
y_pred
=
[
0
,
0
,
2
,
1
,
0
]
>>>
target_names
=
[
'class 0'
,
'class 1'
,
'class 2'
]
>>>
print
(
classification_report
(
y_true
,
y_pred
,
target_names
=
target_names
))
precision recall f1-score support
class 0 0.67 1.00 0.80 2
class 1 0.00 0.00 0.00 1
class 2 1.00 0.50 0.67 2
accuracy 0.60 5
macro avg 0.56 0.50 0.49 5
weighted avg 0.67 0.60 0.59 5
Examples
See
Recognizing hand-written digits
for an example of classification report usage for
hand-written digits.
See
Custom refit strategy of a grid search with cross-validation
for an example of classification report usage for
grid search with nested cross-validation.
3.4.4.8.
Hamming loss
#
The
hamming_loss
computes the average Hamming loss or
Hamming
distance
between two sets
of samples.
If
y
^
i
,
j
is the predicted value for the
j
-th label of a
given sample
i
,
y
i
,
j
is the corresponding true value,
n
samples
is the number of samples and
n
labels
is the number of labels, then the Hamming loss
L
H
a
m
m
i
n
g
is defined
as:
L
H
a
m
m
i
n
g
(
y
,
y
^
)
=
1
n
samples
ā
n
labels
ā
i
=
0
n
samples
ā
1
ā
j
=
0
n
labels
ā
1
1
(
y
^
i
,
j
ā
y
i
,
j
)
where
1
(
x
)
is the
indicator function
.
The equation above does not hold true in the case of multiclass classification.
Please refer to the note below for more information.
>>>
from
sklearn.metrics
import
hamming_loss
>>>
y_pred
=
[
1
,
2
,
3
,
4
]
>>>
y_true
=
[
2
,
2
,
3
,
4
]
>>>
hamming_loss
(
y_true
,
y_pred
)
0.25
In the multilabel case with binary label indicators:
>>>
hamming_loss
(
np
.
array
([[
0
,
1
],
[
1
,
1
]]),
np
.
zeros
((
2
,
2
)))
0.75
Note
In multiclass classification, the Hamming loss corresponds to the Hamming
distance between
y_true
and
y_pred
which is similar to the
Zero one loss
function. However, while zero-one loss penalizes
prediction sets that do not strictly match true sets, the Hamming loss
penalizes individual labels. Thus the Hamming loss, upper bounded by the zero-one
loss, is always between zero and one, inclusive; and predicting a proper subset
or superset of the true labels will give a Hamming loss between
zero and one, exclusive.
3.4.4.9.
Precision, recall and F-measures
#
Intuitively,
precision
is the ability
of the classifier not to label as positive a sample that is negative, and
recall
is the
ability of the classifier to find all the positive samples.
The
F-measure
(
F
Ī²
and
F
1
measures) can be interpreted as a weighted
harmonic mean of the precision and recall. A
F
Ī²
measure reaches its best value at 1 and its worst score at 0.
With
Ī²
=
1
,
F
Ī²
and
F
1
are equivalent, and the recall and the precision are equally important.
The
precision_recall_curve
computes a precision-recall curve
from the ground truth label and a score given by the classifier
by varying a decision threshold.
The
average_precision_score
function computes the
average precision
(AP) from prediction scores. The value is between 0 and 1 and higher is better.
AP is defined as
AP
=
ā
n
(
R
n
ā
R
n
ā
1
)
P
n
where
P
n
and
R
n
are the precision and recall at the
nth threshold. With random predictions, the AP is the fraction of positive
samples.
References
[Manning2008]
and
[Everingham2010]
present alternative variants of
AP that interpolate the precision-recall curve. Currently,
average_precision_score
does not implement any interpolated variant.
References
[Davis2006]
and
[Flach2015]
describe why a linear interpolation of
points on the precision-recall curve provides an overly-optimistic measure of
classifier performance. This linear interpolation is used when computing area
under the curve with the trapezoidal rule in
auc
.
[Chen2024]
benchmarks different interpolation strategies to demonstrate the effects.
Several functions allow you to analyze the precision, recall and F-measures
score:
average_precision_score
(y_true,Ā y_score,Ā *)
Compute average precision (AP) from prediction scores.
f1_score
(y_true,Ā y_pred,Ā *[,Ā labels,Ā ...])
Compute the F1 score, also known as balanced F-score or F-measure.
fbeta_score
(y_true,Ā y_pred,Ā *,Ā beta[,Ā ...])
Compute the F-beta score.
precision_recall_curve
(y_true,Ā y_score,Ā *[,Ā ...])
Compute precision-recall pairs for different probability thresholds.
precision_recall_fscore_support
(y_true,Ā ...)
Compute precision, recall, F-measure and support for each class.
precision_score
(y_true,Ā y_pred,Ā *[,Ā labels,Ā ...])
Compute the precision.
recall_score
(y_true,Ā y_pred,Ā *[,Ā labels,Ā ...])
Compute the recall.
Note that the
precision_recall_curve
function is restricted to the
binary case. The
average_precision_score
function supports multiclass
and multilabel formats by computing each class score in a One-vs-the-rest (OvR)
fashion and averaging them or not depending of its
average
argument value.
The
PrecisionRecallDisplay.from_estimator
and
PrecisionRecallDisplay.from_predictions
functions will plot the
precision-recall curve as follows.
Examples
See
Custom refit strategy of a grid search with cross-validation
for an example of
precision_score
and
recall_score
usage
to estimate parameters using grid search with nested cross-validation.
See
Precision-Recall
for an example of
precision_recall_curve
usage to evaluate
classifier output quality.
References
[
Chen2024
]
W. Chen, C. Miao, Z. Zhang, C.S. Fung, R. Wang, Y. Chen, Y. Qian, L. Cheng, K.Y. Yip, S.K
Tsui, Q. Cao,
Commonly used software tools produce conflicting and overly-optimistic AUPRC values
, Genome Biology 2024.
3.4.4.9.1.
Binary classification
#
In a binary classification task, the terms āāpositiveāā and āānegativeāā refer
to the classifierās prediction, and the terms āātrueāā and āāfalseāā refer to
whether that prediction corresponds to the external judgment (sometimes known
as the āāobservationāā). Given these definitions, we can formulate the
following table:
In this context, we can define the notions of precision and recall:
precision
=
tp
tp
+
fp
,
recall
=
tp
tp
+
fn
,
(Sometimes recall is also called āāsensitivityāā)
F-measure is the weighted harmonic mean of precision and recall, with precisionās
contribution to the mean weighted by some parameter
Ī²
:
F
Ī²
=
(
1
+
Ī²
2
)
precision
Ć
recall
Ī²
2
precision
+
recall
To avoid division by zero when precision and recall are zero, Scikit-Learn calculates F-measure with this
otherwise-equivalent formula:
F
Ī²
=
(
1
+
Ī²
2
)
tp
(
1
+
Ī²
2
)
tp
+
fp
+
Ī²
2
fn
Note that this formula is still undefined when there are no true positives, false
positives, or false negatives. By default, F-1 for a set of exclusively true negatives
is calculated as 0, however this behavior can be changed using the
zero_division
parameter.
Here are some small examples in binary classification:
>>>
from
sklearn
import
metrics
>>>
y_pred
=
[
0
,
1
,
0
,
0
]
>>>
y_true
=
[
0
,
1
,
0
,
1
]
>>>
metrics
.
precision_score
(
y_true
,
y_pred
)
1.0
>>>
metrics
.
recall_score
(
y_true
,
y_pred
)
0.5
>>>
metrics
.
f1_score
(
y_true
,
y_pred
)
0.66
>>>
metrics
.
fbeta_score
(
y_true
,
y_pred
,
beta
=
0.5
)
0.83
>>>
metrics
.
fbeta_score
(
y_true
,
y_pred
,
beta
=
1
)
0.66
>>>
metrics
.
fbeta_score
(
y_true
,
y_pred
,
beta
=
2
)
0.55
>>>
metrics
.
precision_recall_fscore_support
(
y_true
,
y_pred
,
beta
=
0.5
)
(array([0.66, 1. ]), array([1. , 0.5]), array([0.71, 0.83]), array([2, 2]))
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
precision_recall_curve
>>>
from
sklearn.metrics
import
average_precision_score
>>>
y_true
=
np
.
array
([
0
,
0
,
1
,
1
])
>>>
y_scores
=
np
.
array
([
0.1
,
0.4
,
0.35
,
0.8
])
>>>
precision
,
recall
,
threshold
=
precision_recall_curve
(
y_true
,
y_scores
)
>>>
precision
array([0.5 , 0.66, 0.5 , 1. , 1. ])
>>>
recall
array([1. , 1. , 0.5, 0.5, 0. ])
>>>
threshold
array([0.1 , 0.35, 0.4 , 0.8 ])
>>>
average_precision_score
(
y_true
,
y_scores
)
0.83
3.4.4.9.2.
Multiclass and multilabel classification
#
In a multiclass and multilabel classification task, the notions of precision,
recall, and F-measures can be applied to each label independently.
There are a few ways to combine results across labels,
specified by the
average
argument to the
average_precision_score
,
f1_score
,
fbeta_score
,
precision_recall_fscore_support
,
precision_score
and
recall_score
functions, as described
above
.
Note the following behaviors when averaging:
If all labels are included, āmicroā-averaging in a multiclass setting will produce
precision, recall and
F
that are all identical to accuracy.
āweightedā averaging may produce a F-score that is not between precision and recall.
āmacroā averaging for F-measures is calculated as the arithmetic mean over
per-label/class F-measures, not the harmonic mean over the arithmetic precision and
recall means. Both calculations can be seen in the literature but are not equivalent,
see
[OB2019]
for details.
To make this more explicit, consider the following notation:
y
the set of
true
(
s
a
m
p
l
e
,
l
a
b
e
l
)
pairs
y
^
the set of
predicted
(
s
a
m
p
l
e
,
l
a
b
e
l
)
pairs
L
the set of labels
S
the set of samples
y
s
the subset of
y
with sample
s
,
i.e.
y
s
:=
{
(
s
ā²
,
l
)
ā
y
|
s
ā²
=
s
}
y
l
the subset of
y
with label
l
similarly,
y
^
s
and
y
^
l
are subsets of
y
^
P
(
A
,
B
)
:=
|
A
ā©
B
|
|
B
|
for some
sets
A
and
B
R
(
A
,
B
)
:=
|
A
ā©
B
|
|
A
|
(Conventions vary on handling
A
=
ā
; this implementation uses
R
(
A
,
B
)
:=
0
, and similar for
P
.)
F
Ī²
(
A
,
B
)
:=
(
1
+
Ī²
2
)
P
(
A
,
B
)
Ć
R
(
A
,
B
)
Ī²
2
P
(
A
,
B
)
+
R
(
A
,
B
)
Then the metrics are defined as:
average
Precision
Recall
F_beta
"micro"
P
(
y
,
y
^
)
R
(
y
,
y
^
)
F
Ī²
(
y
,
y
^
)
"samples"
1
|
S
|
ā
s
ā
S
P
(
y
s
,
y
^
s
)
1
|
S
|
ā
s
ā
S
R
(
y
s
,
y
^
s
)
1
|
S
|
ā
s
ā
S
F
Ī²
(
y
s
,
y
^
s
)
"macro"
1
|
L
|
ā
l
ā
L
P
(
y
l
,
y
^
l
)
1
|
L
|
ā
l
ā
L
R
(
y
l
,
y
^
l
)
1
|
L
|
ā
l
ā
L
F
Ī²
(
y
l
,
y
^
l
)
"weighted"
1
ā
l
ā
L
|
y
l
|
ā
l
ā
L
|
y
l
|
P
(
y
l
,
y
^
l
)
1
ā
l
ā
L
|
y
l
|
ā
l
ā
L
|
y
l
|
R
(
y
l
,
y
^
l
)
1
ā
l
ā
L
|
y
l
|
ā
l
ā
L
|
y
l
|
F
Ī²
(
y
l
,
y
^
l
)
None
āØ
P
(
y
l
,
y
^
l
)
|
l
ā
L
ā©
āØ
R
(
y
l
,
y
^
l
)
|
l
ā
L
ā©
āØ
F
Ī²
(
y
l
,
y
^
l
)
|
l
ā
L
ā©
>>>
from
sklearn
import
metrics
>>>
y_true
=
[
0
,
1
,
2
,
0
,
1
,
2
]
>>>
y_pred
=
[
0
,
2
,
1
,
0
,
0
,
1
]
>>>
metrics
.
precision_score
(
y_true
,
y_pred
,
average
=
'macro'
)
0.22
>>>
metrics
.
recall_score
(
y_true
,
y_pred
,
average
=
'micro'
)
0.33
>>>
metrics
.
f1_score
(
y_true
,
y_pred
,
average
=
'weighted'
)
0.267
>>>
metrics
.
fbeta_score
(
y_true
,
y_pred
,
average
=
'macro'
,
beta
=
0.5
)
0.238
>>>
metrics
.
precision_recall_fscore_support
(
y_true
,
y_pred
,
beta
=
0.5
,
average
=
None
)
(array([0.667, 0., 0.]), array([1., 0., 0.]), array([0.714, 0., 0.]), array([2, 2, 2]))
For multiclass classification with a ānegative classā, it is possible to exclude some labels:
>>>
metrics
.
recall_score
(
y_true
,
y_pred
,
labels
=
[
1
,
2
],
average
=
'micro'
)
...
# excluding 0, no labels were correctly recalled
0.0
Similarly, labels not present in the data sample may be accounted for in macro-averaging.
>>>
metrics
.
precision_score
(
y_true
,
y_pred
,
labels
=
[
0
,
1
,
2
,
3
],
average
=
'macro'
)
0.166
References
3.4.4.10.
Jaccard similarity coefficient score
#
The
jaccard_score
function computes the average of
Jaccard similarity
coefficients
, also called the
Jaccard index, between pairs of label sets.
The Jaccard similarity coefficient with a ground truth label set
y
and
predicted label set
y
^
, is defined as
J
(
y
,
y
^
)
=
|
y
ā©
y
^
|
|
y
āŖ
y
^
|
.
The
jaccard_score
(like
precision_recall_fscore_support
) applies
natively to binary targets. By computing it set-wise it can be extended to apply
to multilabel and multiclass through the use of
average
(see
above
).
In the binary case:
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
jaccard_score
>>>
y_true
=
np
.
array
([[
0
,
1
,
1
],
...
[
1
,
1
,
0
]])
>>>
y_pred
=
np
.
array
([[
1
,
1
,
1
],
...
[
1
,
0
,
0
]])
>>>
jaccard_score
(
y_true
[
0
],
y_pred
[
0
])
0.6666
In the 2D comparison case (e.g. image similarity):
>>>
jaccard_score
(
y_true
,
y_pred
,
average
=
"micro"
)
0.6
In the multilabel case with binary label indicators:
>>>
jaccard_score
(
y_true
,
y_pred
,
average
=
'samples'
)
0.5833
>>>
jaccard_score
(
y_true
,
y_pred
,
average
=
'macro'
)
0.6666
>>>
jaccard_score
(
y_true
,
y_pred
,
average
=
None
)
array([0.5, 0.5, 1. ])
Multiclass problems are binarized and treated like the corresponding
multilabel problem:
>>>
y_pred
=
[
0
,
2
,
1
,
2
]
>>>
y_true
=
[
0
,
1
,
2
,
2
]
>>>
jaccard_score
(
y_true
,
y_pred
,
average
=
None
)
array([1. , 0. , 0.33])
>>>
jaccard_score
(
y_true
,
y_pred
,
average
=
'macro'
)
0.44
>>>
jaccard_score
(
y_true
,
y_pred
,
average
=
'micro'
)
0.33
3.4.4.11.
Hinge loss
#
The
hinge_loss
function computes the average distance between
the model and the data using
hinge loss
, a one-sided metric
that considers only prediction errors. (Hinge
loss is used in maximal margin classifiers such as support vector machines.)
If the true label
y
i
of a binary classification task is encoded as
y
i
=
{
ā
1
,
+
1
}
for every sample
i
; and
w
i
is the corresponding predicted decision (an array of shape (
n_samples
,) as
output by the
decision_function
method), then the hinge loss is defined as:
L
Hinge
(
y
,
w
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
max
{
1
ā
w
i
y
i
,
0
}
If there are more than two labels,
hinge_loss
uses a multiclass variant
due to Crammer & Singer.
Here
is
the paper describing it.
In this case the predicted decision is an array of shape (
n_samples
,
n_labels
). If
w
i
,
y
i
is the predicted decision for the true label
y
i
of the
i
-th sample; and
w
^
i
,
y
i
=
max
{
w
i
,
y
j
Ā
|
Ā
y
j
ā
y
i
}
is the maximum of the
predicted decisions for all the other labels, then the multi-class hinge loss
is defined by:
L
Hinge
(
y
,
w
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
max
{
1
+
w
^
i
,
y
i
ā
w
i
,
y
i
,
0
}
Here is a small example demonstrating the use of the
hinge_loss
function
with an svm classifier in a binary class problem:
>>>
from
sklearn
import
svm
>>>
from
sklearn.metrics
import
hinge_loss
>>>
X
=
[[
0
],
[
1
]]
>>>
y
=
[
-
1
,
1
]
>>>
est
=
svm
.
LinearSVC
(
random_state
=
0
)
>>>
est
.
fit
(
X
,
y
)
LinearSVC(random_state=0)
>>>
pred_decision
=
est
.
decision_function
([[
-
2
],
[
3
],
[
0.5
]])
>>>
pred_decision
array([-2.18, 2.36, 0.09])
>>>
hinge_loss
([
-
1
,
1
,
1
],
pred_decision
)
0.3
Here is an example demonstrating the use of the
hinge_loss
function
with an svm classifier in a multiclass problem:
>>>
X
=
np
.
array
([[
0
],
[
1
],
[
2
],
[
3
]])
>>>
Y
=
np
.
array
([
0
,
1
,
2
,
3
])
>>>
labels
=
np
.
array
([
0
,
1
,
2
,
3
])
>>>
est
=
svm
.
LinearSVC
()
>>>
est
.
fit
(
X
,
Y
)
LinearSVC()
>>>
pred_decision
=
est
.
decision_function
([[
-
1
],
[
2
],
[
3
]])
>>>
y_true
=
[
0
,
2
,
3
]
>>>
hinge_loss
(
y_true
,
pred_decision
,
labels
=
labels
)
0.56
3.4.4.12.
Log loss
#
Log loss, also called logistic regression loss or
cross-entropy loss, is defined on probability estimates. It is
commonly used in (multinomial) logistic regression and neural networks, as well
as in some variants of expectation-maximization, and can be used to evaluate the
probability outputs (
predict_proba
) of a classifier instead of its
discrete predictions.
For binary classification with a true label
y
ā
{
0
,
1
}
and a probability estimate
p
^
ā
Pr
(
y
=
1
)
,
the log loss per sample is the negative log-likelihood
of the classifier given the true label:
L
log
(
y
,
p
^
)
=
ā
log
ā”
Pr
(
y
|
p
^
)
=
ā
(
y
log
ā”
(
p
^
)
+
(
1
ā
y
)
log
ā”
(
1
ā
p
^
)
)
This extends to the multiclass case as follows.
Let the true labels for a set of samples
be encoded as a 1-of-K binary indicator matrix
Y
,
i.e.,
y
i
,
k
=
1
if sample
i
has label
k
taken from a set of
K
labels.
Let
P
^
be a matrix of probability estimates,
with elements
p
^
i
,
k
ā
Pr
(
y
i
,
k
=
1
)
.
Then the log loss of the whole set is
L
log
(
Y
,
P
^
)
=
ā
log
ā”
Pr
(
Y
|
P
^
)
=
ā
1
N
ā
i
=
0
N
ā
1
ā
k
=
0
K
ā
1
y
i
,
k
log
ā”
p
^
i
,
k
To see how this generalizes the binary log loss given above,
note that in the binary case,
p
^
i
,
0
=
1
ā
p
^
i
,
1
and
y
i
,
0
=
1
ā
y
i
,
1
,
so expanding the inner sum over
y
i
,
k
ā
{
0
,
1
}
gives the binary log loss.
The
log_loss
function computes log loss given a list of ground-truth
labels and a probability matrix, as returned by an estimatorās
predict_proba
method.
>>>
from
sklearn.metrics
import
log_loss
>>>
y_true
=
[
0
,
0
,
1
,
1
]
>>>
y_pred
=
[[
.9
,
.1
],
[
.8
,
.2
],
[
.3
,
.7
],
[
.01
,
.99
]]
>>>
log_loss
(
y_true
,
y_pred
)
0.1738
The first
[.9,
.1]
in
y_pred
denotes 90% probability that the first
sample has label 0. The log loss is non-negative.
3.4.4.13.
Matthews correlation coefficient
#
The
matthews_corrcoef
function computes the
Matthewās correlation coefficient (MCC)
for binary classes. Quoting Wikipedia:
āThe Matthews correlation coefficient is used in machine learning as a
measure of the quality of binary (two-class) classifications. It takes
into account true and false positives and negatives and is generally
regarded as a balanced measure which can be used even if the classes are
of very different sizes. The MCC is in essence a correlation coefficient
value between -1 and +1. A coefficient of +1 represents a perfect
prediction, 0 an average random prediction and -1 an inverse prediction.
The statistic is also known as the phi coefficient.ā
In the binary (two-class) case,
t
p
,
t
n
,
f
p
and
f
n
are respectively the number of true positives, true negatives, false
positives and false negatives, the MCC is defined as
M
C
C
=
t
p
Ć
t
n
ā
f
p
Ć
f
n
(
t
p
+
f
p
)
(
t
p
+
f
n
)
(
t
n
+
f
p
)
(
t
n
+
f
n
)
.
In the multiclass case, the Matthews correlation coefficient can be
defined
in terms of a
confusion_matrix
C
for
K
classes. To simplify the
definition consider the following intermediate variables:
t
k
=
ā
i
K
C
i
k
the number of times class
k
truly occurred,
p
k
=
ā
i
K
C
k
i
the number of times class
k
was predicted,
c
=
ā
k
K
C
k
k
the total number of samples correctly predicted,
s
=
ā
i
K
ā
j
K
C
i
j
the total number of samples.
Then the multiclass MCC is defined as:
M
C
C
=
c
Ć
s
ā
ā
k
K
p
k
Ć
t
k
(
s
2
ā
ā
k
K
p
k
2
)
Ć
(
s
2
ā
ā
k
K
t
k
2
)
When there are more than two labels, the value of the MCC will no longer range
between -1 and +1. Instead the minimum value will be somewhere between -1 and 0
depending on the number and distribution of ground truth labels. The maximum
value is always +1.
For additional information, see
[WikipediaMCC2021]
.
Here is a small example illustrating the usage of the
matthews_corrcoef
function:
>>>
from
sklearn.metrics
import
matthews_corrcoef
>>>
y_true
=
[
+
1
,
+
1
,
+
1
,
-
1
]
>>>
y_pred
=
[
+
1
,
-
1
,
+
1
,
+
1
]
>>>
matthews_corrcoef
(
y_true
,
y_pred
)
-0.33
References
3.4.4.14.
Multi-label confusion matrix
#
The
multilabel_confusion_matrix
function computes class-wise (default)
or sample-wise (samplewise=True) multilabel confusion matrix to evaluate
the accuracy of a classification. multilabel_confusion_matrix also treats
multiclass data as if it were multilabel, as this is a transformation commonly
applied to evaluate multiclass problems with binary classification metrics
(such as precision, recall, etc.).
When calculating class-wise multilabel confusion matrix
C
, the
count of true negatives for class
i
is
C
i
,
0
,
0
, false
negatives is
C
i
,
1
,
0
, true positives is
C
i
,
1
,
1
and false positives is
C
i
,
0
,
1
.
Here is an example demonstrating the use of the
multilabel_confusion_matrix
function with
multilabel indicator matrix
input:
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
multilabel_confusion_matrix
>>>
y_true
=
np
.
array
([[
1
,
0
,
1
],
...
[
0
,
1
,
0
]])
>>>
y_pred
=
np
.
array
([[
1
,
0
,
0
],
...
[
0
,
1
,
1
]])
>>>
multilabel_confusion_matrix
(
y_true
,
y_pred
)
array([[[1, 0],
[0, 1]],
[[1, 0],
[0, 1]],
[[0, 1],
[1, 0]]])
Or a confusion matrix can be constructed for each sampleās labels:
>>>
multilabel_confusion_matrix
(
y_true
,
y_pred
,
samplewise
=
True
)
array([[[1, 0],
[1, 1]],
[[1, 1],
[0, 1]]])
Here is an example demonstrating the use of the
multilabel_confusion_matrix
function with
multiclass
input:
>>>
y_true
=
[
"cat"
,
"ant"
,
"cat"
,
"cat"
,
"ant"
,
"bird"
]
>>>
y_pred
=
[
"ant"
,
"ant"
,
"cat"
,
"cat"
,
"ant"
,
"cat"
]
>>>
multilabel_confusion_matrix
(
y_true
,
y_pred
,
...
labels
=
[
"ant"
,
"bird"
,
"cat"
])
array([[[3, 1],
[0, 2]],
[[5, 0],
[1, 0]],
[[2, 1],
[1, 2]]])
Here are some examples demonstrating the use of the
multilabel_confusion_matrix
function to calculate recall
(or sensitivity), specificity, fall out and miss rate for each class in a
problem with multilabel indicator matrix input.
Calculating
recall
(also called the true positive rate or the sensitivity) for each class:
>>>
y_true
=
np
.
array
([[
0
,
0
,
1
],
...
[
0
,
1
,
0
],
...
[
1
,
1
,
0
]])
>>>
y_pred
=
np
.
array
([[
0
,
1
,
0
],
...
[
0
,
0
,
1
],
...
[
1
,
1
,
0
]])
>>>
mcm
=
multilabel_confusion_matrix
(
y_true
,
y_pred
)
>>>
tn
=
mcm
[:,
0
,
0
]
>>>
tp
=
mcm
[:,
1
,
1
]
>>>
fn
=
mcm
[:,
1
,
0
]
>>>
fp
=
mcm
[:,
0
,
1
]
>>>
tp
/
(
tp
+
fn
)
array([1. , 0.5, 0. ])
Calculating
specificity
(also called the true negative rate) for each class:
>>>
tn
/
(
tn
+
fp
)
array([1. , 0. , 0.5])
Calculating
fall out
(also called the false positive rate) for each class:
>>>
fp
/
(
fp
+
tn
)
array([0. , 1. , 0.5])
Calculating
miss rate
(also called the false negative rate) for each class:
>>>
fn
/
(
fn
+
tp
)
array([0. , 0.5, 1. ])
3.4.4.15.
Receiver operating characteristic (ROC)
#
The function
roc_curve
computes the
receiver operating characteristic curve, or ROC curve
.
Quoting Wikipedia :
āA receiver operating characteristic (ROC), or simply ROC curve, is a
graphical plot which illustrates the performance of a binary classifier
system as its discrimination threshold is varied. It is created by plotting
the fraction of true positives out of the positives (TPR = true positive
rate) vs. the fraction of false positives out of the negatives (FPR = false
positive rate), at various threshold settings. TPR is also known as
sensitivity, and FPR is one minus the specificity or true negative rate.ā
This function requires the true binary value and the target scores, which can
either be probability estimates of the positive class, confidence values, or
binary decisions. Here is a small example of how to use the
roc_curve
function:
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
roc_curve
>>>
y
=
np
.
array
([
1
,
1
,
2
,
2
])
>>>
scores
=
np
.
array
([
0.1
,
0.4
,
0.35
,
0.8
])
>>>
fpr
,
tpr
,
thresholds
=
roc_curve
(
y
,
scores
,
pos_label
=
2
)
>>>
fpr
array([0. , 0. , 0.5, 0.5, 1. ])
>>>
tpr
array([0. , 0.5, 0.5, 1. , 1. ])
>>>
thresholds
array([ inf, 0.8 , 0.4 , 0.35, 0.1 ])
Compared to metrics such as the subset accuracy, the Hamming loss, or the
F1 score, ROC doesnāt require optimizing a threshold for each label.
The
roc_auc_score
function, denoted by ROC-AUC or AUROC, computes the
area under the ROC curve. By doing so, the curve information is summarized in
one number.
The following figure shows the ROC curve and ROC-AUC score for a classifier
aimed to distinguish the virginica flower from the rest of the species in the
Iris plants dataset
:
For more information see the
Wikipedia article on AUC
.
3.4.4.15.1.
Binary case
#
In the
binary case
, you can either provide the probability estimates, using
the
classifier.predict_proba()
method, or the non-thresholded decision values
given by the
classifier.decision_function()
method. In the case of providing
the probability estimates, the probability of the class with the
āgreater labelā should be provided. The āgreater labelā corresponds to
classifier.classes_[1]
and thus
classifier.predict_proba(X)[:,
1]
.
Therefore, the
y_score
parameter is of size (n_samples,).
>>>
from
sklearn.datasets
import
load_breast_cancer
>>>
from
sklearn.linear_model
import
LogisticRegression
>>>
from
sklearn.metrics
import
roc_auc_score
>>>
X
,
y
=
load_breast_cancer
(
return_X_y
=
True
)
>>>
clf
=
LogisticRegression
()
.
fit
(
X
,
y
)
>>>
clf
.
classes_
array([0, 1])
We can use the probability estimates corresponding to
clf.classes_[1]
.
>>>
y_score
=
clf
.
predict_proba
(
X
)[:,
1
]
>>>
roc_auc_score
(
y
,
y_score
)
0.99
Otherwise, we can use the non-thresholded decision values
>>>
roc_auc_score
(
y
,
clf
.
decision_function
(
X
))
0.99
3.4.4.15.2.
Multi-class case
#
The
roc_auc_score
function can also be used in
multi-class
classification
. Two averaging strategies are currently supported: the
one-vs-one algorithm computes the average of the pairwise ROC AUC scores, and
the one-vs-rest algorithm computes the average of the ROC AUC scores for each
class against all other classes. In both cases, the predicted labels are
provided in an array with values from 0 to
n_classes
, and the scores
correspond to the probability estimates that a sample belongs to a particular
class. The OvO and OvR algorithms support weighting uniformly
(
average='macro'
) and by prevalence (
average='weighted'
).
Computes the average AUC of all possible pairwise
combinations of classes.
[HT2001]
defines a multiclass AUC metric weighted
uniformly:
1
c
(
c
ā
1
)
ā
j
=
1
c
ā
k
>
j
c
(
AUC
(
j
|
k
)
+
AUC
(
k
|
j
)
)
where
c
is the number of classes and
AUC
(
j
|
k
)
is the
AUC with class
j
as the positive class and class
k
as the
negative class. In general,
AUC
(
j
|
k
)
ā
AUC
(
k
|
j
)
in the multiclass
case. This algorithm is used by setting the keyword argument
multiclass
to
'ovo'
and
average
to
'macro'
.
The
[HT2001]
multiclass AUC metric can be extended to be weighted by the
prevalence:
1
c
(
c
ā
1
)
ā
j
=
1
c
ā
k
>
j
c
p
(
j
āŖ
k
)
(
AUC
(
j
|
k
)
+
AUC
(
k
|
j
)
)
where
c
is the number of classes. This algorithm is used by setting
the keyword argument
multiclass
to
'ovo'
and
average
to
'weighted'
. The
'weighted'
option returns a prevalence-weighted average
as described in
[FC2009]
.
Computes the AUC of each class against the rest
[PD2000]
. The algorithm is functionally the same as the multilabel case. To
enable this algorithm set the keyword argument
multiclass
to
'ovr'
.
Additionally to
'macro'
[F2006]
and
'weighted'
[F2001]
averaging, OvR
supports
'micro'
averaging.
In applications where a high false positive rate is not tolerable the parameter
max_fpr
of
roc_auc_score
can be used to summarize the ROC curve up
to the given limit.
The following figure shows the micro-averaged ROC curve and its corresponding
ROC-AUC score for a classifier aimed to distinguish the different species in
the
Iris plants dataset
:
3.4.4.15.3.
Multi-label case
#
In
multi-label classification
, the
roc_auc_score
function is
extended by averaging over the labels as
above
. In this case,
you should provide a
y_score
of shape
(n_samples,
n_classes)
. Thus, when
using the probability estimates, one needs to select the probability of the
class with the greater label for each output.
>>>
from
sklearn.datasets
import
make_multilabel_classification
>>>
from
sklearn.multioutput
import
MultiOutputClassifier
>>>
X
,
y
=
make_multilabel_classification
(
random_state
=
0
)
>>>
inner_clf
=
LogisticRegression
(
random_state
=
0
)
>>>
clf
=
MultiOutputClassifier
(
inner_clf
)
.
fit
(
X
,
y
)
>>>
y_score
=
np
.
transpose
([
y_pred
[:,
1
]
for
y_pred
in
clf
.
predict_proba
(
X
)])
>>>
roc_auc_score
(
y
,
y_score
,
average
=
None
)
array([0.828, 0.851, 0.94, 0.87, 0.95])
And the decision values do not require such processing.
>>>
from
sklearn.linear_model
import
RidgeClassifierCV
>>>
clf
=
RidgeClassifierCV
()
.
fit
(
X
,
y
)
>>>
y_score
=
clf
.
decision_function
(
X
)
>>>
roc_auc_score
(
y
,
y_score
,
average
=
None
)
array([0.82, 0.85, 0.93, 0.87, 0.94])
Examples
See
Multiclass Receiver Operating Characteristic (ROC)
for an example of
using ROC to evaluate the quality of the output of a classifier.
See
Receiver Operating Characteristic (ROC) with cross validation
for an
example of using ROC to evaluate classifier output quality, using cross-validation.
See
Species distribution modeling
for an example of using ROC to model species distribution.
References
3.4.4.16.
Detection error tradeoff (DET)
#
The function
det_curve
computes the
detection error tradeoff curve (DET) curve
[WikipediaDET2017]
.
Quoting Wikipedia:
āA detection error tradeoff (DET) graph is a graphical plot of error rates
for binary classification systems, plotting false reject rate vs. false
accept rate. The x- and y-axes are scaled non-linearly by their standard
normal deviates (or just by logarithmic transformation), yielding tradeoff
curves that are more linear than ROC curves, and use most of the image area
to highlight the differences of importance in the critical operating region.ā
DET curves are a variation of receiver operating characteristic (ROC) curves
where False Negative Rate is plotted on the y-axis instead of True Positive
Rate.
DET curves are commonly plotted in normal deviate scale by transformation with
Ļ
ā
1
(with
Ļ
being the cumulative distribution
function).
The resulting performance curves explicitly visualize the tradeoff of error
types for given classification algorithms.
See
[Martin1997]
for examples and further motivation.
This figure compares the ROC and DET curves of two example classifiers on the
same classification task:
DET curves form a linear curve in normal deviate scale if the detection
scores are normally (or close-to normally) distributed.
It was shown by
[Navratil2007]
that the reverse is not necessarily true and
even more general distributions are able to produce linear DET curves.
The normal deviate scale transformation spreads out the points such that a
comparatively larger space of plot is occupied.
Therefore curves with similar classification performance might be easier to
distinguish on a DET plot.
With False Negative Rate being āinverseā to True Positive Rate the point
of perfection for DET curves is the origin (in contrast to the top left
corner for ROC curves).
DET curves are intuitive to read and hence allow quick visual assessment of a
classifierās performance.
Additionally DET curves can be consulted for threshold analysis and operating
point selection.
This is particularly helpful if a comparison of error types is required.
On the other hand DET curves do not provide their metric as a single number.
Therefore for either automated evaluation or comparison to other
classification tasks metrics like the derived area under ROC curve might be
better suited.
Examples
See
Detection error tradeoff (DET) curve
for an example comparison between receiver operating characteristic (ROC)
curves and Detection error tradeoff (DET) curves.
References
[
Navratil2007
]
J. Navratil and D. Klusacek,
āOn Linear DETsā
,
2007 IEEE International Conference on Acoustics,
Speech and Signal Processing - ICASSP ā07, Honolulu,
HI, 2007, pp. IV-229-IV-232.
3.4.4.17.
Zero one loss
#
The
zero_one_loss
function computes the sum or the average of the 0-1
classification loss (
L
0
ā
1
) over
n
samples
. By
default, the function normalizes over the sample. To get the sum of the
L
0
ā
1
, set
normalize
to
False
.
In multilabel classification, the
zero_one_loss
scores a subset as
one if its labels strictly match the predictions, and as a zero if there
are any errors. By default, the function returns the percentage of imperfectly
predicted subsets. To get the count of such subsets instead, set
normalize
to
False
.
If
y
^
i
is the predicted value of
the
i
-th sample and
y
i
is the corresponding true value,
then the 0-1 loss
L
0
ā
1
is defined as:
L
0
ā
1
(
y
,
y
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
1
(
y
^
i
ā
y
i
)
where
1
(
x
)
is the
indicator function
. The zero-one
loss can also be computed as
zero-one loss
=
1
ā
accuracy
.
>>>
from
sklearn.metrics
import
zero_one_loss
>>>
y_pred
=
[
1
,
2
,
3
,
4
]
>>>
y_true
=
[
2
,
2
,
3
,
4
]
>>>
zero_one_loss
(
y_true
,
y_pred
)
0.25
>>>
zero_one_loss
(
y_true
,
y_pred
,
normalize
=
False
)
1.0
In the multilabel case with binary label indicators, where the first label
set [0,1] has an error:
>>>
zero_one_loss
(
np
.
array
([[
0
,
1
],
[
1
,
1
]]),
np
.
ones
((
2
,
2
)))
0.5
>>>
zero_one_loss
(
np
.
array
([[
0
,
1
],
[
1
,
1
]]),
np
.
ones
((
2
,
2
)),
normalize
=
False
)
1.0
Examples
See
Recursive feature elimination with cross-validation
for an example of zero one loss usage to perform recursive feature
elimination with cross-validation.
3.4.4.18.
Brier score loss
#
The
brier_score_loss
function computes the
Brier score
for binary and multiclass
probabilistic predictions and is equivalent to the mean squared error.
Quoting Wikipedia:
āThe Brier score is a strictly proper scoring rule that measures the accuracy of
probabilistic predictions. [ā¦] [It] is applicable to tasks in which predictions
must assign probabilities to a set of mutually exclusive discrete outcomes or
classes.ā
Let the true labels for a set of
N
data points be encoded as a 1-of-K binary
indicator matrix
Y
, i.e.,
y
i
,
k
=
1
if sample
i
has
label
k
taken from a set of
K
labels. Let
P
^
be a matrix
of probability estimates with elements
p
^
i
,
k
ā
Pr
(
y
i
,
k
=
1
)
.
Following the original definition by
[Brier1950]
, the Brier score is given by:
B
S
(
Y
,
P
^
)
=
1
N
ā
i
=
0
N
ā
1
ā
k
=
0
K
ā
1
(
y
i
,
k
ā
p
^
i
,
k
)
2
The Brier score lies in the interval
[
0
,
2
]
and the lower the value the
better the probability estimates are (the mean squared difference is smaller).
Actually, the Brier score is a strictly proper scoring rule, meaning that it
achieves the best score only when the estimated probabilities equal the
true ones.
Note that in the binary case, the Brier score is usually divided by two and
ranges between
[
0
,
1
]
. For binary targets
y
i
ā
{
0
,
1
}
and
probability estimates
p
^
i
ā
Pr
(
y
i
=
1
)
for the positive class, the Brier score is then equal to:
B
S
(
y
,
p
^
)
=
1
N
ā
i
=
0
N
ā
1
(
y
i
ā
p
^
i
)
2
The
brier_score_loss
function computes the Brier score given the
ground-truth labels and predicted probabilities, as returned by an estimatorās
predict_proba
method. The
scale_by_half
parameter controls which of the
two above definitions to follow.
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
brier_score_loss
>>>
y_true
=
np
.
array
([
0
,
1
,
1
,
0
])
>>>
y_true_categorical
=
np
.
array
([
"spam"
,
"ham"
,
"ham"
,
"spam"
])
>>>
y_prob
=
np
.
array
([
0.1
,
0.9
,
0.8
,
0.4
])
>>>
brier_score_loss
(
y_true
,
y_prob
)
0.055
>>>
brier_score_loss
(
y_true
,
1
-
y_prob
,
pos_label
=
0
)
0.055
>>>
brier_score_loss
(
y_true_categorical
,
y_prob
,
pos_label
=
"ham"
)
0.055
>>>
brier_score_loss
(
...
[
"eggs"
,
"ham"
,
"spam"
],
...
[[
0.8
,
0.1
,
0.1
],
[
0.2
,
0.7
,
0.1
],
[
0.2
,
0.2
,
0.6
]],
...
labels
=
[
"eggs"
,
"ham"
,
"spam"
],
...
)
0.146
The Brier score can be used to assess how well a classifier is calibrated.
However, a lower Brier score loss does not always mean a better calibration.
This is because, by analogy with the bias-variance decomposition of the mean
squared error, the Brier score loss can be decomposed as the sum of calibration
loss and refinement loss
[Bella2012]
. Calibration loss is defined as the mean
squared deviation from empirical probabilities derived from the slope of ROC
segments. Refinement loss can be defined as the expected optimal loss as
measured by the area under the optimal cost curve. Refinement loss can change
independently from calibration loss, thus a lower Brier score loss does not
necessarily mean a better calibrated model. āOnly when refinement loss remains
the same does a lower Brier score loss always mean better calibrationā
[Bella2012]
,
[Flach2008]
.
Examples
See
Probability calibration of classifiers
for an example of Brier score loss usage to perform probability
calibration of classifiers.
References
[
Bella2012
]
(
1
,
2
)
Bella, Ferri, HernĆ”ndez-Orallo, and RamĆrez-Quintana
āCalibration of Machine Learning Modelsā
in Khosrow-Pour, M. āMachine learning: concepts, methodologies, tools
and applications.ā Hershey, PA: Information Science Reference (2012).
3.4.4.19.
Class likelihood ratios
#
The
class_likelihood_ratios
function computes the
positive and negative
likelihood ratios
L
R
Ā±
for binary classes, which can be interpreted as the ratio of
post-test to pre-test odds as explained below. As a consequence, this metric is
invariant w.r.t. the class prevalence (the number of samples in the positive
class divided by the total number of samples) and
can be extrapolated between
populations regardless of any possible class imbalance.
The
L
R
Ā±
metrics are therefore very useful in settings where the data
available to learn and evaluate a classifier is a study population with nearly
balanced classes, such as a case-control study, while the target application,
i.e. the general population, has very low prevalence.
The positive likelihood ratio
L
R
+
is the probability of a classifier to
correctly predict that a sample belongs to the positive class divided by the
probability of predicting the positive class for a sample belonging to the
negative class:
L
R
+
=
PR
(
P
+
|
T
+
)
PR
(
P
+
|
T
ā
)
.
The notation here refers to predicted (
P
) or true (
T
) label and
the sign
+
and
ā
refer to the positive and negative class,
respectively, e.g.
P
+
stands for āpredicted positiveā.
Analogously, the negative likelihood ratio
L
R
ā
is the probability of a
sample of the positive class being classified as belonging to the negative class
divided by the probability of a sample of the negative class being correctly
classified:
L
R
ā
=
PR
(
P
ā
|
T
+
)
PR
(
P
ā
|
T
ā
)
.
For classifiers above chance
L
R
+
above 1
higher is better
, while
L
R
ā
ranges from 0 to 1 and
lower is better
.
Values of
L
R
Ā±
ā
1
correspond to chance level.
Notice that probabilities differ from counts, for instance
PR
(
P
+
|
T
+
)
is not equal to the number of true positive
counts
tp
(see
the wikipedia page
for
the actual formulas).
Examples
Class Likelihood Ratios to measure classification performance
Both class likelihood ratios are interpretable in terms of an odds ratio
(pre-test and post-tests):
post-test odds
=
Likelihood ratio
Ć
pre-test odds
.
Odds are in general related to probabilities via
odds
=
probability
1
ā
probability
,
or equivalently
probability
=
odds
1
+
odds
.
On a given population, the pre-test probability is given by the prevalence. By
converting odds to probabilities, the likelihood ratios can be translated into a
probability of truly belonging to either class before and after a classifier
prediction:
post-test odds
=
Likelihood ratio
Ć
pre-test probability
1
ā
pre-test probability
,
post-test probability
=
post-test odds
1
+
post-test odds
.
The positive likelihood ratio (
LR+
) is undefined when
f
p
=
0
, meaning the
classifier does not misclassify any negative labels as positives. This condition can
either indicate a perfect identification of all the negative cases or, if there are
also no true positive predictions (
t
p
=
0
), that the classifier does not predict
the positive class at all. In the first case,
LR+
can be interpreted as
np.inf
, in
the second case (for instance, with highly imbalanced data) it can be interpreted as
np.nan
.
The negative likelihood ratio (
LR-
) is undefined when
t
n
=
0
. Such
divergence is invalid, as
L
R
ā
>
1.0
would indicate an increase in the odds of
a sample belonging to the positive class after being classified as negative, as if the
act of classifying caused the positive condition. This includes the case of a
DummyClassifier
that always predicts the positive class
(i.e. when
t
n
=
f
n
=
0
).
Both class likelihood ratios (
LR+
and
LR-
) are undefined when
t
p
=
f
n
=
0
, which
means that no samples of the positive class were present in the test set. This can
happen when cross-validating on highly imbalanced data and also leads to a division by
zero.
If a division by zero occurs and
raise_warning
is set to
True
(default),
class_likelihood_ratios
raises an
UndefinedMetricWarning
and returns
np.nan
by default to avoid pollution when averaging over cross-validation folds.
Users can set return values in case of a division by zero with the
replace_undefined_by
param.
For a worked-out demonstration of the
class_likelihood_ratios
function,
see the example below.
Wikipedia entry for Likelihood ratios in diagnostic testing
Brenner, H., & Gefeller, O. (1997).
Variation of sensitivity, specificity, likelihood ratios and predictive
values with disease prevalence. Statistics in medicine, 16(9), 981-991.
3.4.4.20.
DĀ² score for classification
#
The DĀ² score computes the fraction of deviance explained.
It is a generalization of RĀ², where the squared error is generalized and replaced
by a classification deviance of choice
dev
(
y
,
y
^
)
(e.g., Log loss, Brier score,). DĀ² is a form of a
skill score
.
It is calculated as
D
2
(
y
,
y
^
)
=
1
ā
dev
(
y
,
y
^
)
dev
(
y
,
y
null
)
.
Where
y
null
is the optimal prediction of an intercept-only model
(e.g., the per-class proportion of
y_true
in the case of the Log loss and Brier score).
Like RĀ², the best possible score is 1.0 and it can be negative (because the
model can be arbitrarily worse). A constant model that always predicts
y
null
, disregarding the input features, would get a DĀ² score
of 0.0.
The
d2_log_loss_score
function implements the special case
of DĀ² with the log loss, see
Log loss
, i.e.:
dev
(
y
,
y
^
)
=
log_loss
(
y
,
y
^
)
.
Here are some usage examples of the
d2_log_loss_score
function:
>>>
from
sklearn.metrics
import
d2_log_loss_score
>>>
y_true
=
[
1
,
1
,
2
,
3
]
>>>
y_pred
=
[
...
[
0.5
,
0.25
,
0.25
],
...
[
0.5
,
0.25
,
0.25
],
...
[
0.5
,
0.25
,
0.25
],
...
[
0.5
,
0.25
,
0.25
],
...
]
>>>
d2_log_loss_score
(
y_true
,
y_pred
)
0.0
>>>
y_true
=
[
1
,
2
,
3
]
>>>
y_pred
=
[
...
[
0.98
,
0.01
,
0.01
],
...
[
0.01
,
0.98
,
0.01
],
...
[
0.01
,
0.01
,
0.98
],
...
]
>>>
d2_log_loss_score
(
y_true
,
y_pred
)
0.981
>>>
y_true
=
[
1
,
2
,
3
]
>>>
y_pred
=
[
...
[
0.1
,
0.6
,
0.3
],
...
[
0.1
,
0.6
,
0.3
],
...
[
0.4
,
0.5
,
0.1
],
...
]
>>>
d2_log_loss_score
(
y_true
,
y_pred
)
-0.552
The
d2_brier_score
function implements the special case
of DĀ² with the Brier score, see
Brier score loss
, i.e.:
dev
(
y
,
y
^
)
=
brier_score_loss
(
y
,
y
^
)
.
This is also referred to as the Brier Skill Score (BSS).
Here are some usage examples of the
d2_brier_score
function:
>>>
from
sklearn.metrics
import
d2_brier_score
>>>
y_true
=
[
1
,
1
,
2
,
3
]
>>>
y_pred
=
[
...
[
0.5
,
0.25
,
0.25
],
...
[
0.5
,
0.25
,
0.25
],
...
[
0.5
,
0.25
,
0.25
],
...
[
0.5
,
0.25
,
0.25
],
...
]
>>>
d2_brier_score
(
y_true
,
y_pred
)
0.0
>>>
y_true
=
[
1
,
2
,
3
]
>>>
y_pred
=
[
...
[
0.98
,
0.01
,
0.01
],
...
[
0.01
,
0.98
,
0.01
],
...
[
0.01
,
0.01
,
0.98
],
...
]
>>>
d2_brier_score
(
y_true
,
y_pred
)
0.9991
>>>
y_true
=
[
1
,
2
,
3
]
>>>
y_pred
=
[
...
[
0.1
,
0.6
,
0.3
],
...
[
0.1
,
0.6
,
0.3
],
...
[
0.4
,
0.5
,
0.1
],
...
]
>>>
d2_brier_score
(
y_true
,
y_pred
)
-0.370...
3.4.5.
Multilabel ranking metrics
#
In multilabel learning, each sample can have any number of ground truth labels
associated with it. The goal is to give high scores and better rank to
the ground truth labels.
3.4.5.1.
Coverage error
#
The
coverage_error
function computes the average number of labels that
have to be included in the final prediction such that all true labels
are predicted. This is useful if you want to know how many top-scored-labels
you have to predict in average without missing any true one. The best value
of this metric is thus the average number of true labels.
Note
Our implementationās score is 1 greater than the one given in Tsoumakas
et al., 2010. This extends it to handle the degenerate case in which an
instance has 0 true labels.
Formally, given a binary indicator matrix of the ground truth labels
y
ā
{
0
,
1
}
n
samples
Ć
n
labels
and the
score associated with each label
f
^
ā
R
n
samples
Ć
n
labels
,
the coverage is defined as
c
o
v
e
r
a
g
e
(
y
,
f
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
max
j
:
y
i
j
=
1
rank
i
j
with
rank
i
j
=
|
{
k
:
f
^
i
k
ā„
f
^
i
j
}
|
.
Given the rank definition, ties in
y_scores
are broken by giving the
maximal rank that would have been assigned to all tied values.
Here is a small example of usage of this function:
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
coverage_error
>>>
y_true
=
np
.
array
([[
1
,
0
,
0
],
[
0
,
0
,
1
]])
>>>
y_score
=
np
.
array
([[
0.75
,
0.5
,
1
],
[
1
,
0.2
,
0.1
]])
>>>
coverage_error
(
y_true
,
y_score
)
2.5
3.4.5.2.
Label ranking average precision
#
The
label_ranking_average_precision_score
function
implements label ranking average precision (LRAP). This metric is linked to
the
average_precision_score
function, but is based on the notion of
label ranking instead of precision and recall.
Label ranking average precision (LRAP) averages over the samples the answer to
the following question: for each ground truth label, what fraction of
higher-ranked labels were true labels? This performance measure will be higher
if you are able to give better rank to the labels associated with each sample.
The obtained score is always strictly greater than 0, and the best value is 1.
If there is exactly one relevant label per sample, label ranking average
precision is equivalent to the
mean
reciprocal rank
.
Formally, given a binary indicator matrix of the ground truth labels
y
ā
{
0
,
1
}
n
samples
Ć
n
labels
and the score associated with each label
f
^
ā
R
n
samples
Ć
n
labels
,
the average precision is defined as
L
R
A
P
(
y
,
f
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
1
|
|
y
i
|
|
0
ā
j
:
y
i
j
=
1
|
L
i
j
|
rank
i
j
where
L
i
j
=
{
k
:
y
i
k
=
1
,
f
^
i
k
ā„
f
^
i
j
}
,
rank
i
j
=
|
{
k
:
f
^
i
k
ā„
f
^
i
j
}
|
,
|
ā
|
computes the cardinality of the set (i.e., the number of
elements in the set), and
|
|
ā
|
|
0
is the
ā
0
ānormā
(which computes the number of nonzero elements in a vector).
Here is a small example of usage of this function:
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
label_ranking_average_precision_score
>>>
y_true
=
np
.
array
([[
1
,
0
,
0
],
[
0
,
0
,
1
]])
>>>
y_score
=
np
.
array
([[
0.75
,
0.5
,
1
],
[
1
,
0.2
,
0.1
]])
>>>
label_ranking_average_precision_score
(
y_true
,
y_score
)
0.416
3.4.5.3.
Ranking loss
#
The
label_ranking_loss
function computes the ranking loss which
averages over the samples the number of label pairs that are incorrectly
ordered, i.e. true labels have a lower score than false labels, weighted by
the inverse of the number of ordered pairs of false and true labels.
The lowest achievable ranking loss is zero.
Formally, given a binary indicator matrix of the ground truth labels
y
ā
{
0
,
1
}
n
samples
Ć
n
labels
and the
score associated with each label
f
^
ā
R
n
samples
Ć
n
labels
,
the ranking loss is defined as
r
a
n
k
i
n
g
_
l
o
s
s
(
y
,
f
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
1
|
|
y
i
|
|
0
(
n
labels
ā
|
|
y
i
|
|
0
)
|
{
(
k
,
l
)
:
f
^
i
k
ā¤
f
^
i
l
,
y
i
k
=
1
,
y
i
l
=
0
}
|
where
|
ā
|
computes the cardinality of the set (i.e., the number of
elements in the set) and
|
|
ā
|
|
0
is the
ā
0
ānormā
(which computes the number of nonzero elements in a vector).
Here is a small example of usage of this function:
>>>
import
numpy
as
np
>>>
from
sklearn.metrics
import
label_ranking_loss
>>>
y_true
=
np
.
array
([[
1
,
0
,
0
],
[
0
,
0
,
1
]])
>>>
y_score
=
np
.
array
([[
0.75
,
0.5
,
1
],
[
1
,
0.2
,
0.1
]])
>>>
label_ranking_loss
(
y_true
,
y_score
)
0.75
>>>
# With the following prediction, we have perfect and minimal loss
>>>
y_score
=
np
.
array
([[
1.0
,
0.1
,
0.2
],
[
0.1
,
0.2
,
0.9
]])
>>>
label_ranking_loss
(
y_true
,
y_score
)
0.0
Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In
Data mining and knowledge discovery handbook (pp. 667-685). Springer US.
3.4.5.4.
Normalized Discounted Cumulative Gain
#
Discounted Cumulative Gain (DCG) and Normalized Discounted Cumulative Gain
(NDCG) are ranking metrics implemented in
dcg_score
and
ndcg_score
; they compare a predicted order to
ground-truth scores, such as the relevance of answers to a query.
From the Wikipedia page for Discounted Cumulative Gain:
āDiscounted cumulative gain (DCG) is a measure of ranking quality. In
information retrieval, it is often used to measure effectiveness of web search
engine algorithms or related applications. Using a graded relevance scale of
documents in a search-engine result set, DCG measures the usefulness, or gain,
of a document based on its position in the result list. The gain is accumulated
from the top of the result list to the bottom, with the gain of each result
discounted at lower ranks.ā
DCG orders the true targets (e.g. relevance of query answers) in the predicted
order, then multiplies them by a logarithmic decay and sums the result. The sum
can be truncated after the first
K
results, in which case we call it
DCG@K.
NDCG, or NDCG@K is DCG divided by the DCG obtained by a perfect prediction, so
that it is always between 0 and 1. Usually, NDCG is preferred to DCG.
Compared with the ranking loss, NDCG can take into account relevance scores,
rather than a ground-truth ranking. So if the ground-truth consists only of an
ordering, the ranking loss should be preferred; if the ground-truth consists of
actual usefulness scores (e.g. 0 for irrelevant, 1 for relevant, 2 for very
relevant), NDCG can be used.
For one sample, given the vector of continuous ground-truth values for each
target
y
ā
R
M
, where
M
is the number of outputs, and
the prediction
y
^
, which induces the ranking function
f
, the
DCG score is
ā
r
=
1
min
(
K
,
M
)
y
f
(
r
)
log
ā”
(
1
+
r
)
and the NDCG score is the DCG score divided by the DCG score obtained for
y
.
Wikipedia entry for Discounted Cumulative Gain
Jarvelin, K., & Kekalainen, J. (2002).
Cumulated gain-based evaluation of IR techniques. ACM Transactions on
Information Systems (TOIS), 20(4), 422-446.
Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013, May).
A theoretical analysis of NDCG ranking measures. In Proceedings of the 26th
Annual Conference on Learning Theory (COLT 2013)
McSherry, F., & Najork, M. (2008, March). Computing information retrieval
performance measures efficiently in the presence of tied scores. In
European conference on information retrieval (pp. 414-421). Springer,
Berlin, Heidelberg.
3.4.6.
Regression metrics
#
The
sklearn.metrics
module implements several loss, score, and utility
functions to measure regression performance. Some of those have been enhanced
to handle the multioutput case:
mean_squared_error
,
mean_absolute_error
,
r2_score
,
explained_variance_score
,
mean_pinball_loss
,
d2_pinball_score
and
d2_absolute_error_score
.
These functions have a
multioutput
keyword argument which specifies the
way the scores or losses for each individual target should be averaged. The
default is
'uniform_average'
, which specifies a uniformly weighted mean
over outputs. If an
ndarray
of shape
(n_outputs,)
is passed, then its
entries are interpreted as weights and an according weighted average is
returned. If
multioutput
is
'raw_values'
, then all unaltered
individual scores or losses will be returned in an array of shape
(n_outputs,)
.
The
r2_score
and
explained_variance_score
accept an additional
value
'variance_weighted'
for the
multioutput
parameter. This option
leads to a weighting of each individual score by the variance of the
corresponding target variable. This setting quantifies the globally captured
unscaled variance. If the target variables are of different scale, then this
score puts more importance on explaining the higher variance variables.
3.4.6.1.
RĀ² score, the coefficient of determination
#
The
r2_score
function computes the
coefficient of
determination
,
usually denoted as
R
2
.
It represents the proportion of variance (of y) that has been explained by the
independent variables in the model. It provides an indication of goodness of
fit and therefore a measure of how well unseen samples are likely to be
predicted by the model, through the proportion of explained variance.
As such variance is dataset dependent,
R
2
may not be meaningfully comparable
across different datasets. Best possible score is 1.0 and it can be negative
(because the model can be arbitrarily worse). A constant model that always
predicts the expected (average) value of y, disregarding the input features,
would get an
R
2
score of 0.0.
Note: when the prediction residuals have zero mean, the
R
2
score and
the
Explained variance score
are identical.
If
y
^
i
is the predicted value of the
i
-th sample
and
y
i
is the corresponding true value for total
n
samples,
the estimated
R
2
is defined as:
R
2
(
y
,
y
^
)
=
1
ā
ā
i
=
1
n
(
y
i
ā
y
^
i
)
2
ā
i
=
1
n
(
y
i
ā
y
ĀÆ
)
2
where
y
ĀÆ
=
1
n
ā
i
=
1
n
y
i
and
ā
i
=
1
n
(
y
i
ā
y
^
i
)
2
=
ā
i
=
1
n
Ļµ
i
2
.
Note that
r2_score
calculates unadjusted
R
2
without correcting for
bias in sample variance of y.
In the particular case where the true target is constant, the
R
2
score is
not finite: it is either
NaN
(perfect predictions) or
-Inf
(imperfect
predictions). Such non-finite scores may prevent correct model optimization
such as grid-search cross-validation to be performed correctly. For this reason
the default behaviour of
r2_score
is to replace them with 1.0 (perfect
predictions) or 0.0 (imperfect predictions). If
force_finite
is set to
False
, this score falls back on the original
R
2
definition.
Here is a small example of usage of the
r2_score
function:
>>>
from
sklearn.metrics
import
r2_score
>>>
y_true
=
[
3
,
-
0.5
,
2
,
7
]
>>>
y_pred
=
[
2.5
,
0.0
,
2
,
8
]
>>>
r2_score
(
y_true
,
y_pred
)
0.948
>>>
y_true
=
[[
0.5
,
1
],
[
-
1
,
1
],
[
7
,
-
6
]]
>>>
y_pred
=
[[
0
,
2
],
[
-
1
,
2
],
[
8
,
-
5
]]
>>>
r2_score
(
y_true
,
y_pred
,
multioutput
=
'variance_weighted'
)
0.938
>>>
y_true
=
[[
0.5
,
1
],
[
-
1
,
1
],
[
7
,
-
6
]]
>>>
y_pred
=
[[
0
,
2
],
[
-
1
,
2
],
[
8
,
-
5
]]
>>>
r2_score
(
y_true
,
y_pred
,
multioutput
=
'uniform_average'
)
0.936
>>>
r2_score
(
y_true
,
y_pred
,
multioutput
=
'raw_values'
)
array([0.965, 0.908])
>>>
r2_score
(
y_true
,
y_pred
,
multioutput
=
[
0.3
,
0.7
])
0.925
>>>
y_true
=
[
-
2
,
-
2
,
-
2
]
>>>
y_pred
=
[
-
2
,
-
2
,
-
2
]
>>>
r2_score
(
y_true
,
y_pred
)
1.0
>>>
r2_score
(
y_true
,
y_pred
,
force_finite
=
False
)
nan
>>>
y_true
=
[
-
2
,
-
2
,
-
2
]
>>>
y_pred
=
[
-
2
,
-
2
,
-
2
+
1e-8
]
>>>
r2_score
(
y_true
,
y_pred
)
0.0
>>>
r2_score
(
y_true
,
y_pred
,
force_finite
=
False
)
-inf
Examples
See
L1-based models for Sparse Signals
for an example of RĀ² score usage to
evaluate Lasso and Elastic Net on sparse signals.
3.4.6.2.
Mean absolute error
#
The
mean_absolute_error
function computes
mean absolute
error
, a risk
metric corresponding to the expected value of the absolute error loss or
l
1
-norm loss.
If
y
^
i
is the predicted value of the
i
-th sample,
and
y
i
is the corresponding true value, then the mean absolute error
(MAE) estimated over
n
samples
is defined as
MAE
(
y
,
y
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
|
y
i
ā
y
^
i
|
.
Here is a small example of usage of the
mean_absolute_error
function:
>>>
from
sklearn.metrics
import
mean_absolute_error
>>>
y_true
=
[
3
,
-
0.5
,
2
,
7
]
>>>
y_pred
=
[
2.5
,
0.0
,
2
,
8
]
>>>
mean_absolute_error
(
y_true
,
y_pred
)
0.5
>>>
y_true
=
[[
0.5
,
1
],
[
-
1
,
1
],
[
7
,
-
6
]]
>>>
y_pred
=
[[
0
,
2
],
[
-
1
,
2
],
[
8
,
-
5
]]
>>>
mean_absolute_error
(
y_true
,
y_pred
)
0.75
>>>
mean_absolute_error
(
y_true
,
y_pred
,
multioutput
=
'raw_values'
)
array([0.5, 1. ])
>>>
mean_absolute_error
(
y_true
,
y_pred
,
multioutput
=
[
0.3
,
0.7
])
0.85
3.4.6.3.
Mean squared error
#
The
mean_squared_error
function computes
mean squared
error
, a risk
metric corresponding to the expected value of the squared (quadratic) error or
loss.
If
y
^
i
is the predicted value of the
i
-th sample,
and
y
i
is the corresponding true value, then the mean squared error
(MSE) estimated over
n
samples
is defined as
MSE
(
y
,
y
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
(
y
i
ā
y
^
i
)
2
.
Here is a small example of usage of the
mean_squared_error
function:
>>>
from
sklearn.metrics
import
mean_squared_error
>>>
y_true
=
[
3
,
-
0.5
,
2
,
7
]
>>>
y_pred
=
[
2.5
,
0.0
,
2
,
8
]
>>>
mean_squared_error
(
y_true
,
y_pred
)
0.375
>>>
y_true
=
[[
0.5
,
1
],
[
-
1
,
1
],
[
7
,
-
6
]]
>>>
y_pred
=
[[
0
,
2
],
[
-
1
,
2
],
[
8
,
-
5
]]
>>>
mean_squared_error
(
y_true
,
y_pred
)
0.7083
Examples
See
Gradient Boosting regression
for an example of mean squared error usage to evaluate gradient boosting regression.
Taking the square root of the MSE, called the root mean squared error (RMSE), is another
common metric that provides a measure in the same units as the target variable. RMSE is
available through the
root_mean_squared_error
function.
3.4.6.4.
Mean squared logarithmic error
#
The
mean_squared_log_error
function computes a risk metric
corresponding to the expected value of the squared logarithmic (quadratic)
error or loss.
If
y
^
i
is the predicted value of the
i
-th sample,
and
y
i
is the corresponding true value, then the mean squared
logarithmic error (MSLE) estimated over
n
samples
is
defined as
MSLE
(
y
,
y
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
(
log
e
ā”
(
1
+
y
i
)
ā
log
e
ā”
(
1
+
y
^
i
)
)
2
.
Where
log
e
ā”
(
x
)
means the natural logarithm of
x
. This metric
is best to use when targets having exponential growth, such as population
counts, average sales of a commodity over a span of years etc. Note that this
metric penalizes an under-predicted estimate greater than an over-predicted
estimate.
Here is a small example of usage of the
mean_squared_log_error
function:
>>>
from
sklearn.metrics
import
mean_squared_log_error
>>>
y_true
=
[
3
,
5
,
2.5
,
7
]
>>>
y_pred
=
[
2.5
,
5
,
4
,
8
]
>>>
mean_squared_log_error
(
y_true
,
y_pred
)
0.0397
>>>
y_true
=
[[
0.5
,
1
],
[
1
,
2
],
[
7
,
6
]]
>>>
y_pred
=
[[
0.5
,
2
],
[
1
,
2.5
],
[
8
,
8
]]
>>>
mean_squared_log_error
(
y_true
,
y_pred
)
0.044
The root mean squared logarithmic error (RMSLE) is available through the
root_mean_squared_log_error
function.
3.4.6.5.
Mean absolute percentage error
#
The
mean_absolute_percentage_error
(MAPE), also known as mean absolute
percentage deviation (MAPD), is an evaluation metric for regression problems.
The idea of this metric is to be sensitive to relative errors. It is for example
not changed by a global scaling of the target variable.
If
y
^
i
is the predicted value of the
i
-th sample
and
y
i
is the corresponding true value, then the mean absolute percentage
error (MAPE) estimated over
n
samples
is defined as
MAPE
(
y
,
y
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
|
y
i
ā
y
^
i
|
max
(
Ļµ
,
|
y
i
|
)
where
Ļµ
is an arbitrary small yet strictly positive number to
avoid undefined results when y is zero.
The
mean_absolute_percentage_error
function supports multioutput.
Here is a small example of usage of the
mean_absolute_percentage_error
function:
>>>
from
sklearn.metrics
import
mean_absolute_percentage_error
>>>
y_true
=
[
1
,
10
,
1e6
]
>>>
y_pred
=
[
0.9
,
15
,
1.2e6
]
>>>
mean_absolute_percentage_error
(
y_true
,
y_pred
)
0.2666
In above example, if we had used
mean_absolute_error
, it would have ignored
the small magnitude values and only reflected the error in prediction of highest
magnitude value. But that problem is resolved in case of MAPE because it calculates
relative percentage error with respect to actual output.
Note
The MAPE formula here does not represent the common āpercentageā definition: the
percentage in the range [0, 100] is converted to a relative value in the range [0,
1] by dividing by 100. Thus, an error of 200% corresponds to a relative error of 2.
The motivation here is to have a range of values that is more consistent with other
error metrics in scikit-learn, such as
accuracy_score
.
To obtain the mean absolute percentage error as per the Wikipedia formula,
multiply the
mean_absolute_percentage_error
computed here by 100.
3.4.6.6.
Median absolute error
#
The
median_absolute_error
is particularly interesting because it is
robust to outliers. The loss is calculated by taking the median of all absolute
differences between the target and the prediction.
If
y
^
i
is the predicted value of the
i
-th sample
and
y
i
is the corresponding true value, then the median absolute error
(MedAE) estimated over
n
samples
is defined as
MedAE
(
y
,
y
^
)
=
median
(
ā£
y
1
ā
y
^
1
ā£
,
ā¦
,
ā£
y
n
ā
y
^
n
ā£
)
.
The
median_absolute_error
does not support multioutput.
Here is a small example of usage of the
median_absolute_error
function:
>>>
from
sklearn.metrics
import
median_absolute_error
>>>
y_true
=
[
3
,
-
0.5
,
2
,
7
]
>>>
y_pred
=
[
2.5
,
0.0
,
2
,
8
]
>>>
median_absolute_error
(
y_true
,
y_pred
)
0.5
3.4.6.7.
Max error
#
The
max_error
function computes the maximum
residual error
, a metric
that captures the worst case error between the predicted value and
the true value. In a perfectly fitted single output regression
model,
max_error
would be
0
on the training set and though this
would be highly unlikely in the real world, this metric shows the
extent of error that the model had when it was fitted.
If
y
^
i
is the predicted value of the
i
-th sample,
and
y
i
is the corresponding true value, then the max error is
defined as
Max Error
(
y
,
y
^
)
=
max
(
|
y
i
ā
y
^
i
|
)
Here is a small example of usage of the
max_error
function:
>>>
from
sklearn.metrics
import
max_error
>>>
y_true
=
[
3
,
2
,
7
,
1
]
>>>
y_pred
=
[
9
,
2
,
7
,
1
]
>>>
max_error
(
y_true
,
y_pred
)
6.0
The
max_error
does not support multioutput.
3.4.6.8.
Explained variance score
#
The
explained_variance_score
computes the
explained variance
regression score
.
If
y
^
is the estimated target output,
y
the corresponding
(correct) target output, and
V
a
r
is
Variance
, the square of the standard deviation,
then the explained variance is estimated as follow:
e
x
p
l
a
i
n
e
d
_
v
a
r
i
a
n
c
e
(
y
,
y
^
)
=
1
ā
V
a
r
{
y
ā
y
^
}
V
a
r
{
y
}
The best possible score is 1.0, lower values are worse.
In the particular case where the true target is constant, the Explained
Variance score is not finite: it is either
NaN
(perfect predictions) or
-Inf
(imperfect predictions). Such non-finite scores may prevent correct
model optimization such as grid-search cross-validation to be performed
correctly. For this reason the default behaviour of
explained_variance_score
is to replace them with 1.0 (perfect
predictions) or 0.0 (imperfect predictions). You can set the
force_finite
parameter to
False
to prevent this fix from happening and fallback on the
original Explained Variance score.
Here is a small example of usage of the
explained_variance_score
function:
>>>
from
sklearn.metrics
import
explained_variance_score
>>>
y_true
=
[
3
,
-
0.5
,
2
,
7
]
>>>
y_pred
=
[
2.5
,
0.0
,
2
,
8
]
>>>
explained_variance_score
(
y_true
,
y_pred
)
0.957
>>>
y_true
=
[[
0.5
,
1
],
[
-
1
,
1
],
[
7
,
-
6
]]
>>>
y_pred
=
[[
0
,
2
],
[
-
1
,
2
],
[
8
,
-
5
]]
>>>
explained_variance_score
(
y_true
,
y_pred
,
multioutput
=
'raw_values'
)
array([0.967, 1. ])
>>>
explained_variance_score
(
y_true
,
y_pred
,
multioutput
=
[
0.3
,
0.7
])
0.990
>>>
y_true
=
[
-
2
,
-
2
,
-
2
]
>>>
y_pred
=
[
-
2
,
-
2
,
-
2
]
>>>
explained_variance_score
(
y_true
,
y_pred
)
1.0
>>>
explained_variance_score
(
y_true
,
y_pred
,
force_finite
=
False
)
nan
>>>
y_true
=
[
-
2
,
-
2
,
-
2
]
>>>
y_pred
=
[
-
2
,
-
2
,
-
2
+
1e-8
]
>>>
explained_variance_score
(
y_true
,
y_pred
)
0.0
>>>
explained_variance_score
(
y_true
,
y_pred
,
force_finite
=
False
)
-inf
3.4.6.9.
Mean Poisson, Gamma, and Tweedie deviances
#
The
mean_tweedie_deviance
function computes the
mean Tweedie
deviance error
with a
power
parameter (
p
). This is a metric that elicits
predicted expectation values of regression targets.
Following special cases exist,
when
power=0
it is equivalent to
mean_squared_error
.
when
power=1
it is equivalent to
mean_poisson_deviance
.
when
power=2
it is equivalent to
mean_gamma_deviance
.
If
y
^
i
is the predicted value of the
i
-th sample,
and
y
i
is the corresponding true value, then the mean Tweedie
deviance error (D) for power
p
, estimated over
n
samples
is defined as
D
(
y
,
y
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
{
(
y
i
ā
y
^
i
)
2
,
forĀ
p
=
0
Ā (Normal)
2
(
y
i
log
ā”
(
y
i
/
y
^
i
)
+
y
^
i
ā
y
i
)
,
forĀ
p
=
1
Ā (Poisson)
2
(
log
ā”
(
y
^
i
/
y
i
)
+
y
i
/
y
^
i
ā
1
)
,
forĀ
p
=
2
Ā (Gamma)
2
(
max
(
y
i
,
0
)
2
ā
p
(
1
ā
p
)
(
2
ā
p
)
ā
y
i
y
^
i
1
ā
p
1
ā
p
+
y
^
i
2
ā
p
2
ā
p
)
,
otherwise
Tweedie deviance is a homogeneous function of degree
2-power
.
Thus, Gamma distribution with
power=2
means that simultaneously scaling
y_true
and
y_pred
has no effect on the deviance. For Poisson
distribution
power=1
the deviance scales linearly, and for Normal
distribution (
power=0
), quadratically. In general, the higher
power
the less weight is given to extreme deviations between true
and predicted targets.
For instance, letās compare the two predictions 1.5 and 150 that are both
50% larger than their corresponding true value.
The mean squared error (
power=0
) is very sensitive to the
prediction difference of the second point,:
>>>
from
sklearn.metrics
import
mean_tweedie_deviance
>>>
mean_tweedie_deviance
([
1.0
],
[
1.5
],
power
=
0
)
0.25
>>>
mean_tweedie_deviance
([
100.
],
[
150.
],
power
=
0
)
2500.0
If we increase
power
to 1,:
>>>
mean_tweedie_deviance
([
1.0
],
[
1.5
],
power
=
1
)
0.189
>>>
mean_tweedie_deviance
([
100.
],
[
150.
],
power
=
1
)
18.9
the difference in errors decreases. Finally, by setting,
power=2
:
>>>
mean_tweedie_deviance
([
1.0
],
[
1.5
],
power
=
2
)
0.144
>>>
mean_tweedie_deviance
([
100.
],
[
150.
],
power
=
2
)
0.144
we would get identical errors. The deviance when
power=2
is thus only
sensitive to relative errors.
3.4.6.10.
Pinball loss
#
The
mean_pinball_loss
function is used to evaluate the predictive
performance of
quantile regression
models.
pinball
(
y
,
y
^
)
=
1
n
samples
ā
i
=
0
n
samples
ā
1
Ī±
max
(
y
i
ā
y
^
i
,
0
)
+
(
1
ā
Ī±
)
max
(
y
^
i
ā
y
i
,
0
)
The value of pinball loss is equivalent to half of
mean_absolute_error
when the quantile
parameter
alpha
is set to 0.5.
Here is a small example of usage of the
mean_pinball_loss
function:
>>>
from
sklearn.metrics
import
mean_pinball_loss
>>>
y_true
=
[
1
,
2
,
3
]
>>>
mean_pinball_loss
(
y_true
,
[
0
,
2
,
3
],
alpha
=
0.1
)
0.033
>>>
mean_pinball_loss
(
y_true
,
[
1
,
2
,
4
],
alpha
=
0.1
)
0.3
>>>
mean_pinball_loss
(
y_true
,
[
0
,
2
,
3
],
alpha
=
0.9
)
0.3
>>>
mean_pinball_loss
(
y_true
,
[
1
,
2
,
4
],
alpha
=
0.9
)
0.033
>>>
mean_pinball_loss
(
y_true
,
y_true
,
alpha
=
0.1
)
0.0
>>>
mean_pinball_loss
(
y_true
,
y_true
,
alpha
=
0.9
)
0.0
It is possible to build a scorer object with a specific choice of
alpha
:
>>>
from
sklearn.metrics
import
make_scorer
>>>
mean_pinball_loss_95p
=
make_scorer
(
mean_pinball_loss
,
alpha
=
0.95
)
Such a scorer can be used to evaluate the generalization performance of a
quantile regressor via cross-validation:
>>>
from
sklearn.datasets
import
make_regression
>>>
from
sklearn.model_selection
import
cross_val_score
>>>
from
sklearn.ensemble
import
GradientBoostingRegressor
>>>
>>>
X
,
y
=
make_regression
(
n_samples
=
100
,
random_state
=
0
)
>>>
estimator
=
GradientBoostingRegressor
(
...
loss
=
"quantile"
,
...
alpha
=
0.95
,
...
random_state
=
0
,
...
)
>>>
cross_val_score
(
estimator
,
X
,
y
,
cv
=
5
,
scoring
=
mean_pinball_loss_95p
)
array([13.6, 9.7, 23.3, 9.5, 10.4])
It is also possible to build scorer objects for hyper-parameter tuning. The
sign of the loss must be switched to ensure that greater means better as
explained in the example linked below.
Examples
See
Prediction Intervals for Gradient Boosting Regression
for an example of using the pinball loss to evaluate and tune the
hyper-parameters of quantile regression models on data with non-symmetric
noise and outliers.
3.4.6.11.
DĀ² score
#
The DĀ² score computes the fraction of deviance explained.
It is a generalization of RĀ², where the squared error is generalized and replaced
by a deviance of choice
dev
(
y
,
y
^
)
(e.g., Tweedie, pinball or mean absolute error). DĀ² is a form of a
skill score
.
It is calculated as
D
2
(
y
,
y
^
)
=
1
ā
dev
(
y
,
y
^
)
dev
(
y
,
y
null
)
.
Where
y
null
is the optimal prediction of an intercept-only model
(e.g., the mean of
y_true
for the Tweedie case, the median for absolute
error and the alpha-quantile for pinball loss).
Like RĀ², the best possible score is 1.0 and it can be negative (because the
model can be arbitrarily worse). A constant model that always predicts
y
null
, disregarding the input features, would get a DĀ² score
of 0.0.
The
d2_tweedie_score
function implements the special case of DĀ²
where
dev
(
y
,
y
^
)
is the Tweedie deviance, see
Mean Poisson, Gamma, and Tweedie deviances
.
It is also known as DĀ² Tweedie and is related to McFaddenās likelihood ratio index.
The argument
power
defines the Tweedie power as for
mean_tweedie_deviance
. Note that for
power=0
,
d2_tweedie_score
equals
r2_score
(for single targets).
A scorer object with a specific choice of
power
can be built by:
>>>
from
sklearn.metrics
import
d2_tweedie_score
,
make_scorer
>>>
d2_tweedie_score_15
=
make_scorer
(
d2_tweedie_score
,
power
=
1.5
)
The
d2_pinball_score
function implements the special case
of DĀ² with the pinball loss, see
Pinball loss
, i.e.:
dev
(
y
,
y
^
)
=
pinball
(
y
,
y
^
)
.
The argument
alpha
defines the slope of the pinball loss as for
mean_pinball_loss
(
Pinball loss
). It determines the
quantile level
alpha
for which the pinball loss and also DĀ²
are optimal. Note that for
alpha=0.5
(the default)
d2_pinball_score
equals
d2_absolute_error_score
.
A scorer object with a specific choice of
alpha
can be built by:
>>>
from
sklearn.metrics
import
d2_pinball_score
,
make_scorer
>>>
d2_pinball_score_08
=
make_scorer
(
d2_pinball_score
,
alpha
=
0.8
)
The
d2_absolute_error_score
function implements the special case of
the
Mean absolute error
:
dev
(
y
,
y
^
)
=
MAE
(
y
,
y
^
)
.
Here are some usage examples of the
d2_absolute_error_score
function:
>>>
from
sklearn.metrics
import
d2_absolute_error_score
>>>
y_true
=
[
3
,
-
0.5
,
2
,
7
]
>>>
y_pred
=
[
2.5
,
0.0
,
2
,
8
]
>>>
d2_absolute_error_score
(
y_true
,
y_pred
)
0.764
>>>
y_true
=
[
1
,
2
,
3
]
>>>
y_pred
=
[
1
,
2
,
3
]
>>>
d2_absolute_error_score
(
y_true
,
y_pred
)
1.0
>>>
y_true
=
[
1
,
2
,
3
]
>>>
y_pred
=
[
2
,
2
,
2
]
>>>
d2_absolute_error_score
(
y_true
,
y_pred
)
0.0
3.4.6.12.
Visual evaluation of regression models
#
Among methods to assess the quality of regression models, scikit-learn provides
the
PredictionErrorDisplay
class. It allows to
visually inspect the prediction errors of a model in two different manners.
The plot on the left shows the actual values vs predicted values. For a
noise-free regression task aiming to predict the (conditional) expectation of
y
, a perfect regression model would display data points on the diagonal
defined by predicted equal to actual values. The further away from this optimal
line, the larger the error of the model. In a more realistic setting with
irreducible noise, that is, when not all the variations of
y
can be explained
by features in
X
, then the best model would lead to a cloud of points densely
arranged around the diagonal.
Note that the above only holds when the predicted values is the expected value
of
y
given
X
. This is typically the case for regression models that
minimize the mean squared error objective function or more generally the
mean Tweedie deviance
for any value of its
āpowerā parameter.
When plotting the predictions of an estimator that predicts a quantile
of
y
given
X
, e.g.
QuantileRegressor
or any other model minimizing the
pinball loss
, a
fraction of the points are either expected to lie above or below the diagonal
depending on the estimated quantile level.
All in all, while intuitive to read, this plot does not really inform us on
what to do to obtain a better model.
The right-hand side plot shows the residuals (i.e. the difference between the
actual and the predicted values) vs. the predicted values.
This plot makes it easier to visualize if the residuals follow and
homoscedastic or heteroschedastic
distribution.
In particular, if the true distribution of
y|X
is Poisson or Gamma
distributed, it is expected that the variance of the residuals of the optimal
model would grow with the predicted value of
E[y|X]
(either linearly for
Poisson or quadratically for Gamma).
When fitting a linear least squares regression model (see
LinearRegression
and
Ridge
), we can use this plot to check
if some of the
model assumptions
are met, in particular that the residuals should be uncorrelated, their
expected value should be null and that their variance should be constant
(homoschedasticity).
If this is not the case, and in particular if the residuals plot show some
banana-shaped structure, this is a hint that the model is likely mis-specified
and that non-linear feature engineering or switching to a non-linear regression
model might be useful.
Refer to the example below to see a model evaluation that makes use of this
display.
Examples
See
Effect of transforming the targets in regression model
for
an example on how to use
PredictionErrorDisplay
to visualize the prediction quality improvement of a regression model
obtained by transforming the target before learning.
3.4.7.
Clustering metrics
#
The
sklearn.metrics
module implements several loss, score, and utility
functions to measure clustering performance. For more information see the
Clustering performance evaluation
section for instance clustering, and
Biclustering evaluation
for biclustering.
3.4.8.
Dummy estimators
#
When doing supervised learning, a simple sanity check consists of comparing
oneās estimator against simple rules of thumb.
DummyClassifier
implements several such simple strategies for classification:
stratified
generates random predictions by respecting the training
set class distribution.
most_frequent
always predicts the most frequent label in the training set.
prior
always predicts the class that maximizes the class prior
(like
most_frequent
) and
predict_proba
returns the class prior.
uniform
generates predictions uniformly at random.
constant
always predicts a constant label that is provided by the user.
A major motivation of this method is F1-scoring, when the positive class
is in the minority.
Note that with all these strategies, the
predict
method completely ignores
the input data!
To illustrate
DummyClassifier
, first letās create an imbalanced
dataset:
>>>
from
sklearn.datasets
import
load_iris
>>>
from
sklearn.model_selection
import
train_test_split
>>>
X
,
y
=
load_iris
(
return_X_y
=
True
)
>>>
y
[
y
!=
1
]
=
-
1
>>>
X_train
,
X_test
,
y_train
,
y_test
=
train_test_split
(
X
,
y
,
random_state
=
0
)
Next, letās compare the accuracy of
SVC
and
most_frequent
:
>>>
from
sklearn.dummy
import
DummyClassifier
>>>
from
sklearn.svm
import
SVC
>>>
clf
=
SVC
(
kernel
=
'linear'
,
C
=
1
)
.
fit
(
X_train
,
y_train
)
>>>
clf
.
score
(
X_test
,
y_test
)
0.63
>>>
clf
=
DummyClassifier
(
strategy
=
'most_frequent'
,
random_state
=
0
)
>>>
clf
.
fit
(
X_train
,
y_train
)
DummyClassifier(random_state=0, strategy='most_frequent')
>>>
clf
.
score
(
X_test
,
y_test
)
0.579
We see that
SVC
doesnāt do much better than a dummy classifier. Now, letās
change the kernel:
>>>
clf
=
SVC
(
kernel
=
'rbf'
,
C
=
1
)
.
fit
(
X_train
,
y_train
)
>>>
clf
.
score
(
X_test
,
y_test
)
0.94
We see that the accuracy was boosted to almost 100%. A cross validation
strategy is recommended for a better estimate of the accuracy, if it
is not too CPU costly. For more information see the
Cross-validation: evaluating estimator performance
section. Moreover if you want to optimize over the parameter space, it is highly
recommended to use an appropriate methodology; see the
Tuning the hyper-parameters of an estimator
section for details.
More generally, when the accuracy of a classifier is too close to random, it
probably means that something went wrong: features are not helpful, a
hyperparameter is not correctly tuned, the classifier is suffering from class
imbalance, etcā¦
DummyRegressor
also implements four simple rules of thumb for regression:
mean
always predicts the mean of the training targets.
median
always predicts the median of the training targets.
quantile
always predicts a user provided quantile of the training targets.
constant
always predicts a constant value that is provided by the user.
In all these strategies, the
predict
method completely ignores
the input data. |
| Markdown | [Skip to main content](https://scikit-learn.org/stable/modules/model_evaluation.html#main-content)
Back to top
[ ](https://scikit-learn.org/stable/index.html)
- [Install](https://scikit-learn.org/stable/install.html)
- [User Guide](https://scikit-learn.org/stable/user_guide.html)
- [API](https://scikit-learn.org/stable/api/index.html)
- [Examples](https://scikit-learn.org/stable/auto_examples/index.html)
- [Community](https://blog.scikit-learn.org/)
- More
- [Getting Started](https://scikit-learn.org/stable/getting_started.html)
- [Release History](https://scikit-learn.org/stable/whats_new.html)
- [Glossary](https://scikit-learn.org/stable/glossary.html)
- [Development](https://scikit-learn.org/dev/developers/index.html)
- [FAQ](https://scikit-learn.org/stable/faq.html)
- [Support](https://scikit-learn.org/stable/support.html)
- [Related Projects](https://scikit-learn.org/stable/related_projects.html)
- [Roadmap](https://scikit-learn.org/stable/roadmap.html)
- [Governance](https://scikit-learn.org/stable/governance.html)
- [About us](https://scikit-learn.org/stable/about.html)
- [GitHub](https://github.com/scikit-learn/scikit-learn)
1\.8.0 (stable)
[1\.9.dev0 (dev)](https://scikit-learn.org/dev/modules/model_evaluation.html)[1\.8.0 (stable)](https://scikit-learn.org/stable/modules/model_evaluation.html)[1\.7.2](https://scikit-learn.org/1.7/modules/model_evaluation.html)[1\.6.1](https://scikit-learn.org/1.6/modules/model_evaluation.html)[1\.5.2](https://scikit-learn.org/1.5/modules/model_evaluation.html)[1\.4.2](https://scikit-learn.org/1.4/modules/model_evaluation.html)[1\.3.2](https://scikit-learn.org/1.3/modules/model_evaluation.html)
- [Install](https://scikit-learn.org/stable/install.html)
- [User Guide](https://scikit-learn.org/stable/user_guide.html)
- [API](https://scikit-learn.org/stable/api/index.html)
- [Examples](https://scikit-learn.org/stable/auto_examples/index.html)
- [Community](https://blog.scikit-learn.org/)
- [Getting Started](https://scikit-learn.org/stable/getting_started.html)
- [Release History](https://scikit-learn.org/stable/whats_new.html)
- [Glossary](https://scikit-learn.org/stable/glossary.html)
- [Development](https://scikit-learn.org/dev/developers/index.html)
- [FAQ](https://scikit-learn.org/stable/faq.html)
- [Support](https://scikit-learn.org/stable/support.html)
- [Related Projects](https://scikit-learn.org/stable/related_projects.html)
- [Roadmap](https://scikit-learn.org/stable/roadmap.html)
- [Governance](https://scikit-learn.org/stable/governance.html)
- [About us](https://scikit-learn.org/stable/about.html)
- [GitHub](https://github.com/scikit-learn/scikit-learn)
1\.8.0 (stable)
[1\.9.dev0 (dev)](https://scikit-learn.org/dev/modules/model_evaluation.html)[1\.8.0 (stable)](https://scikit-learn.org/stable/modules/model_evaluation.html)[1\.7.2](https://scikit-learn.org/1.7/modules/model_evaluation.html)[1\.6.1](https://scikit-learn.org/1.6/modules/model_evaluation.html)[1\.5.2](https://scikit-learn.org/1.5/modules/model_evaluation.html)[1\.4.2](https://scikit-learn.org/1.4/modules/model_evaluation.html)[1\.3.2](https://scikit-learn.org/1.3/modules/model_evaluation.html)
Section Navigation
- [1\. Supervised learning](https://scikit-learn.org/stable/supervised_learning.html)
- [1\.1. Linear Models](https://scikit-learn.org/stable/modules/linear_model.html)
- [1\.2. Linear and Quadratic Discriminant Analysis](https://scikit-learn.org/stable/modules/lda_qda.html)
- [1\.3. Kernel ridge regression](https://scikit-learn.org/stable/modules/kernel_ridge.html)
- [1\.4. Support Vector Machines](https://scikit-learn.org/stable/modules/svm.html)
- [1\.5. Stochastic Gradient Descent](https://scikit-learn.org/stable/modules/sgd.html)
- [1\.6. Nearest Neighbors](https://scikit-learn.org/stable/modules/neighbors.html)
- [1\.7. Gaussian Processes](https://scikit-learn.org/stable/modules/gaussian_process.html)
- [1\.8. Cross decomposition](https://scikit-learn.org/stable/modules/cross_decomposition.html)
- [1\.9. Naive Bayes](https://scikit-learn.org/stable/modules/naive_bayes.html)
- [1\.10. Decision Trees](https://scikit-learn.org/stable/modules/tree.html)
- [1\.11. Ensembles: Gradient boosting, random forests, bagging, voting, stacking](https://scikit-learn.org/stable/modules/ensemble.html)
- [1\.12. Multiclass and multioutput algorithms](https://scikit-learn.org/stable/modules/multiclass.html)
- [1\.13. Feature selection](https://scikit-learn.org/stable/modules/feature_selection.html)
- [1\.14. Semi-supervised learning](https://scikit-learn.org/stable/modules/semi_supervised.html)
- [1\.15. Isotonic regression](https://scikit-learn.org/stable/modules/isotonic.html)
- [1\.16. Probability calibration](https://scikit-learn.org/stable/modules/calibration.html)
- [1\.17. Neural network models (supervised)](https://scikit-learn.org/stable/modules/neural_networks_supervised.html)
- [2\. Unsupervised learning](https://scikit-learn.org/stable/unsupervised_learning.html)
- [2\.1. Gaussian mixture models](https://scikit-learn.org/stable/modules/mixture.html)
- [2\.2. Manifold learning](https://scikit-learn.org/stable/modules/manifold.html)
- [2\.3. Clustering](https://scikit-learn.org/stable/modules/clustering.html)
- [2\.4. Biclustering](https://scikit-learn.org/stable/modules/biclustering.html)
- [2\.5. Decomposing signals in components (matrix factorization problems)](https://scikit-learn.org/stable/modules/decomposition.html)
- [2\.6. Covariance estimation](https://scikit-learn.org/stable/modules/covariance.html)
- [2\.7. Novelty and Outlier Detection](https://scikit-learn.org/stable/modules/outlier_detection.html)
- [2\.8. Density Estimation](https://scikit-learn.org/stable/modules/density.html)
- [2\.9. Neural network models (unsupervised)](https://scikit-learn.org/stable/modules/neural_networks_unsupervised.html)
- [3\. Model selection and evaluation](https://scikit-learn.org/stable/model_selection.html)
- [3\.1. Cross-validation: evaluating estimator performance](https://scikit-learn.org/stable/modules/cross_validation.html)
- [3\.2. Tuning the hyper-parameters of an estimator](https://scikit-learn.org/stable/modules/grid_search.html)
- [3\.3. Tuning the decision threshold for class prediction](https://scikit-learn.org/stable/modules/classification_threshold.html)
- [3\.4. Metrics and scoring: quantifying the quality of predictions](https://scikit-learn.org/stable/modules/model_evaluation.html)
- [3\.5. Validation curves: plotting scores to evaluate models](https://scikit-learn.org/stable/modules/learning_curve.html)
- [4\. Metadata Routing](https://scikit-learn.org/stable/metadata_routing.html)
- [5\. Inspection](https://scikit-learn.org/stable/inspection.html)
- [5\.1. Partial Dependence and Individual Conditional Expectation plots](https://scikit-learn.org/stable/modules/partial_dependence.html)
- [5\.2. Permutation feature importance](https://scikit-learn.org/stable/modules/permutation_importance.html)
- [6\. Visualizations](https://scikit-learn.org/stable/visualizations.html)
- [7\. Dataset transformations](https://scikit-learn.org/stable/data_transforms.html)
- [7\.1. Pipelines and composite estimators](https://scikit-learn.org/stable/modules/compose.html)
- [7\.2. Feature extraction](https://scikit-learn.org/stable/modules/feature_extraction.html)
- [7\.3. Preprocessing data](https://scikit-learn.org/stable/modules/preprocessing.html)
- [7\.4. Imputation of missing values](https://scikit-learn.org/stable/modules/impute.html)
- [7\.5. Unsupervised dimensionality reduction](https://scikit-learn.org/stable/modules/unsupervised_reduction.html)
- [7\.6. Random Projection](https://scikit-learn.org/stable/modules/random_projection.html)
- [7\.7. Kernel Approximation](https://scikit-learn.org/stable/modules/kernel_approximation.html)
- [7\.8. Pairwise metrics, Affinities and Kernels](https://scikit-learn.org/stable/modules/metrics.html)
- [7\.9. Transforming the prediction target (`y`)](https://scikit-learn.org/stable/modules/preprocessing_targets.html)
- [8\. Dataset loading utilities](https://scikit-learn.org/stable/datasets.html)
- [8\.1. Toy datasets](https://scikit-learn.org/stable/datasets/toy_dataset.html)
- [8\.2. Real world datasets](https://scikit-learn.org/stable/datasets/real_world.html)
- [8\.3. Generated datasets](https://scikit-learn.org/stable/datasets/sample_generators.html)
- [8\.4. Loading other datasets](https://scikit-learn.org/stable/datasets/loading_other_datasets.html)
- [9\. Computing with scikit-learn](https://scikit-learn.org/stable/computing.html)
- [9\.1. Strategies to scale computationally: bigger data](https://scikit-learn.org/stable/computing/scaling_strategies.html)
- [9\.2. Computational Performance](https://scikit-learn.org/stable/computing/computational_performance.html)
- [9\.3. Parallelism, resource management, and configuration](https://scikit-learn.org/stable/computing/parallelism.html)
- [10\. Model persistence](https://scikit-learn.org/stable/model_persistence.html)
- [11\. Common pitfalls and recommended practices](https://scikit-learn.org/stable/common_pitfalls.html)
- [12\. Dispatching](https://scikit-learn.org/stable/dispatching.html)
- [12\.1. Array API support (experimental)](https://scikit-learn.org/stable/modules/array_api.html)
- [13\. Choosing the right estimator](https://scikit-learn.org/stable/machine_learning_map.html)
- [14\. External Resources, Videos and Talks](https://scikit-learn.org/stable/presentations.html)
- [User Guide](https://scikit-learn.org/stable/user_guide.html)
- [3\. Model selection and evaluation](https://scikit-learn.org/stable/model_selection.html)
- 3\.4. Metrics and scoring: quantifying the quality of predictions
# 3\.4. Metrics and scoring: quantifying the quality of predictions[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#metrics-and-scoring-quantifying-the-quality-of-predictions "Link to this heading")
## 3\.4.1. Which scoring function should I use?[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#which-scoring-function-should-i-use "Link to this heading")
Before we take a closer look into the details of the many scores and [evaluation metrics](https://scikit-learn.org/stable/glossary.html#term-evaluation-metrics), we want to give some guidance, inspired by statistical decision theory, on the choice of **scoring functions** for **supervised learning**, see [\[Gneiting2009\]](https://scikit-learn.org/stable/modules/model_evaluation.html#gneiting2009):
- *Which scoring function should I use?*
- *Which scoring function is a good one for my task?*
In a nutshell, if the scoring function is given, e.g. in a kaggle competition or in a business context, use that one. If you are free to choose, it starts by considering the ultimate goal and application of the prediction. It is useful to distinguish two steps:
- Predicting
- Decision making
**Predicting:** Usually, the response variable Y is a random variable, in the sense that there is *no deterministic* function Y \= g ( X ) of the features X. Instead, there is a probability distribution F of Y. One can aim to predict the whole distribution, known as *probabilistic prediction*, orāmore the focus of scikit-learnāissue a *point prediction* (or point forecast) by choosing a property or functional of that distribution F. Typical examples are the mean (expected value), the median or a quantile of the response variable Y (conditionally on X).
Once that is settled, use a **strictly consistent** scoring function for that (target) functional, see [\[Gneiting2009\]](https://scikit-learn.org/stable/modules/model_evaluation.html#gneiting2009). This means using a scoring function that is aligned with *measuring the distance between predictions* `y_pred` *and the true target functional using observations of* Y, i.e. `y_true`. For classification **strictly proper scoring rules**, see [Wikipedia entry for Scoring rule](https://en.wikipedia.org/wiki/Scoring_rule) and [\[Gneiting2007\]](https://scikit-learn.org/stable/modules/model_evaluation.html#gneiting2007), coincide with strictly consistent scoring functions. The table further below provides examples. One could say that consistent scoring functions act as *truth serum* in that they guarantee *āthat truth telling \[ā¦\] is an optimal strategy in expectationā* [\[Gneiting2014\]](https://scikit-learn.org/stable/modules/model_evaluation.html#gneiting2014).
Once a strictly consistent scoring function is chosen, it is best used for both: as loss function for model training and as metric/score in model evaluation and model comparison.
Note that for regressors, the prediction is done with [predict](https://scikit-learn.org/stable/glossary.html#term-predict) while for classifiers it is usually [predict\_proba](https://scikit-learn.org/stable/glossary.html#term-predict_proba).
**Decision Making:** The most common decisions are done on binary classification tasks, where the result of [predict\_proba](https://scikit-learn.org/stable/glossary.html#term-predict_proba) is turned into a single outcome, e.g., from the predicted probability of rain a decision is made on how to act (whether to take mitigating measures like an umbrella or not). For classifiers, this is what [predict](https://scikit-learn.org/stable/glossary.html#term-predict) returns. See also [Tuning the decision threshold for class prediction](https://scikit-learn.org/stable/modules/classification_threshold.html#tunedthresholdclassifiercv). There are many scoring functions which measure different aspects of such a decision, most of them are covered with or derived from the [`metrics.confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix "sklearn.metrics.confusion_matrix").
**List of strictly consistent scoring functions:** Here, we list some of the most relevant statistical functionals and corresponding strictly consistent scoring functions for tasks in practice. Note that the list is not complete and that there are more of them. For further criteria on how to select a specific one, see [\[Fissler2022\]](https://scikit-learn.org/stable/modules/model_evaluation.html#fissler2022).
| functional | scoring or loss function | response `y` | prediction |
|---|---|---|---|
| **Classification** | | | |
| mean | [Brier score](https://scikit-learn.org/stable/modules/model_evaluation.html#brier-score-loss) 1 | multi-class | `predict_proba` |
| mean | [log loss](https://scikit-learn.org/stable/modules/model_evaluation.html#log-loss) | multi-class | `predict_proba` |
| mode | [zero-one loss](https://scikit-learn.org/stable/modules/model_evaluation.html#zero-one-loss) 2 | multi-class | `predict`, categorical |
| **Regression** | | | |
| mean | [squared error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-error) 3 | all reals | `predict`, all reals |
| mean | [Poisson deviance](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance) | non-negative | `predict`, strictly positive |
| mean | [Gamma deviance](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance) | strictly positive | `predict`, strictly positive |
| mean | [Tweedie deviance](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance) | depends on `power` | `predict`, depends on `power` |
| median | [absolute error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-error) | all reals | `predict`, all reals |
| quantile | [pinball loss](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss) | all reals | `predict`, all reals |
| mode | no consistent one exists | reals | |
1 The Brier score is just a different name for the squared error in case of classification with one-hot encoded targets.
2 The zero-one loss is only consistent but not strictly consistent for the mode. The zero-one loss is equivalent to one minus the accuracy score, meaning it gives different score values but the same ranking.
3 RĀ² gives the same ranking as squared error.
**Fictitious Example:** Letās make the above arguments more tangible. Consider a setting in network reliability engineering, such as maintaining stable internet or Wi-Fi connections. As provider of the network, you have access to the dataset of log entries of network connections containing network load over time and many interesting features. Your goal is to improve the reliability of the connections. In fact, you promise your customers that on at least 99% of all days there are no connection discontinuities larger than 1 minute. Therefore, you are interested in a prediction of the 99% quantile (of longest connection interruption duration per day) in order to know in advance when to add more bandwidth and thereby satisfy your customers. So the *target functional* is the 99% quantile. From the table above, you choose the pinball loss as scoring function (fair enough, not much choice given), for model training (e.g. `HistGradientBoostingRegressor(loss="quantile", quantile=0.99)`) as well as model evaluation (`mean_pinball_loss(..., alpha=0.99)` - we apologize for the different argument names, `quantile` and `alpha`) be it in grid search for finding hyperparameters or in comparing to other models like `QuantileRegressor(quantile=0.99)`.
References
\[[Gneiting2007](https://scikit-learn.org/stable/modules/model_evaluation.html#id3)\]
T. Gneiting and A. E. Raftery. [Strictly Proper Scoring Rules, Prediction, and Estimation](https://doi.org/10.1198/016214506000001437) In: Journal of the American Statistical Association 102 (2007), pp. 359ā 378. [link to pdf](https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf)
\[Gneiting2009\] ([1](https://scikit-learn.org/stable/modules/model_evaluation.html#id1),[2](https://scikit-learn.org/stable/modules/model_evaluation.html#id2))
T. Gneiting. [Making and Evaluating Point Forecasts](https://arxiv.org/abs/0912.0902) Journal of the American Statistical Association 106 (2009): 746 - 762.
\[[Gneiting2014](https://scikit-learn.org/stable/modules/model_evaluation.html#id4)\]
T. Gneiting and M. Katzfuss. [Probabilistic Forecasting](https://doi.org/10.1146/annurev-statistics-062713-085831). In: Annual Review of Statistics and Its Application 1.1 (2014), pp. 125ā151.
\[[Fissler2022](https://scikit-learn.org/stable/modules/model_evaluation.html#id5)\]
T. Fissler, C. Lorentzen and M. Mayer. [Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice.](https://arxiv.org/abs/2202.12780)
## 3\.4.2. Scoring API overview[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-api-overview "Link to this heading")
There are 3 different APIs for evaluating the quality of a modelās predictions:
- **Estimator score method**: Estimators have a `score` method providing a default evaluation criterion for the problem they are designed to solve. Most commonly this is [accuracy](https://scikit-learn.org/stable/modules/model_evaluation.html#accuracy-score) for classifiers and the [coefficient of determination](https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score) (R 2) for regressors. Details for each estimator can be found in its documentation.
- **Scoring parameter**: Model-evaluation tools that use [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation) (such as [`model_selection.GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV "sklearn.model_selection.GridSearchCV"), [`model_selection.validation_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.validation_curve.html#sklearn.model_selection.validation_curve "sklearn.model_selection.validation_curve") and [`linear_model.LogisticRegressionCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV "sklearn.linear_model.LogisticRegressionCV")) rely on an internal *scoring* strategy. This can be specified using the `scoring` parameter of that tool and is discussed in the section [The scoring parameter: defining model evaluation rules](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter).
- **Metric functions**: The [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") module implements functions assessing prediction error for specific purposes. These metrics are detailed in sections on [Classification metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics), [Multilabel ranking metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#multilabel-ranking-metrics), [Regression metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics) and [Clustering metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#clustering-metrics).
Finally, [Dummy estimators](https://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators) are useful to get a baseline value of those metrics for random predictions.
See also
For āpairwiseā metrics, between *samples* and not estimators or predictions, see the [Pairwise metrics, Affinities and Kernels](https://scikit-learn.org/stable/modules/metrics.html#metrics) section.
## 3\.4.3. The `scoring` parameter: defining model evaluation rules[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#the-scoring-parameter-defining-model-evaluation-rules "Link to this heading")
Model selection and evaluation tools that internally use [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation) (such as [`model_selection.GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV "sklearn.model_selection.GridSearchCV"), [`model_selection.validation_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.validation_curve.html#sklearn.model_selection.validation_curve "sklearn.model_selection.validation_curve") and [`linear_model.LogisticRegressionCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV "sklearn.linear_model.LogisticRegressionCV")) take a `scoring` parameter that controls what metric they apply to the estimators evaluated.
They can be specified in several ways:
- `None`: the estimatorās default evaluation criterion (i.e., the metric used in the estimatorās `score` method) is used.
- [String name](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-string-names): common metrics can be passed via a string name.
- [Callable](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-callable): more complex metrics can be passed via a custom metric callable (e.g., function).
Some tools do also accept multiple metric evaluation. See [Using multiple metric evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#multimetric-scoring) for details.
### 3\.4.3.1. String name scorers[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#string-name-scorers "Link to this heading")
For the most common use cases, you can designate a scorer object with the `scoring` parameter via a string name; the table below shows all possible values. All scorer objects follow the convention that **higher return values are better than lower return values**. Thus metrics which measure the distance between the model and the data, like [`metrics.mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error"), are available as āneg\_mean\_squared\_errorā which return the negated value of the metric.
| Scoring string name | Function | Comment |
|---|---|---|
| **Classification** | | |
| āaccuracyā | [`metrics.accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score") | |
| ābalanced\_accuracyā | [`metrics.balanced_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html#sklearn.metrics.balanced_accuracy_score "sklearn.metrics.balanced_accuracy_score") | |
| ātop\_k\_accuracyā | [`metrics.top_k_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html#sklearn.metrics.top_k_accuracy_score "sklearn.metrics.top_k_accuracy_score") | |
| āaverage\_precisionā | [`metrics.average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") | |
| āneg\_brier\_scoreā | [`metrics.brier_score_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss "sklearn.metrics.brier_score_loss") | requires `predict_proba` support |
| āf1ā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | for binary targets |
| āf1\_microā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | micro-averaged |
| āf1\_macroā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | macro-averaged |
| āf1\_weightedā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | weighted average |
| āf1\_samplesā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | by multilabel sample |
| āneg\_log\_lossā | [`metrics.log_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html#sklearn.metrics.log_loss "sklearn.metrics.log_loss") | requires `predict_proba` support |
| āprecisionā etc. | [`metrics.precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score") | suffixes apply as with āf1ā |
| ārecallā etc. | [`metrics.recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score") | suffixes apply as with āf1ā |
| ājaccardā etc. | [`metrics.jaccard_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html#sklearn.metrics.jaccard_score "sklearn.metrics.jaccard_score") | suffixes apply as with āf1ā |
| āroc\_aucā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| āroc\_auc\_ovrā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| āroc\_auc\_ovoā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| āroc\_auc\_ovr\_weightedā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| āroc\_auc\_ovo\_weightedā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| ād2\_log\_loss\_scoreā | [`metrics.d2_log_loss_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_log_loss_score.html#sklearn.metrics.d2_log_loss_score "sklearn.metrics.d2_log_loss_score") | requires `predict_proba` support |
| ād2\_brier\_scoreā | [`metrics.d2_brier_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_brier_score.html#sklearn.metrics.d2_brier_score "sklearn.metrics.d2_brier_score") | requires `predict_proba` support |
| **Clustering** | | |
| āadjusted\_mutual\_info\_scoreā | [`metrics.adjusted_mutual_info_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_mutual_info_score.html#sklearn.metrics.adjusted_mutual_info_score "sklearn.metrics.adjusted_mutual_info_score") | |
| āadjusted\_rand\_scoreā | [`metrics.adjusted_rand_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html#sklearn.metrics.adjusted_rand_score "sklearn.metrics.adjusted_rand_score") | |
| ācompleteness\_scoreā | [`metrics.completeness_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.completeness_score.html#sklearn.metrics.completeness_score "sklearn.metrics.completeness_score") | |
| āfowlkes\_mallows\_scoreā | [`metrics.fowlkes_mallows_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fowlkes_mallows_score.html#sklearn.metrics.fowlkes_mallows_score "sklearn.metrics.fowlkes_mallows_score") | |
| āhomogeneity\_scoreā | [`metrics.homogeneity_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.homogeneity_score.html#sklearn.metrics.homogeneity_score "sklearn.metrics.homogeneity_score") | |
| āmutual\_info\_scoreā | [`metrics.mutual_info_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mutual_info_score.html#sklearn.metrics.mutual_info_score "sklearn.metrics.mutual_info_score") | |
| ānormalized\_mutual\_info\_scoreā | [`metrics.normalized_mutual_info_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.normalized_mutual_info_score.html#sklearn.metrics.normalized_mutual_info_score "sklearn.metrics.normalized_mutual_info_score") | |
| ārand\_scoreā | [`metrics.rand_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.rand_score.html#sklearn.metrics.rand_score "sklearn.metrics.rand_score") | |
| āv\_measure\_scoreā | [`metrics.v_measure_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.v_measure_score.html#sklearn.metrics.v_measure_score "sklearn.metrics.v_measure_score") | |
| **Regression** | | |
| āexplained\_varianceā | [`metrics.explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") | |
| āneg\_max\_errorā | [`metrics.max_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.max_error.html#sklearn.metrics.max_error "sklearn.metrics.max_error") | |
| āneg\_mean\_absolute\_errorā | [`metrics.mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error") | |
| āneg\_mean\_squared\_errorā | [`metrics.mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error") | |
| āneg\_root\_mean\_squared\_errorā | [`metrics.root_mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.root_mean_squared_error.html#sklearn.metrics.root_mean_squared_error "sklearn.metrics.root_mean_squared_error") | |
| āneg\_mean\_squared\_log\_errorā | [`metrics.mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_log_error.html#sklearn.metrics.mean_squared_log_error "sklearn.metrics.mean_squared_log_error") | |
| āneg\_root\_mean\_squared\_log\_errorā | [`metrics.root_mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.root_mean_squared_log_error.html#sklearn.metrics.root_mean_squared_log_error "sklearn.metrics.root_mean_squared_log_error") | |
| āneg\_median\_absolute\_errorā | [`metrics.median_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.median_absolute_error.html#sklearn.metrics.median_absolute_error "sklearn.metrics.median_absolute_error") | |
| ār2ā | [`metrics.r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") | |
| āneg\_mean\_poisson\_devianceā | [`metrics.mean_poisson_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_poisson_deviance.html#sklearn.metrics.mean_poisson_deviance "sklearn.metrics.mean_poisson_deviance") | |
| āneg\_mean\_gamma\_devianceā | [`metrics.mean_gamma_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_gamma_deviance.html#sklearn.metrics.mean_gamma_deviance "sklearn.metrics.mean_gamma_deviance") | |
| āneg\_mean\_absolute\_percentage\_errorā | [`metrics.mean_absolute_percentage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html#sklearn.metrics.mean_absolute_percentage_error "sklearn.metrics.mean_absolute_percentage_error") | |
| ād2\_absolute\_error\_scoreā | [`metrics.d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score") | |
Usage examples:
```
>>> from sklearn import svm, datasets
>>> from sklearn.model_selection import cross_val_score
>>> X, y = datasets.load_iris(return_X_y=True)
>>> clf = svm.SVC(random_state=0)
>>> cross_val_score(clf, X, y, cv=5, scoring='recall_macro')
array([0.96, 0.96, 0.96, 0.93, 1. ])
```
Note
If a wrong scoring name is passed, an `InvalidParameterError` is raised. You can retrieve the names of all available scorers by calling [`get_scorer_names`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.get_scorer_names.html#sklearn.metrics.get_scorer_names "sklearn.metrics.get_scorer_names").
### 3\.4.3.2. Callable scorers[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#callable-scorers "Link to this heading")
For more complex use cases and more flexibility, you can pass a callable to the `scoring` parameter. This can be done by:
- [Adapting predefined metrics via make\_scorer](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-adapt-metric)
- [Creating a custom scorer object](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-custom) (most flexible)
#### 3\.4.3.2.1. Adapting predefined metrics via `make_scorer`[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#adapting-predefined-metrics-via-make-scorer "Link to this heading")
The following metric functions are not implemented as named scorers, sometimes because they require additional parameters, such as [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score"). They cannot be passed to the `scoring` parameters; instead their callable needs to be passed to [`make_scorer`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer "sklearn.metrics.make_scorer") together with the value of the user-settable parameters.
| Function | Parameter | Example usage |
|---|---|---|
| **Classification** | | |
| `metrics.fbeta_score` | `beta` | `make_scorer(fbeta_score, beta=2)` |
| **Regression** | | |
| `metrics.mean_tweedie_deviance` | `power` | `make_scorer(mean_tweedie_deviance, power=1.5)` |
| `metrics.mean_pinball_loss` | `alpha` | `make_scorer(mean_pinball_loss, alpha=0.95)` |
| `metrics.d2_tweedie_score` | `power` | `make_scorer(d2_tweedie_score, power=1.5)` |
| `metrics.d2_pinball_score` | `alpha` | `make_scorer(d2_pinball_score, alpha=0.95)` |
One typical use case is to wrap an existing metric function from the library with non-default values for its parameters, such as the `beta` parameter for the [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score") function:
```
>>> from sklearn.metrics import fbeta_score, make_scorer
>>> ftwo_scorer = make_scorer(fbeta_score, beta=2)
>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.svm import LinearSVC
>>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]},
... scoring=ftwo_scorer, cv=5)
```
The module [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") also exposes a set of simple functions measuring a prediction error given ground truth and prediction:
- functions ending with `_score` return a value to maximize, the higher the better.
- functions ending with `_error`, `_loss`, or `_deviance` return a value to minimize, the lower the better. When converting into a scorer object using [`make_scorer`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer "sklearn.metrics.make_scorer"), set the `greater_is_better` parameter to `False` (`True` by default; see the parameter description below).
#### 3\.4.3.2.2. Creating a custom scorer object[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#creating-a-custom-scorer-object "Link to this heading")
You can create your own custom scorer object using [`make_scorer`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer "sklearn.metrics.make_scorer").
Custom scorer objects using `make_scorer`[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#custom-scorer-objects-using-make_scorer "Link to this dropdown")
You can build a completely custom scorer object from a simple python function using [`make_scorer`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer "sklearn.metrics.make_scorer"), which can take several parameters:
- the python function you want to use (`my_custom_loss_func` in the example below)
- whether the python function returns a score (`greater_is_better=True`, the default) or a loss (`greater_is_better=False`). If a loss, the output of the python function is negated by the scorer object, conforming to the cross validation convention that scorers return higher values for better models.
- for classification metrics only: whether the python function you provided requires continuous decision certainties. If the scoring function only accepts probability estimates (e.g. `metrics.log_loss`), then one needs to set the parameter `response_method="predict_proba"`. Some scoring functions do not necessarily require probability estimates but rather non-thresholded decision values (e.g. `metrics.roc_auc_score`). In this case, one can provide a list (e.g., `response_method=["decision_function", "predict_proba"]`), and scorer will use the first available method, in the order given in the list, to compute the scores.
- any additional parameters of the scoring function, such as `beta` or `labels`.
Here is an example of building custom scorers, and of using the `greater_is_better` parameter:
```
>>> import numpy as np
>>> def my_custom_loss_func(y_true, y_pred):
... diff = np.abs(y_true - y_pred).max()
... return float(np.log1p(diff))
...
>>> # score will negate the return value of my_custom_loss_func,
>>> # which will be np.log(2), 0.693, given the values for X
>>> # and y defined below.
>>> score = make_scorer(my_custom_loss_func, greater_is_better=False)
>>> X = [[1], [1]]
>>> y = [0, 1]
>>> from sklearn.dummy import DummyClassifier
>>> clf = DummyClassifier(strategy='most_frequent', random_state=0)
>>> clf = clf.fit(X, y)
>>> my_custom_loss_func(y, clf.predict(X))
0.69
>>> score(clf, X, y)
-0.69
```
Using custom scorers in functions where n\_jobs \> 1[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#using-custom-scorers-in-functions-where-n_jobs->-1 "Link to this dropdown")
While defining the custom scoring function alongside the calling function should work out of the box with the default joblib backend (loky), importing it from another module will be a more robust approach and work independently of the joblib backend.
For example, to use `n_jobs` greater than 1 in the example below, `custom_scoring_function` function is saved in a user-created module (`custom_scorer_module.py`) and imported:
```
>>> from custom_scorer_module import custom_scoring_function
>>> cross_val_score(model,
... X_train,
... y_train,
... scoring=make_scorer(custom_scoring_function, greater_is_better=False),
... cv=5,
... n_jobs=-1)
```
### 3\.4.3.3. Using multiple metric evaluation[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#using-multiple-metric-evaluation "Link to this heading")
Scikit-learn also permits evaluation of multiple metrics in `GridSearchCV`, `RandomizedSearchCV` and `cross_validate`.
There are three ways to specify multiple scoring metrics for the `scoring` parameter:
- As an iterable of string metrics:
```
>>> scoring = ['accuracy', 'precision']
```
- As a `dict` mapping the scorer name to the scoring function:
```
>>> from sklearn.metrics import accuracy_score
>>> from sklearn.metrics import make_scorer
>>> scoring = {'accuracy': make_scorer(accuracy_score),
... 'prec': 'precision'}
```
Note that the dict values can either be scorer functions or one of the predefined metric strings.
- As a callable that returns a dictionary of scores:
```
>>> from sklearn.model_selection import cross_validate
>>> from sklearn.metrics import confusion_matrix
>>> # A sample toy binary classification dataset
>>> X, y = datasets.make_classification(n_classes=2, random_state=0)
>>> svm = LinearSVC(random_state=0)
>>> def confusion_matrix_scorer(clf, X, y):
... y_pred = clf.predict(X)
... cm = confusion_matrix(y, y_pred)
... return {'tn': cm[0, 0], 'fp': cm[0, 1],
... 'fn': cm[1, 0], 'tp': cm[1, 1]}
>>> cv_results = cross_validate(svm, X, y, cv=5,
... scoring=confusion_matrix_scorer)
>>> # Getting the test set true positive scores
>>> print(cv_results['test_tp'])
[10 9 8 7 8]
>>> # Getting the test set false negative scores
>>> print(cv_results['test_fn'])
[0 1 2 3 2]
```
## 3\.4.4. Classification metrics[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics "Link to this heading")
The [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") module implements several loss, score, and utility functions to measure classification performance. Some metrics might require probability estimates of the positive class, confidence values, or binary decisions values. Most implementations allow each sample to provide a weighted contribution to the overall score, through the `sample_weight` parameter.
Some of these are restricted to the binary classification case:
| | |
|---|---|
| [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve "sklearn.metrics.precision_recall_curve")(y\_true, y\_score, \*\[, ...\]) | Compute precision-recall pairs for different probability thresholds. |
| [`roc_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve "sklearn.metrics.roc_curve")(y\_true, y\_score, \*\[, pos\_label, ...\]) | Compute Receiver operating characteristic (ROC). |
| [`class_likelihood_ratios`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.class_likelihood_ratios.html#sklearn.metrics.class_likelihood_ratios "sklearn.metrics.class_likelihood_ratios")(y\_true, y\_pred, \*\[, ...\]) | Compute binary classification positive and negative likelihood ratios. |
| [`det_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.det_curve.html#sklearn.metrics.det_curve "sklearn.metrics.det_curve")(y\_true, y\_score\[, pos\_label, ...\]) | Compute Detection Error Tradeoff (DET) for different probability thresholds. |
| [`confusion_matrix_at_thresholds`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix_at_thresholds.html#sklearn.metrics.confusion_matrix_at_thresholds "sklearn.metrics.confusion_matrix_at_thresholds")(y\_true, y\_score) | Calculate [binary](https://scikit-learn.org/stable/glossary.html#term-binary) confusion matrix terms per classification threshold. |
Others also work in the multiclass case:
| | |
|---|---|
| [`balanced_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html#sklearn.metrics.balanced_accuracy_score "sklearn.metrics.balanced_accuracy_score")(y\_true, y\_pred, \*\[, ...\]) | Compute the balanced accuracy. |
| [`cohen_kappa_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html#sklearn.metrics.cohen_kappa_score "sklearn.metrics.cohen_kappa_score")(y1, y2, \*\[, labels, ...\]) | Compute Cohen's kappa: a statistic that measures inter-annotator agreement. |
| [`confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix "sklearn.metrics.confusion_matrix")(y\_true, y\_pred, \*\[, ...\]) | Compute confusion matrix to evaluate the accuracy of a classification. |
| [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss")(y\_true, pred\_decision, \*\[, ...\]) | Average hinge loss (non-regularized). |
| [`matthews_corrcoef`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html#sklearn.metrics.matthews_corrcoef "sklearn.metrics.matthews_corrcoef")(y\_true, y\_pred, \*\[, ...\]) | Compute the Matthews correlation coefficient (MCC). |
| [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score")(y\_true, y\_score, \*\[, average, ...\]) | Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. |
| [`top_k_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html#sklearn.metrics.top_k_accuracy_score "sklearn.metrics.top_k_accuracy_score")(y\_true, y\_score, \*\[, ...\]) | Top-k Accuracy classification score. |
Some also work in the multilabel case:
| | |
|---|---|
| [`accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score")(y\_true, y\_pred, \*\[, ...\]) | Accuracy classification score. |
| [`classification_report`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html#sklearn.metrics.classification_report "sklearn.metrics.classification_report")(y\_true, y\_pred, \*\[, ...\]) | Build a text report showing the main classification metrics. |
| [`f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the F1 score, also known as balanced F-score or F-measure. |
| [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score")(y\_true, y\_pred, \*, beta\[, ...\]) | Compute the F-beta score. |
| [`hamming_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hamming_loss.html#sklearn.metrics.hamming_loss "sklearn.metrics.hamming_loss")(y\_true, y\_pred, \*\[, sample\_weight\]) | Compute the average Hamming loss. |
| [`jaccard_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html#sklearn.metrics.jaccard_score "sklearn.metrics.jaccard_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Jaccard similarity coefficient score. |
| [`log_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html#sklearn.metrics.log_loss "sklearn.metrics.log_loss")(y\_true, y\_pred, \*\[, normalize, ...\]) | Log loss, aka logistic loss or cross-entropy loss. |
| [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix")(y\_true, y\_pred, \*) | Compute a confusion matrix for each class or sample. |
| [`precision_recall_fscore_support`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html#sklearn.metrics.precision_recall_fscore_support "sklearn.metrics.precision_recall_fscore_support")(y\_true, ...) | Compute precision, recall, F-measure and support for each class. |
| [`precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the precision. |
| [`recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the recall. |
| [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score")(y\_true, y\_score, \*\[, average, ...\]) | Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. |
| [`zero_one_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.zero_one_loss.html#sklearn.metrics.zero_one_loss "sklearn.metrics.zero_one_loss")(y\_true, y\_pred, \*\[, ...\]) | Zero-one classification loss. |
| [`d2_log_loss_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_log_loss_score.html#sklearn.metrics.d2_log_loss_score "sklearn.metrics.d2_log_loss_score")(y\_true, y\_pred, \*\[, ...\]) | D 2 score function, fraction of log loss explained. |
And some work with binary and multilabel (but not multiclass) problems:
| | |
|---|---|
| [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score")(y\_true, y\_score, \*) | Compute average precision (AP) from prediction scores. |
In the following sub-sections, we will describe each of those functions, preceded by some notes on common API and metric definition.
### 3\.4.4.1. From binary to multiclass and multilabel[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#from-binary-to-multiclass-and-multilabel "Link to this heading")
Some metrics are essentially defined for binary classification tasks (e.g. [`f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score"), [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score")). In these cases, by default only the positive label is evaluated, assuming by default that the positive class is labelled `1` (though this may be configurable through the `pos_label` parameter).
In extending a binary metric to multiclass or multilabel problems, the data is treated as a collection of binary problems, one for each class. There are then a number of ways to average binary metric calculations across the set of classes, each of which may be useful in some scenario. Where available, you should select among these using the `average` parameter.
- `"macro"` simply calculates the mean of the binary metrics, giving equal weight to each class. In problems where infrequent classes are nonetheless important, macro-averaging may be a means of highlighting their performance. On the other hand, the assumption that all classes are equally important is often untrue, such that macro-averaging will over-emphasize the typically low performance on an infrequent class.
- `"weighted"` accounts for class imbalance by computing the average of binary metrics in which each classās score is weighted by its presence in the true data sample.
- `"micro"` gives each sample-class pair an equal contribution to the overall metric (except as a result of sample-weight). Rather than summing the metric per class, this sums the dividends and divisors that make up the per-class metrics to calculate an overall quotient. Micro-averaging may be preferred in multilabel settings, including multiclass classification where a majority class is to be ignored.
- `"samples"` applies only to multilabel problems. It does not calculate a per-class measure, instead calculating the metric over the true and predicted classes for each sample in the evaluation data, and returning their (`sample_weight`\-weighted) average.
- Selecting `average=None` will return an array with the score for each class.
While multiclass data is provided to the metric, like binary targets, as an array of class labels, multilabel data is specified as an indicator matrix, in which cell `[i, j]` has value 1 if sample `i` has label `j` and value 0 otherwise.
### 3\.4.4.2. Accuracy score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#accuracy-score "Link to this heading")
The [`accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score") function computes the [accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision), either the fraction (default) or the count (normalize=False) of correct predictions.
In multilabel classification, the function returns the subset accuracy. If the entire set of predicted labels for a sample strictly match with the true set of labels, then the subset accuracy is 1.0; otherwise it is 0.0.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value, then the fraction of correct predictions over n samples is defined as
accuracy
(
y
,
y
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
1
(
y
^
i
\=
y
i
)
where 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function).
```
>>> import numpy as np
>>> from sklearn.metrics import accuracy_score
>>> y_pred = [0, 2, 1, 3]
>>> y_true = [0, 1, 2, 3]
>>> accuracy_score(y_true, y_pred)
0.5
>>> accuracy_score(y_true, y_pred, normalize=False)
2.0
```
In the multilabel case with binary label indicators:
```
>>> accuracy_score(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5
```
Examples
- See [Test with permutations the significance of a classification score](https://scikit-learn.org/stable/auto_examples/model_selection/plot_permutation_tests_for_classification.html#sphx-glr-auto-examples-model-selection-plot-permutation-tests-for-classification-py) for an example of accuracy score usage using permutations of the dataset.
### 3\.4.4.3. Top-k accuracy score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#top-k-accuracy-score "Link to this heading")
The [`top_k_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html#sklearn.metrics.top_k_accuracy_score "sklearn.metrics.top_k_accuracy_score") function is a generalization of [`accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score"). The difference is that a prediction is considered correct as long as the true label is associated with one of the `k` highest predicted scores. [`accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score") is the special case of `k = 1`.
The function covers the binary and multiclass classification cases but not the multilabel case.
If f ^ i , j is the predicted class for the i\-th sample corresponding to the j\-th largest predicted score and y i is the corresponding true value, then the fraction of correct predictions over n samples is defined as
top-k accuracy
(
y
,
f
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
ā
j
\=
1
k
1
(
f
^
i
,
j
\=
y
i
)
where k is the number of guesses allowed and 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function).
```
>>> import numpy as np
>>> from sklearn.metrics import top_k_accuracy_score
>>> y_true = np.array([0, 1, 2, 2])
>>> y_score = np.array([[0.5, 0.2, 0.2],
... [0.3, 0.4, 0.2],
... [0.2, 0.4, 0.3],
... [0.7, 0.2, 0.1]])
>>> top_k_accuracy_score(y_true, y_score, k=2)
0.75
>>> # Not normalizing gives the number of "correctly" classified samples
>>> top_k_accuracy_score(y_true, y_score, k=2, normalize=False)
3.0
```
### 3\.4.4.4. Balanced accuracy score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#balanced-accuracy-score "Link to this heading")
The [`balanced_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html#sklearn.metrics.balanced_accuracy_score "sklearn.metrics.balanced_accuracy_score") function computes the [balanced accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision), which avoids inflated performance estimates on imbalanced datasets. It is the macro-average of recall scores per class or, equivalently, raw accuracy where each sample is weighted according to the inverse prevalence of its true class. Thus for balanced datasets, the score is equal to accuracy.
In the binary case, balanced accuracy is equal to the arithmetic mean of [sensitivity](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) (true positive rate) and [specificity](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) (true negative rate), or the area under the ROC curve with binary predictions rather than scores:
balanced-accuracy
\=
1
2
(
T
P
T
P
\+
F
N
\+
T
N
T
N
\+
F
P
)
If the classifier performs equally well on either class, this term reduces to the conventional accuracy (i.e., the number of correct predictions divided by the total number of predictions).
In contrast, if the conventional accuracy is above chance only because the classifier takes advantage of an imbalanced test set, then the balanced accuracy, as appropriate, will drop to 1 n \_ c l a s s e s.
The score ranges from 0 to 1, or when `adjusted=True` is used, it is rescaled to the range 1 1 ā n \_ c l a s s e s to 1, inclusive, with performance at random scoring 0.
If y i is the true value of the i\-th sample, and w i is the corresponding sample weight, then we adjust the sample weight to:
w
^
i
\=
w
i
ā
j
1
(
y
j
\=
y
i
)
w
j
where 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function). Given predicted y ^ i for sample i, balanced accuracy is defined as:
balanced-accuracy
(
y
,
y
^
,
w
)
\=
1
ā
w
^
i
ā
i
1
(
y
^
i
\=
y
i
)
w
^
i
With `adjusted=True`, balanced accuracy reports the relative increase from balanced-accuracy ( y , 0 , w ) \= 1 n \_ c l a s s e s. In the binary case, this is also known as [Youdenās J statistic](https://en.wikipedia.org/wiki/Youden%27s_J_statistic), or *informedness*.
Note
The multiclass definition here seems the most reasonable extension of the metric used in binary classification, though there is no certain consensus in the literature:
- Our definition: [\[Mosley2013\]](https://scikit-learn.org/stable/modules/model_evaluation.html#mosley2013), [\[Kelleher2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#kelleher2015) and [\[Guyon2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#guyon2015), where [\[Guyon2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#guyon2015) adopt the adjusted version to ensure that random predictions have a score of 0 and perfect predictions have a score of 1.
- Class balanced accuracy as described in [\[Mosley2013\]](https://scikit-learn.org/stable/modules/model_evaluation.html#mosley2013): the minimum between the precision and the recall for each class is computed. Those values are then averaged over the total number of classes to get the balanced accuracy.
- Balanced Accuracy as described in [\[Urbanowicz2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#urbanowicz2015): the average of sensitivity and specificity is computed for each class and then averaged over total number of classes.
References
\[Guyon2015\] ([1](https://scikit-learn.org/stable/modules/model_evaluation.html#id15),[2](https://scikit-learn.org/stable/modules/model_evaluation.html#id16))
I. Guyon, K. Bennett, G. Cawley, H.J. Escalante, S. Escalera, T.K. Ho, N. MaciĆ , B. Ray, M. Saeed, A.R. Statnikov, E. Viegas, [Design of the 2015 ChaLearn AutoML Challenge](https://ieeexplore.ieee.org/document/7280767), IJCNN 2015.
\[Mosley2013\] ([1](https://scikit-learn.org/stable/modules/model_evaluation.html#id13),[2](https://scikit-learn.org/stable/modules/model_evaluation.html#id17))
L. Mosley, [A balanced approach to the multi-class imbalance problem](https://lib.dr.iastate.edu/etd/13537/), IJCV 2010.
\[[Kelleher2015](https://scikit-learn.org/stable/modules/model_evaluation.html#id14)\]
John. D. Kelleher, Brian Mac Namee, Aoife DāArcy, [Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies](https://mitpress.mit.edu/books/fundamentals-machine-learning-predictive-data-analytics), 2015.
\[[Urbanowicz2015](https://scikit-learn.org/stable/modules/model_evaluation.html#id18)\]
Urbanowicz R.J., Moore, J.H. [ExSTraCS 2.0: description and evaluation of a scalable learning classifier system](https://doi.org/10.1007/s12065-015-0128-8), Evol. Intel. (2015) 8: 89.
### 3\.4.4.5. Cohenās kappa[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#cohen-s-kappa "Link to this heading")
The function [`cohen_kappa_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html#sklearn.metrics.cohen_kappa_score "sklearn.metrics.cohen_kappa_score") computes [Cohenās kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa) statistic. This measure is intended to compare labelings by different human annotators, not a classifier versus a ground truth.
The kappa score is a number between -1 and 1. Scores above .8 are generally considered good agreement; zero or lower means no agreement (practically random labels).
Kappa scores can be computed for binary or multiclass problems, but not for multilabel problems (except by manually computing a per-label score) and not for more than two annotators.
```
>>> from sklearn.metrics import cohen_kappa_score
>>> labeling1 = [2, 0, 2, 2, 0, 1]
>>> labeling2 = [0, 0, 2, 2, 0, 2]
>>> cohen_kappa_score(labeling1, labeling2)
0.4285714285714286
```
### 3\.4.4.6. Confusion matrix[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix "Link to this heading")
The [`confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix "sklearn.metrics.confusion_matrix") function evaluates classification accuracy by computing the [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) with each row corresponding to the true class (Wikipedia and other references may use different convention for axes).
By definition, entry i , j in a confusion matrix is the number of observations actually in group i, but predicted to be in group j. Here is an example:
```
>>> from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
[0, 0, 1],
[1, 0, 2]])
```
[`ConfusionMatrixDisplay`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay "sklearn.metrics.ConfusionMatrixDisplay") can be used to visually represent a confusion matrix as shown in the [Evaluate the performance of a classifier with Confusion Matrix](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py) example, which creates the following figure:
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html)
The parameter `normalize` allows to report ratios instead of counts. The confusion matrix can be normalized in 3 different ways: `'pred'`, `'true'`, and `'all'` which will divide the counts by the sum of each columns, rows, or the entire matrix, respectively.
```
>>> y_true = [0, 0, 0, 1, 1, 1, 1, 1]
>>> y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
>>> confusion_matrix(y_true, y_pred, normalize='all')
array([[0.25 , 0.125],
[0.25 , 0.375]])
```
For binary problems, we can get counts of true negatives, false positives, false negatives and true positives as follows:
```
>>> y_true = [0, 0, 0, 1, 1, 1, 1, 1]
>>> y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
>>> tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel().tolist()
>>> tn, fp, fn, tp
(2, 1, 2, 3)
```
With [`confusion_matrix_at_thresholds`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix_at_thresholds.html#sklearn.metrics.confusion_matrix_at_thresholds "sklearn.metrics.confusion_matrix_at_thresholds") we can get true negatives, false positives, false negatives and true positives for different thresholds:
```
>>> from sklearn.metrics import confusion_matrix_at_thresholds
>>> y_true = np.array([0., 0., 1., 1.])
>>> y_score = np.array([0.1, 0.4, 0.35, 0.8])
>>> tns, fps, fns, tps, thresholds = confusion_matrix_at_thresholds(y_true, y_score)
>>> tns
array([2., 1., 1., 0.])
>>> fps
array([0., 1., 1., 2.])
>>> fns
array([1., 1., 0., 0.])
>>> tps
array([1., 1., 2., 2.])
>>> thresholds
array([0.8, 0.4, 0.35, 0.1])
```
Note that the thresholds consist of distinct `y_score` values, in decreasing order.
Examples
- See [Evaluate the performance of a classifier with Confusion Matrix](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py) for an example of using a confusion matrix to evaluate classifier output quality.
- See [Recognizing hand-written digits](https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html#sphx-glr-auto-examples-classification-plot-digits-classification-py) for an example of using a confusion matrix to classify hand-written digits.
- See [Classification of text documents using sparse features](https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#sphx-glr-auto-examples-text-plot-document-classification-20newsgroups-py) for an example of using a confusion matrix to classify text documents.
### 3\.4.4.7. Classification report[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-report "Link to this heading")
The [`classification_report`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html#sklearn.metrics.classification_report "sklearn.metrics.classification_report") function builds a text report showing the main classification metrics. Here is a small example with custom `target_names` and inferred labels:
```
>>> from sklearn.metrics import classification_report
>>> y_true = [0, 1, 2, 2, 0]
>>> y_pred = [0, 0, 2, 1, 0]
>>> target_names = ['class 0', 'class 1', 'class 2']
>>> print(classification_report(y_true, y_pred, target_names=target_names))
precision recall f1-score support
class 0 0.67 1.00 0.80 2
class 1 0.00 0.00 0.00 1
class 2 1.00 0.50 0.67 2
accuracy 0.60 5
macro avg 0.56 0.50 0.49 5
weighted avg 0.67 0.60 0.59 5
```
Examples
- See [Recognizing hand-written digits](https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html#sphx-glr-auto-examples-classification-plot-digits-classification-py) for an example of classification report usage for hand-written digits.
- See [Custom refit strategy of a grid search with cross-validation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html#sphx-glr-auto-examples-model-selection-plot-grid-search-digits-py) for an example of classification report usage for grid search with nested cross-validation.
### 3\.4.4.8. Hamming loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#hamming-loss "Link to this heading")
The [`hamming_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hamming_loss.html#sklearn.metrics.hamming_loss "sklearn.metrics.hamming_loss") computes the average Hamming loss or [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) between two sets of samples.
If y ^ i , j is the predicted value for the j\-th label of a given sample i, y i , j is the corresponding true value, n samples is the number of samples and n labels is the number of labels, then the Hamming loss L H a m m i n g is defined as:
L
H
a
m
m
i
n
g
(
y
,
y
^
)
\=
1
n
samples
ā
n
labels
ā
i
\=
0
n
samples
ā
1
ā
j
\=
0
n
labels
ā
1
1
(
y
^
i
,
j
ā
y
i
,
j
)
where 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function).
The equation above does not hold true in the case of multiclass classification. Please refer to the note below for more information.
```
>>> from sklearn.metrics import hamming_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> hamming_loss(y_true, y_pred)
0.25
```
In the multilabel case with binary label indicators:
```
>>> hamming_loss(np.array([[0, 1], [1, 1]]), np.zeros((2, 2)))
0.75
```
Note
In multiclass classification, the Hamming loss corresponds to the Hamming distance between `y_true` and `y_pred` which is similar to the [Zero one loss](https://scikit-learn.org/stable/modules/model_evaluation.html#zero-one-loss) function. However, while zero-one loss penalizes prediction sets that do not strictly match true sets, the Hamming loss penalizes individual labels. Thus the Hamming loss, upper bounded by the zero-one loss, is always between zero and one, inclusive; and predicting a proper subset or superset of the true labels will give a Hamming loss between zero and one, exclusive.
### 3\.4.4.9. Precision, recall and F-measures[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-and-f-measures "Link to this heading")
Intuitively, [precision](https://en.wikipedia.org/wiki/Precision_and_recall#Precision) is the ability of the classifier not to label as positive a sample that is negative, and [recall](https://en.wikipedia.org/wiki/Precision_and_recall#Recall) is the ability of the classifier to find all the positive samples.
The [F-measure](https://en.wikipedia.org/wiki/F1_score) (F Ī² and F 1 measures) can be interpreted as a weighted harmonic mean of the precision and recall. A F Ī² measure reaches its best value at 1 and its worst score at 0. With Ī² \= 1, F Ī² and F 1 are equivalent, and the recall and the precision are equally important.
The [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve "sklearn.metrics.precision_recall_curve") computes a precision-recall curve from the ground truth label and a score given by the classifier by varying a decision threshold.
The [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") function computes the [average precision](https://en.wikipedia.org/w/index.php?title=Information_retrieval&oldid=793358396#Average_precision) (AP) from prediction scores. The value is between 0 and 1 and higher is better. AP is defined as
AP
\=
ā
n
(
R
n
ā
R
n
ā
1
)
P
n
where P n and R n are the precision and recall at the nth threshold. With random predictions, the AP is the fraction of positive samples.
References [\[Manning2008\]](https://scikit-learn.org/stable/modules/model_evaluation.html#manning2008) and [\[Everingham2010\]](https://scikit-learn.org/stable/modules/model_evaluation.html#everingham2010) present alternative variants of AP that interpolate the precision-recall curve. Currently, [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") does not implement any interpolated variant. References [\[Davis2006\]](https://scikit-learn.org/stable/modules/model_evaluation.html#davis2006) and [\[Flach2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#flach2015) describe why a linear interpolation of points on the precision-recall curve provides an overly-optimistic measure of classifier performance. This linear interpolation is used when computing area under the curve with the trapezoidal rule in [`auc`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.auc.html#sklearn.metrics.auc "sklearn.metrics.auc"). [\[Chen2024\]](https://scikit-learn.org/stable/modules/model_evaluation.html#chen2024) benchmarks different interpolation strategies to demonstrate the effects.
Several functions allow you to analyze the precision, recall and F-measures score:
| | |
|---|---|
| [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score")(y\_true, y\_score, \*) | Compute average precision (AP) from prediction scores. |
| [`f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the F1 score, also known as balanced F-score or F-measure. |
| [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score")(y\_true, y\_pred, \*, beta\[, ...\]) | Compute the F-beta score. |
| [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve "sklearn.metrics.precision_recall_curve")(y\_true, y\_score, \*\[, ...\]) | Compute precision-recall pairs for different probability thresholds. |
| [`precision_recall_fscore_support`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html#sklearn.metrics.precision_recall_fscore_support "sklearn.metrics.precision_recall_fscore_support")(y\_true, ...) | Compute precision, recall, F-measure and support for each class. |
| [`precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the precision. |
| [`recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the recall. |
Note that the [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve "sklearn.metrics.precision_recall_curve") function is restricted to the binary case. The [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") function supports multiclass and multilabel formats by computing each class score in a One-vs-the-rest (OvR) fashion and averaging them or not depending of its `average` argument value.
The [`PrecisionRecallDisplay.from_estimator`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.PrecisionRecallDisplay.html#sklearn.metrics.PrecisionRecallDisplay.from_estimator "sklearn.metrics.PrecisionRecallDisplay.from_estimator") and [`PrecisionRecallDisplay.from_predictions`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.PrecisionRecallDisplay.html#sklearn.metrics.PrecisionRecallDisplay.from_predictions "sklearn.metrics.PrecisionRecallDisplay.from_predictions") functions will plot the precision-recall curve as follows.
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html#plot-the-precision-recall-curve)
Examples
- See [Custom refit strategy of a grid search with cross-validation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html#sphx-glr-auto-examples-model-selection-plot-grid-search-digits-py) for an example of [`precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score") and [`recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score") usage to estimate parameters using grid search with nested cross-validation.
- See [Precision-Recall](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html#sphx-glr-auto-examples-model-selection-plot-precision-recall-py) for an example of [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve "sklearn.metrics.precision_recall_curve") usage to evaluate classifier output quality.
References
\[[Manning2008](https://scikit-learn.org/stable/modules/model_evaluation.html#id25)\]
C.D. Manning, P. Raghavan, H. SchĆ¼tze, [Introduction to Information Retrieval](https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html), 2008.
\[[Everingham2010](https://scikit-learn.org/stable/modules/model_evaluation.html#id26)\]
M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn, A. Zisserman, [The Pascal Visual Object Classes (VOC) Challenge](https://citeseerx.ist.psu.edu/doc_view/pid/b6bebfd529b233f00cb854b7d8070319600cf59d), IJCV 2010.
\[[Davis2006](https://scikit-learn.org/stable/modules/model_evaluation.html#id27)\]
J. Davis, M. Goadrich, [The Relationship Between Precision-Recall and ROC Curves](https://www.biostat.wisc.edu/~page/rocpr.pdf), ICML 2006.
\[[Flach2015](https://scikit-learn.org/stable/modules/model_evaluation.html#id28)\]
P.A. Flach, M. Kull, [Precision-Recall-Gain Curves: PR Analysis Done Right](https://papers.nips.cc/paper/5867-precision-recall-gain-curves-pr-analysis-done-right.pdf), NIPS 2015.
\[[Chen2024](https://scikit-learn.org/stable/modules/model_evaluation.html#id29)\]
W. Chen, C. Miao, Z. Zhang, C.S. Fung, R. Wang, Y. Chen, Y. Qian, L. Cheng, K.Y. Yip, S.K Tsui, Q. Cao, [Commonly used software tools produce conflicting and overly-optimistic AUPRC values](https://doi.org/10.1186/s13059-024-03266-y), Genome Biology 2024.
#### 3\.4.4.9.1. Binary classification[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#binary-classification "Link to this heading")
In a binary classification task, the terms āāpositiveāā and āānegativeāā refer to the classifierās prediction, and the terms āātrueāā and āāfalseāā refer to whether that prediction corresponds to the external judgment (sometimes known as the āāobservationāā). Given these definitions, we can formulate the following table:
| | | |
|---|---|---|
| | Actual class (observation) | |
| Predicted class (expectation) | tp (true positive) Correct result | fp (false positive) Unexpected result |
| fn (false negative) Missing result | tn (true negative) Correct absence of result | |
In this context, we can define the notions of precision and recall:
precision
\=
tp
tp
\+
fp
,
recall
\=
tp
tp
\+
fn
,
(Sometimes recall is also called āāsensitivityāā)
F-measure is the weighted harmonic mean of precision and recall, with precisionās contribution to the mean weighted by some parameter Ī²:
F
Ī²
\=
(
1
\+
Ī²
2
)
precision
Ć
recall
Ī²
2
precision
\+
recall
To avoid division by zero when precision and recall are zero, Scikit-Learn calculates F-measure with this otherwise-equivalent formula:
F
Ī²
\=
(
1
\+
Ī²
2
)
tp
(
1
\+
Ī²
2
)
tp
\+
fp
\+
Ī²
2
fn
Note that this formula is still undefined when there are no true positives, false positives, or false negatives. By default, F-1 for a set of exclusively true negatives is calculated as 0, however this behavior can be changed using the `zero_division` parameter. Here are some small examples in binary classification:
```
>>> from sklearn import metrics
>>> y_pred = [0, 1, 0, 0]
>>> y_true = [0, 1, 0, 1]
>>> metrics.precision_score(y_true, y_pred)
1.0
>>> metrics.recall_score(y_true, y_pred)
0.5
>>> metrics.f1_score(y_true, y_pred)
0.66
>>> metrics.fbeta_score(y_true, y_pred, beta=0.5)
0.83
>>> metrics.fbeta_score(y_true, y_pred, beta=1)
0.66
>>> metrics.fbeta_score(y_true, y_pred, beta=2)
0.55
>>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5)
(array([0.66, 1. ]), array([1. , 0.5]), array([0.71, 0.83]), array([2, 2]))
>>> import numpy as np
>>> from sklearn.metrics import precision_recall_curve
>>> from sklearn.metrics import average_precision_score
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> precision, recall, threshold = precision_recall_curve(y_true, y_scores)
>>> precision
array([0.5 , 0.66, 0.5 , 1. , 1. ])
>>> recall
array([1. , 1. , 0.5, 0.5, 0. ])
>>> threshold
array([0.1 , 0.35, 0.4 , 0.8 ])
>>> average_precision_score(y_true, y_scores)
0.83
```
#### 3\.4.4.9.2. Multiclass and multilabel classification[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multiclass-and-multilabel-classification "Link to this heading")
In a multiclass and multilabel classification task, the notions of precision, recall, and F-measures can be applied to each label independently. There are a few ways to combine results across labels, specified by the `average` argument to the [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score"), [`f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score"), [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score"), [`precision_recall_fscore_support`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html#sklearn.metrics.precision_recall_fscore_support "sklearn.metrics.precision_recall_fscore_support"), [`precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score") and [`recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score") functions, as described [above](https://scikit-learn.org/stable/modules/model_evaluation.html#average).
Note the following behaviors when averaging:
- If all labels are included, āmicroā-averaging in a multiclass setting will produce precision, recall and F that are all identical to accuracy.
- āweightedā averaging may produce a F-score that is not between precision and recall.
- āmacroā averaging for F-measures is calculated as the arithmetic mean over per-label/class F-measures, not the harmonic mean over the arithmetic precision and recall means. Both calculations can be seen in the literature but are not equivalent, see [\[OB2019\]](https://scikit-learn.org/stable/modules/model_evaluation.html#ob2019) for details.
To make this more explicit, consider the following notation:
- y the set of *true* ( s a m p l e , l a b e l ) pairs
- y ^ the set of *predicted* ( s a m p l e , l a b e l ) pairs
- L the set of labels
- S the set of samples
- y s the subset of y with sample s, i.e. y s := { ( s ā² , l ) ā y \| s ā² \= s }
- y l the subset of y with label l
- similarly, y ^ s and y ^ l are subsets of y ^
- P ( A , B ) := \| A ā© B \| \| B \| for some sets A and B
- R ( A , B ) := \| A ā© B \| \| A \| (Conventions vary on handling A \= ā
; this implementation uses R ( A , B ) := 0, and similar for P.)
- F Ī² ( A , B ) := ( 1 \+ Ī² 2 ) P ( A , B ) Ć R ( A , B ) Ī² 2 P ( A , B ) \+ R ( A , B )
Then the metrics are defined as:
| `average` | Precision | Recall | F\_beta |
|---|---|---|---|
| `"micro"` | P ( y , y ^ ) | | |
```
>>> from sklearn import metrics
>>> y_true = [0, 1, 2, 0, 1, 2]
>>> y_pred = [0, 2, 1, 0, 0, 1]
>>> metrics.precision_score(y_true, y_pred, average='macro')
0.22
>>> metrics.recall_score(y_true, y_pred, average='micro')
0.33
>>> metrics.f1_score(y_true, y_pred, average='weighted')
0.267
>>> metrics.fbeta_score(y_true, y_pred, average='macro', beta=0.5)
0.238
>>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5, average=None)
(array([0.667, 0., 0.]), array([1., 0., 0.]), array([0.714, 0., 0.]), array([2, 2, 2]))
```
For multiclass classification with a ānegative classā, it is possible to exclude some labels:
```
>>> metrics.recall_score(y_true, y_pred, labels=[1, 2], average='micro')
... # excluding 0, no labels were correctly recalled
0.0
```
Similarly, labels not present in the data sample may be accounted for in macro-averaging.
```
>>> metrics.precision_score(y_true, y_pred, labels=[0, 1, 2, 3], average='macro')
0.166
```
References
\[[OB2019](https://scikit-learn.org/stable/modules/model_evaluation.html#id30)\]
[Opitz, J., & Burst, S. (2019). āMacro f1 and macro f1.ā](https://arxiv.org/abs/1911.03347)
### 3\.4.4.10. Jaccard similarity coefficient score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#jaccard-similarity-coefficient-score "Link to this heading")
The [`jaccard_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html#sklearn.metrics.jaccard_score "sklearn.metrics.jaccard_score") function computes the average of [Jaccard similarity coefficients](https://en.wikipedia.org/wiki/Jaccard_index), also called the Jaccard index, between pairs of label sets.
The Jaccard similarity coefficient with a ground truth label set y and predicted label set y ^, is defined as
J
(
y
,
y
^
)
\=
\|
y
ā©
y
^
\|
\|
y
āŖ
y
^
\|
.
The [`jaccard_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html#sklearn.metrics.jaccard_score "sklearn.metrics.jaccard_score") (like [`precision_recall_fscore_support`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html#sklearn.metrics.precision_recall_fscore_support "sklearn.metrics.precision_recall_fscore_support")) applies natively to binary targets. By computing it set-wise it can be extended to apply to multilabel and multiclass through the use of `average` (see [above](https://scikit-learn.org/stable/modules/model_evaluation.html#average)).
In the binary case:
```
>>> import numpy as np
>>> from sklearn.metrics import jaccard_score
>>> y_true = np.array([[0, 1, 1],
... [1, 1, 0]])
>>> y_pred = np.array([[1, 1, 1],
... [1, 0, 0]])
>>> jaccard_score(y_true[0], y_pred[0])
0.6666
```
In the 2D comparison case (e.g. image similarity):
```
>>> jaccard_score(y_true, y_pred, average="micro")
0.6
```
In the multilabel case with binary label indicators:
```
>>> jaccard_score(y_true, y_pred, average='samples')
0.5833
>>> jaccard_score(y_true, y_pred, average='macro')
0.6666
>>> jaccard_score(y_true, y_pred, average=None)
array([0.5, 0.5, 1. ])
```
Multiclass problems are binarized and treated like the corresponding multilabel problem:
```
>>> y_pred = [0, 2, 1, 2]
>>> y_true = [0, 1, 2, 2]
>>> jaccard_score(y_true, y_pred, average=None)
array([1. , 0. , 0.33])
>>> jaccard_score(y_true, y_pred, average='macro')
0.44
>>> jaccard_score(y_true, y_pred, average='micro')
0.33
```
### 3\.4.4.11. Hinge loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#hinge-loss "Link to this heading")
The [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss") function computes the average distance between the model and the data using [hinge loss](https://en.wikipedia.org/wiki/Hinge_loss), a one-sided metric that considers only prediction errors. (Hinge loss is used in maximal margin classifiers such as support vector machines.)
If the true label y i of a binary classification task is encoded as y i \= { ā 1 , \+ 1 } for every sample i; and w i is the corresponding predicted decision (an array of shape (`n_samples`,) as output by the `decision_function` method), then the hinge loss is defined as:
L
Hinge
(
y
,
w
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
max
{
1
ā
w
i
y
i
,
0
}
If there are more than two labels, [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss") uses a multiclass variant due to Crammer & Singer. [Here](https://jmlr.csail.mit.edu/papers/volume2/crammer01a/crammer01a.pdf) is the paper describing it.
In this case the predicted decision is an array of shape (`n_samples`, `n_labels`). If w i , y i is the predicted decision for the true label y i of the i\-th sample; and w ^ i , y i \= max { w i , y j \| y j ā y i } is the maximum of the predicted decisions for all the other labels, then the multi-class hinge loss is defined by:
L
Hinge
(
y
,
w
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
max
{
1
\+
w
^
i
,
y
i
ā
w
i
,
y
i
,
0
}
Here is a small example demonstrating the use of the [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss") function with an svm classifier in a binary class problem:
```
>>> from sklearn import svm
>>> from sklearn.metrics import hinge_loss
>>> X = [[0], [1]]
>>> y = [-1, 1]
>>> est = svm.LinearSVC(random_state=0)
>>> est.fit(X, y)
LinearSVC(random_state=0)
>>> pred_decision = est.decision_function([[-2], [3], [0.5]])
>>> pred_decision
array([-2.18, 2.36, 0.09])
>>> hinge_loss([-1, 1, 1], pred_decision)
0.3
```
Here is an example demonstrating the use of the [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss") function with an svm classifier in a multiclass problem:
```
>>> X = np.array([[0], [1], [2], [3]])
>>> Y = np.array([0, 1, 2, 3])
>>> labels = np.array([0, 1, 2, 3])
>>> est = svm.LinearSVC()
>>> est.fit(X, Y)
LinearSVC()
>>> pred_decision = est.decision_function([[-1], [2], [3]])
>>> y_true = [0, 2, 3]
>>> hinge_loss(y_true, pred_decision, labels=labels)
0.56
```
### 3\.4.4.12. Log loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#log-loss "Link to this heading")
Log loss, also called logistic regression loss or cross-entropy loss, is defined on probability estimates. It is commonly used in (multinomial) logistic regression and neural networks, as well as in some variants of expectation-maximization, and can be used to evaluate the probability outputs (`predict_proba`) of a classifier instead of its discrete predictions.
For binary classification with a true label y ā { 0 , 1 } and a probability estimate p ^ ā Pr ( y \= 1 ), the log loss per sample is the negative log-likelihood of the classifier given the true label:
L
log
(
y
,
p
^
)
\=
ā
log
ā”
Pr
(
y
\|
p
^
)
\=
ā
(
y
log
ā”
(
p
^
)
\+
(
1
ā
y
)
log
ā”
(
1
ā
p
^
)
)
This extends to the multiclass case as follows. Let the true labels for a set of samples be encoded as a 1-of-K binary indicator matrix Y, i.e., y i , k \= 1 if sample i has label k taken from a set of K labels. Let P ^ be a matrix of probability estimates, with elements p ^ i , k ā Pr ( y i , k \= 1 ). Then the log loss of the whole set is
L
log
(
Y
,
P
^
)
\=
ā
log
ā”
Pr
(
Y
\|
P
^
)
\=
ā
1
N
ā
i
\=
0
N
ā
1
ā
k
\=
0
K
ā
1
y
i
,
k
log
ā”
p
^
i
,
k
To see how this generalizes the binary log loss given above, note that in the binary case, p ^ i , 0 \= 1 ā p ^ i , 1 and y i , 0 \= 1 ā y i , 1, so expanding the inner sum over y i , k ā { 0 , 1 } gives the binary log loss.
The [`log_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html#sklearn.metrics.log_loss "sklearn.metrics.log_loss") function computes log loss given a list of ground-truth labels and a probability matrix, as returned by an estimatorās `predict_proba` method.
```
>>> from sklearn.metrics import log_loss
>>> y_true = [0, 0, 1, 1]
>>> y_pred = [[.9, .1], [.8, .2], [.3, .7], [.01, .99]]
>>> log_loss(y_true, y_pred)
0.1738
```
The first `[.9, .1]` in `y_pred` denotes 90% probability that the first sample has label 0. The log loss is non-negative.
### 3\.4.4.13. Matthews correlation coefficient[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#matthews-correlation-coefficient "Link to this heading")
The [`matthews_corrcoef`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html#sklearn.metrics.matthews_corrcoef "sklearn.metrics.matthews_corrcoef") function computes the [Matthewās correlation coefficient (MCC)](https://en.wikipedia.org/wiki/Matthews_correlation_coefficient) for binary classes. Quoting Wikipedia:
> āThe Matthews correlation coefficient is used in machine learning as a measure of the quality of binary (two-class) classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient.ā
In the binary (two-class) case, t p, t n, f p and f n are respectively the number of true positives, true negatives, false positives and false negatives, the MCC is defined as
M
C
C
\=
t
p
Ć
t
n
ā
f
p
Ć
f
n
(
t
p
\+
f
p
)
(
t
p
\+
f
n
)
(
t
n
\+
f
p
)
(
t
n
\+
f
n
)
.
In the multiclass case, the Matthews correlation coefficient can be [defined](http://rk.kvl.dk/introduction/index.html) in terms of a [`confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix "sklearn.metrics.confusion_matrix") C for K classes. To simplify the definition consider the following intermediate variables:
- t k \= ā i K C i k the number of times class k truly occurred,
- p k \= ā i K C k i the number of times class k was predicted,
- c \= ā k K C k k the total number of samples correctly predicted,
- s \= ā i K ā j K C i j the total number of samples.
Then the multiclass MCC is defined as:
M
C
C
\=
c
Ć
s
ā
ā
k
K
p
k
Ć
t
k
(
s
2
ā
ā
k
K
p
k
2
)
Ć
(
s
2
ā
ā
k
K
t
k
2
)
When there are more than two labels, the value of the MCC will no longer range between -1 and +1. Instead the minimum value will be somewhere between -1 and 0 depending on the number and distribution of ground truth labels. The maximum value is always +1. For additional information, see [\[WikipediaMCC2021\]](https://scikit-learn.org/stable/modules/model_evaluation.html#wikipediamcc2021).
Here is a small example illustrating the usage of the [`matthews_corrcoef`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html#sklearn.metrics.matthews_corrcoef "sklearn.metrics.matthews_corrcoef") function:
```
>>> from sklearn.metrics import matthews_corrcoef
>>> y_true = [+1, +1, +1, -1]
>>> y_pred = [+1, -1, +1, +1]
>>> matthews_corrcoef(y_true, y_pred)
-0.33
```
References
\[[WikipediaMCC2021](https://scikit-learn.org/stable/modules/model_evaluation.html#id34)\]
Wikipedia contributors. Phi coefficient. Wikipedia, The Free Encyclopedia. April 21, 2021, 12:21 CEST. Available at: <https://en.wikipedia.org/wiki/Phi_coefficient> Accessed April 21, 2021.
### 3\.4.4.14. Multi-label confusion matrix[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multi-label-confusion-matrix "Link to this heading")
The [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix") function computes class-wise (default) or sample-wise (samplewise=True) multilabel confusion matrix to evaluate the accuracy of a classification. multilabel\_confusion\_matrix also treats multiclass data as if it were multilabel, as this is a transformation commonly applied to evaluate multiclass problems with binary classification metrics (such as precision, recall, etc.).
When calculating class-wise multilabel confusion matrix C, the count of true negatives for class i is C i , 0 , 0, false negatives is C i , 1 , 0, true positives is C i , 1 , 1 and false positives is C i , 0 , 1.
Here is an example demonstrating the use of the [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix") function with [multilabel indicator matrix](https://scikit-learn.org/stable/glossary.html#term-multilabel-indicator-matrix) input:
```
>>> import numpy as np
>>> from sklearn.metrics import multilabel_confusion_matrix
>>> y_true = np.array([[1, 0, 1],
... [0, 1, 0]])
>>> y_pred = np.array([[1, 0, 0],
... [0, 1, 1]])
>>> multilabel_confusion_matrix(y_true, y_pred)
array([[[1, 0],
[0, 1]],
[[1, 0],
[0, 1]],
[[0, 1],
[1, 0]]])
```
Or a confusion matrix can be constructed for each sampleās labels:
```
>>> multilabel_confusion_matrix(y_true, y_pred, samplewise=True)
array([[[1, 0],
[1, 1]],
[[1, 1],
[0, 1]]])
```
Here is an example demonstrating the use of the [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix") function with [multiclass](https://scikit-learn.org/stable/glossary.html#term-multiclass) input:
```
>>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
>>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
>>> multilabel_confusion_matrix(y_true, y_pred,
... labels=["ant", "bird", "cat"])
array([[[3, 1],
[0, 2]],
[[5, 0],
[1, 0]],
[[2, 1],
[1, 2]]])
```
Here are some examples demonstrating the use of the [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix") function to calculate recall (or sensitivity), specificity, fall out and miss rate for each class in a problem with multilabel indicator matrix input.
Calculating [recall](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) (also called the true positive rate or the sensitivity) for each class:
```
>>> y_true = np.array([[0, 0, 1],
... [0, 1, 0],
... [1, 1, 0]])
>>> y_pred = np.array([[0, 1, 0],
... [0, 0, 1],
... [1, 1, 0]])
>>> mcm = multilabel_confusion_matrix(y_true, y_pred)
>>> tn = mcm[:, 0, 0]
>>> tp = mcm[:, 1, 1]
>>> fn = mcm[:, 1, 0]
>>> fp = mcm[:, 0, 1]
>>> tp / (tp + fn)
array([1. , 0.5, 0. ])
```
Calculating [specificity](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) (also called the true negative rate) for each class:
```
>>> tn / (tn + fp)
array([1. , 0. , 0.5])
```
Calculating [fall out](https://en.wikipedia.org/wiki/False_positive_rate) (also called the false positive rate) for each class:
```
>>> fp / (fp + tn)
array([0. , 1. , 0.5])
```
Calculating [miss rate](https://en.wikipedia.org/wiki/False_positives_and_false_negatives) (also called the false negative rate) for each class:
```
>>> fn / (fn + tp)
array([0. , 0.5, 1. ])
```
### 3\.4.4.15. Receiver operating characteristic (ROC)[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#receiver-operating-characteristic-roc "Link to this heading")
The function [`roc_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve "sklearn.metrics.roc_curve") computes the [receiver operating characteristic curve, or ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic). Quoting Wikipedia :
> āA receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied. It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate) vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold settings. TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate.ā
This function requires the true binary value and the target scores, which can either be probability estimates of the positive class, confidence values, or binary decisions. Here is a small example of how to use the [`roc_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve "sklearn.metrics.roc_curve") function:
```
>>> import numpy as np
>>> from sklearn.metrics import roc_curve
>>> y = np.array([1, 1, 2, 2])
>>> scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, tpr, thresholds = roc_curve(y, scores, pos_label=2)
>>> fpr
array([0. , 0. , 0.5, 0.5, 1. ])
>>> tpr
array([0. , 0.5, 0.5, 1. , 1. ])
>>> thresholds
array([ inf, 0.8 , 0.4 , 0.35, 0.1 ])
```
Compared to metrics such as the subset accuracy, the Hamming loss, or the F1 score, ROC doesnāt require optimizing a threshold for each label.
The [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") function, denoted by ROC-AUC or AUROC, computes the area under the ROC curve. By doing so, the curve information is summarized in one number.
The following figure shows the ROC curve and ROC-AUC score for a classifier aimed to distinguish the virginica flower from the rest of the species in the [Iris plants dataset](https://scikit-learn.org/stable/datasets/toy_dataset.html#iris-dataset):
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html)
For more information see the [Wikipedia article on AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve).
#### 3\.4.4.15.1. Binary case[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#binary-case "Link to this heading")
In the **binary case**, you can either provide the probability estimates, using the `classifier.predict_proba()` method, or the non-thresholded decision values given by the `classifier.decision_function()` method. In the case of providing the probability estimates, the probability of the class with the āgreater labelā should be provided. The āgreater labelā corresponds to `classifier.classes_[1]` and thus `classifier.predict_proba(X)[:, 1]`. Therefore, the `y_score` parameter is of size (n\_samples,).
```
>>> from sklearn.datasets import load_breast_cancer
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.metrics import roc_auc_score
>>> X, y = load_breast_cancer(return_X_y=True)
>>> clf = LogisticRegression().fit(X, y)
>>> clf.classes_
array([0, 1])
```
We can use the probability estimates corresponding to `clf.classes_[1]`.
```
>>> y_score = clf.predict_proba(X)[:, 1]
>>> roc_auc_score(y, y_score)
0.99
```
Otherwise, we can use the non-thresholded decision values
```
>>> roc_auc_score(y, clf.decision_function(X))
0.99
```
#### 3\.4.4.15.2. Multi-class case[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multi-class-case "Link to this heading")
The [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") function can also be used in **multi-class classification**. Two averaging strategies are currently supported: the one-vs-one algorithm computes the average of the pairwise ROC AUC scores, and the one-vs-rest algorithm computes the average of the ROC AUC scores for each class against all other classes. In both cases, the predicted labels are provided in an array with values from 0 to `n_classes`, and the scores correspond to the probability estimates that a sample belongs to a particular class. The OvO and OvR algorithms support weighting uniformly (`average='macro'`) and by prevalence (`average='weighted'`).
One-vs-one Algorithm[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#one-vs-one-algorithm "Link to this dropdown")
Computes the average AUC of all possible pairwise combinations of classes. [\[HT2001\]](https://scikit-learn.org/stable/modules/model_evaluation.html#ht2001) defines a multiclass AUC metric weighted uniformly:
1
c
(
c
ā
1
)
ā
j
\=
1
c
ā
k
\>
j
c
(
AUC
(
j
\|
k
)
\+
AUC
(
k
\|
j
)
)
where c is the number of classes and AUC ( j \| k ) is the AUC with class j as the positive class and class k as the negative class. In general, AUC ( j \| k ) ā AUC ( k \| j ) in the multiclass case. This algorithm is used by setting the keyword argument `multiclass` to `'ovo'` and `average` to `'macro'`.
The [\[HT2001\]](https://scikit-learn.org/stable/modules/model_evaluation.html#ht2001) multiclass AUC metric can be extended to be weighted by the prevalence:
1
c
(
c
ā
1
)
ā
j
\=
1
c
ā
k
\>
j
c
p
(
j
āŖ
k
)
(
AUC
(
j
\|
k
)
\+
AUC
(
k
\|
j
)
)
where c is the number of classes. This algorithm is used by setting the keyword argument `multiclass` to `'ovo'` and `average` to `'weighted'`. The `'weighted'` option returns a prevalence-weighted average as described in [\[FC2009\]](https://scikit-learn.org/stable/modules/model_evaluation.html#fc2009).
One-vs-rest Algorithm[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#one-vs-rest-algorithm "Link to this dropdown")
Computes the AUC of each class against the rest [\[PD2000\]](https://scikit-learn.org/stable/modules/model_evaluation.html#pd2000). The algorithm is functionally the same as the multilabel case. To enable this algorithm set the keyword argument `multiclass` to `'ovr'`. Additionally to `'macro'` [\[F2006\]](https://scikit-learn.org/stable/modules/model_evaluation.html#f2006) and `'weighted'` [\[F2001\]](https://scikit-learn.org/stable/modules/model_evaluation.html#f2001) averaging, OvR supports `'micro'` averaging.
In applications where a high false positive rate is not tolerable the parameter `max_fpr` of [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") can be used to summarize the ROC curve up to the given limit.
The following figure shows the micro-averaged ROC curve and its corresponding ROC-AUC score for a classifier aimed to distinguish the different species in the [Iris plants dataset](https://scikit-learn.org/stable/datasets/toy_dataset.html#iris-dataset):
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html)
#### 3\.4.4.15.3. Multi-label case[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multi-label-case "Link to this heading")
In **multi-label classification**, the [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") function is extended by averaging over the labels as [above](https://scikit-learn.org/stable/modules/model_evaluation.html#average). In this case, you should provide a `y_score` of shape `(n_samples, n_classes)`. Thus, when using the probability estimates, one needs to select the probability of the class with the greater label for each output.
```
>>> from sklearn.datasets import make_multilabel_classification
>>> from sklearn.multioutput import MultiOutputClassifier
>>> X, y = make_multilabel_classification(random_state=0)
>>> inner_clf = LogisticRegression(random_state=0)
>>> clf = MultiOutputClassifier(inner_clf).fit(X, y)
>>> y_score = np.transpose([y_pred[:, 1] for y_pred in clf.predict_proba(X)])
>>> roc_auc_score(y, y_score, average=None)
array([0.828, 0.851, 0.94, 0.87, 0.95])
```
And the decision values do not require such processing.
```
>>> from sklearn.linear_model import RidgeClassifierCV
>>> clf = RidgeClassifierCV().fit(X, y)
>>> y_score = clf.decision_function(X)
>>> roc_auc_score(y, y_score, average=None)
array([0.82, 0.85, 0.93, 0.87, 0.94])
```
Examples
- See [Multiclass Receiver Operating Characteristic (ROC)](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#sphx-glr-auto-examples-model-selection-plot-roc-py) for an example of using ROC to evaluate the quality of the output of a classifier.
- See [Receiver Operating Characteristic (ROC) with cross validation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossval.html#sphx-glr-auto-examples-model-selection-plot-roc-crossval-py) for an example of using ROC to evaluate classifier output quality, using cross-validation.
- See [Species distribution modeling](https://scikit-learn.org/stable/auto_examples/applications/plot_species_distribution_modeling.html#sphx-glr-auto-examples-applications-plot-species-distribution-modeling-py) for an example of using ROC to model species distribution.
References
\[HT2001\] ([1](https://scikit-learn.org/stable/modules/model_evaluation.html#id35),[2](https://scikit-learn.org/stable/modules/model_evaluation.html#id36))
Hand, D.J. and Till, R.J., (2001). [A simple generalisation of the area under the ROC curve for multiple class classification problems.](http://link.springer.com/article/10.1023/A:1010920819831) Machine learning, 45(2), pp. 171-186.
\[[FC2009](https://scikit-learn.org/stable/modules/model_evaluation.html#id37)\]
Ferri, CĆØsar & Hernandez-Orallo, Jose & Modroiu, R. (2009). [An Experimental Comparison of Performance Measures for Classification.](https://www.math.ucdavis.edu/~saito/data/roc/ferri-class-perf-metrics.pdf) Pattern Recognition Letters. 30. 27-38.
\[[PD2000](https://scikit-learn.org/stable/modules/model_evaluation.html#id38)\]
Provost, F., Domingos, P. (2000). [Well-trained PETs: Improving probability estimation trees](https://fosterprovost.com/publication/well-trained-pets-improving-probability-estimation-trees/) (Section 6.2), CeDER Working Paper \#IS-00-04, Stern School of Business, New York University.
\[[F2006](https://scikit-learn.org/stable/modules/model_evaluation.html#id39)\]
Fawcett, T., 2006. [An introduction to ROC analysis.](http://www.sciencedirect.com/science/article/pii/S016786550500303X) Pattern Recognition Letters, 27(8), pp. 861-874.
\[[F2001](https://scikit-learn.org/stable/modules/model_evaluation.html#id40)\]
Fawcett, T., 2001. [Using rule sets to maximize ROC performance](https://ieeexplore.ieee.org/document/989510/) In Data Mining, 2001. Proceedings IEEE International Conference, pp. 131-138.
### 3\.4.4.16. Detection error tradeoff (DET)[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#detection-error-tradeoff-det "Link to this heading")
The function [`det_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.det_curve.html#sklearn.metrics.det_curve "sklearn.metrics.det_curve") computes the detection error tradeoff curve (DET) curve [\[WikipediaDET2017\]](https://scikit-learn.org/stable/modules/model_evaluation.html#wikipediadet2017). Quoting Wikipedia:
> āA detection error tradeoff (DET) graph is a graphical plot of error rates for binary classification systems, plotting false reject rate vs. false accept rate. The x- and y-axes are scaled non-linearly by their standard normal deviates (or just by logarithmic transformation), yielding tradeoff curves that are more linear than ROC curves, and use most of the image area to highlight the differences of importance in the critical operating region.ā
DET curves are a variation of receiver operating characteristic (ROC) curves where False Negative Rate is plotted on the y-axis instead of True Positive Rate. DET curves are commonly plotted in normal deviate scale by transformation with Ļ ā 1 (with Ļ being the cumulative distribution function). The resulting performance curves explicitly visualize the tradeoff of error types for given classification algorithms. See [\[Martin1997\]](https://scikit-learn.org/stable/modules/model_evaluation.html#martin1997) for examples and further motivation.
This figure compares the ROC and DET curves of two example classifiers on the same classification task:
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_det.html)
Properties[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#properties "Link to this dropdown")
- DET curves form a linear curve in normal deviate scale if the detection scores are normally (or close-to normally) distributed. It was shown by [\[Navratil2007\]](https://scikit-learn.org/stable/modules/model_evaluation.html#navratil2007) that the reverse is not necessarily true and even more general distributions are able to produce linear DET curves.
- The normal deviate scale transformation spreads out the points such that a comparatively larger space of plot is occupied. Therefore curves with similar classification performance might be easier to distinguish on a DET plot.
- With False Negative Rate being āinverseā to True Positive Rate the point of perfection for DET curves is the origin (in contrast to the top left corner for ROC curves).
Applications and limitations[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#applications-and-limitations "Link to this dropdown")
DET curves are intuitive to read and hence allow quick visual assessment of a classifierās performance. Additionally DET curves can be consulted for threshold analysis and operating point selection. This is particularly helpful if a comparison of error types is required.
On the other hand DET curves do not provide their metric as a single number. Therefore for either automated evaluation or comparison to other classification tasks metrics like the derived area under ROC curve might be better suited.
Examples
- See [Detection error tradeoff (DET) curve](https://scikit-learn.org/stable/auto_examples/model_selection/plot_det.html#sphx-glr-auto-examples-model-selection-plot-det-py) for an example comparison between receiver operating characteristic (ROC) curves and Detection error tradeoff (DET) curves.
References
\[[WikipediaDET2017](https://scikit-learn.org/stable/modules/model_evaluation.html#id41)\]
Wikipedia contributors. Detection error tradeoff. Wikipedia, The Free Encyclopedia. September 4, 2017, 23:33 UTC. Available at: <https://en.wikipedia.org/w/index.php?title=Detection_error_tradeoff&oldid=798982054>. Accessed February 19, 2018.
\[[Martin1997](https://scikit-learn.org/stable/modules/model_evaluation.html#id42)\]
A. Martin, G. Doddington, T. Kamm, M. Ordowski, and M. Przybocki, [The DET Curve in Assessment of Detection Task Performance](https://ccc.inaoep.mx/~villasen/bib/martin97det.pdf), NIST 1997.
\[[Navratil2007](https://scikit-learn.org/stable/modules/model_evaluation.html#id43)\]
J. Navratil and D. Klusacek, [āOn Linear DETsā](https://ieeexplore.ieee.org/document/4218079), 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ā07, Honolulu, HI, 2007, pp. IV-229-IV-232.
### 3\.4.4.17. Zero one loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#zero-one-loss "Link to this heading")
The [`zero_one_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.zero_one_loss.html#sklearn.metrics.zero_one_loss "sklearn.metrics.zero_one_loss") function computes the sum or the average of the 0-1 classification loss (L 0 ā 1) over n samples. By default, the function normalizes over the sample. To get the sum of the L 0 ā 1, set `normalize` to `False`.
In multilabel classification, the [`zero_one_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.zero_one_loss.html#sklearn.metrics.zero_one_loss "sklearn.metrics.zero_one_loss") scores a subset as one if its labels strictly match the predictions, and as a zero if there are any errors. By default, the function returns the percentage of imperfectly predicted subsets. To get the count of such subsets instead, set `normalize` to `False`.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value, then the 0-1 loss L 0 ā 1 is defined as:
L
0
ā
1
(
y
,
y
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
1
(
y
^
i
ā
y
i
)
where 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function). The zero-one loss can also be computed as zero-one loss \= 1 ā accuracy.
```
>>> from sklearn.metrics import zero_one_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> zero_one_loss(y_true, y_pred)
0.25
>>> zero_one_loss(y_true, y_pred, normalize=False)
1.0
```
In the multilabel case with binary label indicators, where the first label set \[0,1\] has an error:
```
>>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5
>>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2)), normalize=False)
1.0
```
Examples
- See [Recursive feature elimination with cross-validation](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html#sphx-glr-auto-examples-feature-selection-plot-rfe-with-cross-validation-py) for an example of zero one loss usage to perform recursive feature elimination with cross-validation.
### 3\.4.4.18. Brier score loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#brier-score-loss "Link to this heading")
The [`brier_score_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss "sklearn.metrics.brier_score_loss") function computes the [Brier score](https://en.wikipedia.org/wiki/Brier_score) for binary and multiclass probabilistic predictions and is equivalent to the mean squared error. Quoting Wikipedia:
> āThe Brier score is a strictly proper scoring rule that measures the accuracy of probabilistic predictions. \[ā¦\] \[It\] is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes or classes.ā
Let the true labels for a set of N data points be encoded as a 1-of-K binary indicator matrix Y, i.e., y i , k \= 1 if sample i has label k taken from a set of K labels. Let P ^ be a matrix of probability estimates with elements p ^ i , k ā Pr ( y i , k \= 1 ). Following the original definition by [\[Brier1950\]](https://scikit-learn.org/stable/modules/model_evaluation.html#brier1950), the Brier score is given by:
B
S
(
Y
,
P
^
)
\=
1
N
ā
i
\=
0
N
ā
1
ā
k
\=
0
K
ā
1
(
y
i
,
k
ā
p
^
i
,
k
)
2
The Brier score lies in the interval \[ 0 , 2 \] and the lower the value the better the probability estimates are (the mean squared difference is smaller). Actually, the Brier score is a strictly proper scoring rule, meaning that it achieves the best score only when the estimated probabilities equal the true ones.
Note that in the binary case, the Brier score is usually divided by two and ranges between \[ 0 , 1 \]. For binary targets y i ā { 0 , 1 } and probability estimates p ^ i ā Pr ( y i \= 1 ) for the positive class, the Brier score is then equal to:
B
S
(
y
,
p
^
)
\=
1
N
ā
i
\=
0
N
ā
1
(
y
i
ā
p
^
i
)
2
The [`brier_score_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss "sklearn.metrics.brier_score_loss") function computes the Brier score given the ground-truth labels and predicted probabilities, as returned by an estimatorās `predict_proba` method. The `scale_by_half` parameter controls which of the two above definitions to follow.
```
>>> import numpy as np
>>> from sklearn.metrics import brier_score_loss
>>> y_true = np.array([0, 1, 1, 0])
>>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"])
>>> y_prob = np.array([0.1, 0.9, 0.8, 0.4])
>>> brier_score_loss(y_true, y_prob)
0.055
>>> brier_score_loss(y_true, 1 - y_prob, pos_label=0)
0.055
>>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham")
0.055
>>> brier_score_loss(
... ["eggs", "ham", "spam"],
... [[0.8, 0.1, 0.1], [0.2, 0.7, 0.1], [0.2, 0.2, 0.6]],
... labels=["eggs", "ham", "spam"],
... )
0.146
```
The Brier score can be used to assess how well a classifier is calibrated. However, a lower Brier score loss does not always mean a better calibration. This is because, by analogy with the bias-variance decomposition of the mean squared error, the Brier score loss can be decomposed as the sum of calibration loss and refinement loss [\[Bella2012\]](https://scikit-learn.org/stable/modules/model_evaluation.html#bella2012). Calibration loss is defined as the mean squared deviation from empirical probabilities derived from the slope of ROC segments. Refinement loss can be defined as the expected optimal loss as measured by the area under the optimal cost curve. Refinement loss can change independently from calibration loss, thus a lower Brier score loss does not necessarily mean a better calibrated model. āOnly when refinement loss remains the same does a lower Brier score loss always mean better calibrationā [\[Bella2012\]](https://scikit-learn.org/stable/modules/model_evaluation.html#bella2012), [\[Flach2008\]](https://scikit-learn.org/stable/modules/model_evaluation.html#flach2008).
Examples
- See [Probability calibration of classifiers](https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration.html#sphx-glr-auto-examples-calibration-plot-calibration-py) for an example of Brier score loss usage to perform probability calibration of classifiers.
References
\[[Brier1950](https://scikit-learn.org/stable/modules/model_evaluation.html#id47)\]
G. Brier, [Verification of forecasts expressed in terms of probability](ftp://ftp.library.noaa.gov/docs.lib/htdocs/rescue/mwr/078/mwr-078-01-0001.pdf), Monthly weather review 78.1 (1950)
\[Bella2012\] ([1](https://scikit-learn.org/stable/modules/model_evaluation.html#id48),[2](https://scikit-learn.org/stable/modules/model_evaluation.html#id49))
Bella, Ferri, HernĆ”ndez-Orallo, and RamĆrez-Quintana [āCalibration of Machine Learning Modelsā](http://dmip.webs.upv.es/papers/BFHRHandbook2010.pdf) in Khosrow-Pour, M. āMachine learning: concepts, methodologies, tools and applications.ā Hershey, PA: Information Science Reference (2012).
\[[Flach2008](https://scikit-learn.org/stable/modules/model_evaluation.html#id50)\]
Flach, Peter, and Edson Matsubara. [āOn classification, ranking, and probability estimation.ā](https://drops.dagstuhl.de/opus/volltexte/2008/1382/) Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum fĆ¼r Informatik (2008).
### 3\.4.4.19. Class likelihood ratios[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#class-likelihood-ratios "Link to this heading")
The [`class_likelihood_ratios`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.class_likelihood_ratios.html#sklearn.metrics.class_likelihood_ratios "sklearn.metrics.class_likelihood_ratios") function computes the [positive and negative likelihood ratios](https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing) L R Ā± for binary classes, which can be interpreted as the ratio of post-test to pre-test odds as explained below. As a consequence, this metric is invariant w.r.t. the class prevalence (the number of samples in the positive class divided by the total number of samples) and **can be extrapolated between populations regardless of any possible class imbalance.**
The L R Ā± metrics are therefore very useful in settings where the data available to learn and evaluate a classifier is a study population with nearly balanced classes, such as a case-control study, while the target application, i.e. the general population, has very low prevalence.
The positive likelihood ratio L R \+ is the probability of a classifier to correctly predict that a sample belongs to the positive class divided by the probability of predicting the positive class for a sample belonging to the negative class:
L
R
\+
\=
PR
(
P
\+
\|
T
\+
)
PR
(
P
\+
\|
T
ā
)
.
The notation here refers to predicted (P) or true (T) label and the sign \+ and ā refer to the positive and negative class, respectively, e.g. P \+ stands for āpredicted positiveā.
Analogously, the negative likelihood ratio L R ā is the probability of a sample of the positive class being classified as belonging to the negative class divided by the probability of a sample of the negative class being correctly classified:
L
R
ā
\=
PR
(
P
ā
\|
T
\+
)
PR
(
P
ā
\|
T
ā
)
.
For classifiers above chance L R \+ above 1 **higher is better**, while L R ā ranges from 0 to 1 and **lower is better**. Values of L R Ā± ā 1 correspond to chance level.
Notice that probabilities differ from counts, for instance PR ( P \+ \| T \+ ) is not equal to the number of true positive counts `tp` (see [the wikipedia page](https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing) for the actual formulas).
Examples
- [Class Likelihood Ratios to measure classification performance](https://scikit-learn.org/stable/auto_examples/model_selection/plot_likelihood_ratios.html#sphx-glr-auto-examples-model-selection-plot-likelihood-ratios-py)
Interpretation across varying prevalence[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#interpretation-across-varying-prevalence "Link to this dropdown")
Both class likelihood ratios are interpretable in terms of an odds ratio (pre-test and post-tests):
post-test odds
\=
Likelihood ratio
Ć
pre-test odds
.
Odds are in general related to probabilities via
odds
\=
probability
1
ā
probability
,
or equivalently
probability
\=
odds
1
\+
odds
.
On a given population, the pre-test probability is given by the prevalence. By converting odds to probabilities, the likelihood ratios can be translated into a probability of truly belonging to either class before and after a classifier prediction:
post-test odds
\=
Likelihood ratio
Ć
pre-test probability
1
ā
pre-test probability
,
post-test probability
\=
post-test odds
1
\+
post-test odds
.
Mathematical divergences[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mathematical-divergences "Link to this dropdown")
The positive likelihood ratio (`LR+`) is undefined when f p \= 0, meaning the classifier does not misclassify any negative labels as positives. This condition can either indicate a perfect identification of all the negative cases or, if there are also no true positive predictions (t p \= 0), that the classifier does not predict the positive class at all. In the first case, `LR+` can be interpreted as `np.inf`, in the second case (for instance, with highly imbalanced data) it can be interpreted as `np.nan`.
The negative likelihood ratio (`LR-`) is undefined when t n \= 0. Such divergence is invalid, as L R ā \> 1\.0 would indicate an increase in the odds of a sample belonging to the positive class after being classified as negative, as if the act of classifying caused the positive condition. This includes the case of a [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html#sklearn.dummy.DummyClassifier "sklearn.dummy.DummyClassifier") that always predicts the positive class (i.e. when t n \= f n \= 0).
Both class likelihood ratios (`LR+ and LR-`) are undefined when t p \= f n \= 0, which means that no samples of the positive class were present in the test set. This can happen when cross-validating on highly imbalanced data and also leads to a division by zero.
If a division by zero occurs and `raise_warning` is set to `True` (default), [`class_likelihood_ratios`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.class_likelihood_ratios.html#sklearn.metrics.class_likelihood_ratios "sklearn.metrics.class_likelihood_ratios") raises an `UndefinedMetricWarning` and returns `np.nan` by default to avoid pollution when averaging over cross-validation folds. Users can set return values in case of a division by zero with the `replace_undefined_by` param.
For a worked-out demonstration of the [`class_likelihood_ratios`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.class_likelihood_ratios.html#sklearn.metrics.class_likelihood_ratios "sklearn.metrics.class_likelihood_ratios") function, see the example below.
References[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#references "Link to this dropdown")
- [Wikipedia entry for Likelihood ratios in diagnostic testing](https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing)
- Brenner, H., & Gefeller, O. (1997). Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Statistics in medicine, 16(9), 981-991.
### 3\.4.4.20. DĀ² score for classification[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#d2-score-for-classification "Link to this heading")
The DĀ² score computes the fraction of deviance explained. It is a generalization of RĀ², where the squared error is generalized and replaced by a classification deviance of choice dev ( y , y ^ ) (e.g., Log loss, Brier score,). DĀ² is a form of a *skill score*. It is calculated as
D
2
(
y
,
y
^
)
\=
1
ā
dev
(
y
,
y
^
)
dev
(
y
,
y
null
)
.
Where y null is the optimal prediction of an intercept-only model (e.g., the per-class proportion of `y_true` in the case of the Log loss and Brier score).
Like RĀ², the best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts y null, disregarding the input features, would get a DĀ² score of 0.0.
D2 log loss score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#d2-log-loss-score "Link to this dropdown")
The [`d2_log_loss_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_log_loss_score.html#sklearn.metrics.d2_log_loss_score "sklearn.metrics.d2_log_loss_score") function implements the special case of DĀ² with the log loss, see [Log loss](https://scikit-learn.org/stable/modules/model_evaluation.html#log-loss), i.e.:
dev
(
y
,
y
^
)
\=
log\_loss
(
y
,
y
^
)
.
Here are some usage examples of the [`d2_log_loss_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_log_loss_score.html#sklearn.metrics.d2_log_loss_score "sklearn.metrics.d2_log_loss_score") function:
```
>>> from sklearn.metrics import d2_log_loss_score
>>> y_true = [1, 1, 2, 3]
>>> y_pred = [
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... ]
>>> d2_log_loss_score(y_true, y_pred)
0.0
>>> y_true = [1, 2, 3]
>>> y_pred = [
... [0.98, 0.01, 0.01],
... [0.01, 0.98, 0.01],
... [0.01, 0.01, 0.98],
... ]
>>> d2_log_loss_score(y_true, y_pred)
0.981
>>> y_true = [1, 2, 3]
>>> y_pred = [
... [0.1, 0.6, 0.3],
... [0.1, 0.6, 0.3],
... [0.4, 0.5, 0.1],
... ]
>>> d2_log_loss_score(y_true, y_pred)
-0.552
```
D2 Brier score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#d2-brier-score "Link to this dropdown")
The [`d2_brier_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_brier_score.html#sklearn.metrics.d2_brier_score "sklearn.metrics.d2_brier_score") function implements the special case of DĀ² with the Brier score, see [Brier score loss](https://scikit-learn.org/stable/modules/model_evaluation.html#brier-score-loss), i.e.:
dev
(
y
,
y
^
)
\=
brier\_score\_loss
(
y
,
y
^
)
.
This is also referred to as the Brier Skill Score (BSS).
Here are some usage examples of the [`d2_brier_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_brier_score.html#sklearn.metrics.d2_brier_score "sklearn.metrics.d2_brier_score") function:
```
>>> from sklearn.metrics import d2_brier_score
>>> y_true = [1, 1, 2, 3]
>>> y_pred = [
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... ]
>>> d2_brier_score(y_true, y_pred)
0.0
>>> y_true = [1, 2, 3]
>>> y_pred = [
... [0.98, 0.01, 0.01],
... [0.01, 0.98, 0.01],
... [0.01, 0.01, 0.98],
... ]
>>> d2_brier_score(y_true, y_pred)
0.9991
>>> y_true = [1, 2, 3]
>>> y_pred = [
... [0.1, 0.6, 0.3],
... [0.1, 0.6, 0.3],
... [0.4, 0.5, 0.1],
... ]
>>> d2_brier_score(y_true, y_pred)
-0.370...
```
## 3\.4.5. Multilabel ranking metrics[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multilabel-ranking-metrics "Link to this heading")
In multilabel learning, each sample can have any number of ground truth labels associated with it. The goal is to give high scores and better rank to the ground truth labels.
### 3\.4.5.1. Coverage error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#coverage-error "Link to this heading")
The [`coverage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.coverage_error.html#sklearn.metrics.coverage_error "sklearn.metrics.coverage_error") function computes the average number of labels that have to be included in the final prediction such that all true labels are predicted. This is useful if you want to know how many top-scored-labels you have to predict in average without missing any true one. The best value of this metric is thus the average number of true labels.
Note
Our implementationās score is 1 greater than the one given in Tsoumakas et al., 2010. This extends it to handle the degenerate case in which an instance has 0 true labels.
Formally, given a binary indicator matrix of the ground truth labels y ā { 0 , 1 } n samples Ć n labels and the score associated with each label f ^ ā R n samples Ć n labels, the coverage is defined as
c
o
v
e
r
a
g
e
(
y
,
f
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
max
j
:
y
i
j
\=
1
rank
i
j
with rank i j \= \| { k : f ^ i k ā„ f ^ i j } \|. Given the rank definition, ties in `y_scores` are broken by giving the maximal rank that would have been assigned to all tied values.
Here is a small example of usage of this function:
```
>>> import numpy as np
>>> from sklearn.metrics import coverage_error
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> coverage_error(y_true, y_score)
2.5
```
### 3\.4.5.2. Label ranking average precision[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#label-ranking-average-precision "Link to this heading")
The [`label_ranking_average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.label_ranking_average_precision_score.html#sklearn.metrics.label_ranking_average_precision_score "sklearn.metrics.label_ranking_average_precision_score") function implements label ranking average precision (LRAP). This metric is linked to the [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") function, but is based on the notion of label ranking instead of precision and recall.
Label ranking average precision (LRAP) averages over the samples the answer to the following question: for each ground truth label, what fraction of higher-ranked labels were true labels? This performance measure will be higher if you are able to give better rank to the labels associated with each sample. The obtained score is always strictly greater than 0, and the best value is 1. If there is exactly one relevant label per sample, label ranking average precision is equivalent to the [mean reciprocal rank](https://en.wikipedia.org/wiki/Mean_reciprocal_rank).
Formally, given a binary indicator matrix of the ground truth labels y ā { 0 , 1 } n samples Ć n labels and the score associated with each label f ^ ā R n samples Ć n labels, the average precision is defined as
L
R
A
P
(
y
,
f
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
1
\|
\|
y
i
\|
\|
0
ā
j
:
y
i
j
\=
1
\|
L
i
j
\|
rank
i
j
where L i j \= { k : y i k \= 1 , f ^ i k ā„ f ^ i j }, rank i j \= \| { k : f ^ i k ā„ f ^ i j } \|, \| ā
\| computes the cardinality of the set (i.e., the number of elements in the set), and \| \| ā
\| \| 0 is the ā 0 ānormā (which computes the number of nonzero elements in a vector).
Here is a small example of usage of this function:
```
>>> import numpy as np
>>> from sklearn.metrics import label_ranking_average_precision_score
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> label_ranking_average_precision_score(y_true, y_score)
0.416
```
### 3\.4.5.3. Ranking loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#ranking-loss "Link to this heading")
The [`label_ranking_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.label_ranking_loss.html#sklearn.metrics.label_ranking_loss "sklearn.metrics.label_ranking_loss") function computes the ranking loss which averages over the samples the number of label pairs that are incorrectly ordered, i.e. true labels have a lower score than false labels, weighted by the inverse of the number of ordered pairs of false and true labels. The lowest achievable ranking loss is zero.
Formally, given a binary indicator matrix of the ground truth labels y ā { 0 , 1 } n samples Ć n labels and the score associated with each label f ^ ā R n samples Ć n labels, the ranking loss is defined as
r
a
n
k
i
n
g
\_
l
o
s
s
(
y
,
f
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
1
\|
\|
y
i
\|
\|
0
(
n
labels
ā
\|
\|
y
i
\|
\|
0
)
\|
{
(
k
,
l
)
:
f
^
i
k
ā¤
f
^
i
l
,
y
i
k
\=
1
,
y
i
l
\=
0
}
\|
where \| ā
\| computes the cardinality of the set (i.e., the number of elements in the set) and \| \| ā
\| \| 0 is the ā 0 ānormā (which computes the number of nonzero elements in a vector).
Here is a small example of usage of this function:
```
>>> import numpy as np
>>> from sklearn.metrics import label_ranking_loss
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> label_ranking_loss(y_true, y_score)
0.75
>>> # With the following prediction, we have perfect and minimal loss
>>> y_score = np.array([[1.0, 0.1, 0.2], [0.1, 0.2, 0.9]])
>>> label_ranking_loss(y_true, y_score)
0.0
```
References[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#references-2 "Link to this dropdown")
- Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In Data mining and knowledge discovery handbook (pp. 667-685). Springer US.
### 3\.4.5.4. Normalized Discounted Cumulative Gain[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#normalized-discounted-cumulative-gain "Link to this heading")
Discounted Cumulative Gain (DCG) and Normalized Discounted Cumulative Gain (NDCG) are ranking metrics implemented in [`dcg_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.dcg_score.html#sklearn.metrics.dcg_score "sklearn.metrics.dcg_score") and [`ndcg_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html#sklearn.metrics.ndcg_score "sklearn.metrics.ndcg_score") ; they compare a predicted order to ground-truth scores, such as the relevance of answers to a query.
From the Wikipedia page for Discounted Cumulative Gain:
āDiscounted cumulative gain (DCG) is a measure of ranking quality. In information retrieval, it is often used to measure effectiveness of web search engine algorithms or related applications. Using a graded relevance scale of documents in a search-engine result set, DCG measures the usefulness, or gain, of a document based on its position in the result list. The gain is accumulated from the top of the result list to the bottom, with the gain of each result discounted at lower ranks.ā
DCG orders the true targets (e.g. relevance of query answers) in the predicted order, then multiplies them by a logarithmic decay and sums the result. The sum can be truncated after the first K results, in which case we call it DCG@K. NDCG, or NDCG@K is DCG divided by the DCG obtained by a perfect prediction, so that it is always between 0 and 1. Usually, NDCG is preferred to DCG.
Compared with the ranking loss, NDCG can take into account relevance scores, rather than a ground-truth ranking. So if the ground-truth consists only of an ordering, the ranking loss should be preferred; if the ground-truth consists of actual usefulness scores (e.g. 0 for irrelevant, 1 for relevant, 2 for very relevant), NDCG can be used.
For one sample, given the vector of continuous ground-truth values for each target y ā R M, where M is the number of outputs, and the prediction y ^, which induces the ranking function f, the DCG score is
ā
r
\=
1
min
(
K
,
M
)
y
f
(
r
)
log
ā”
(
1
\+
r
)
and the NDCG score is the DCG score divided by the DCG score obtained for y.
References[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#references-3 "Link to this dropdown")
- [Wikipedia entry for Discounted Cumulative Gain](https://en.wikipedia.org/wiki/Discounted_cumulative_gain)
- Jarvelin, K., & Kekalainen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422-446.
- Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013, May). A theoretical analysis of NDCG ranking measures. In Proceedings of the 26th Annual Conference on Learning Theory (COLT 2013)
- McSherry, F., & Najork, M. (2008, March). Computing information retrieval performance measures efficiently in the presence of tied scores. In European conference on information retrieval (pp. 414-421). Springer, Berlin, Heidelberg.
## 3\.4.6. Regression metrics[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics "Link to this heading")
The [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") module implements several loss, score, and utility functions to measure regression performance. Some of those have been enhanced to handle the multioutput case: [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error"), [`mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error"), [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score"), [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score"), [`mean_pinball_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_pinball_loss.html#sklearn.metrics.mean_pinball_loss "sklearn.metrics.mean_pinball_loss"), [`d2_pinball_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_pinball_score.html#sklearn.metrics.d2_pinball_score "sklearn.metrics.d2_pinball_score") and [`d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score").
These functions have a `multioutput` keyword argument which specifies the way the scores or losses for each individual target should be averaged. The default is `'uniform_average'`, which specifies a uniformly weighted mean over outputs. If an `ndarray` of shape `(n_outputs,)` is passed, then its entries are interpreted as weights and an according weighted average is returned. If `multioutput` is `'raw_values'`, then all unaltered individual scores or losses will be returned in an array of shape `(n_outputs,)`.
The [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") and [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") accept an additional value `'variance_weighted'` for the `multioutput` parameter. This option leads to a weighting of each individual score by the variance of the corresponding target variable. This setting quantifies the globally captured unscaled variance. If the target variables are of different scale, then this score puts more importance on explaining the higher variance variables.
### 3\.4.6.1. RĀ² score, the coefficient of determination[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score-the-coefficient-of-determination "Link to this heading")
The [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") function computes the [coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination), usually denoted as R 2.
It represents the proportion of variance (of y) that has been explained by the independent variables in the model. It provides an indication of goodness of fit and therefore a measure of how well unseen samples are likely to be predicted by the model, through the proportion of explained variance.
As such variance is dataset dependent, R 2 may not be meaningfully comparable across different datasets. Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected (average) value of y, disregarding the input features, would get an R 2 score of 0.0.
Note: when the prediction residuals have zero mean, the R 2 score and the [Explained variance score](https://scikit-learn.org/stable/modules/model_evaluation.html#explained-variance-score) are identical.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value for total n samples, the estimated R 2 is defined as:
R
2
(
y
,
y
^
)
\=
1
ā
ā
i
\=
1
n
(
y
i
ā
y
^
i
)
2
ā
i
\=
1
n
(
y
i
ā
y
ĀÆ
)
2
where y ĀÆ \= 1 n ā i \= 1 n y i and ā i \= 1 n ( y i ā y ^ i ) 2 \= ā i \= 1 n Ļµ i 2.
Note that [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") calculates unadjusted R 2 without correcting for bias in sample variance of y.
In the particular case where the true target is constant, the R 2 score is not finite: it is either `NaN` (perfect predictions) or `-Inf` (imperfect predictions). Such non-finite scores may prevent correct model optimization such as grid-search cross-validation to be performed correctly. For this reason the default behaviour of [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") is to replace them with 1.0 (perfect predictions) or 0.0 (imperfect predictions). If `force_finite` is set to `False`, this score falls back on the original R 2 definition.
Here is a small example of usage of the [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") function:
```
>>> from sklearn.metrics import r2_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> r2_score(y_true, y_pred)
0.948
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> r2_score(y_true, y_pred, multioutput='variance_weighted')
0.938
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> r2_score(y_true, y_pred, multioutput='uniform_average')
0.936
>>> r2_score(y_true, y_pred, multioutput='raw_values')
array([0.965, 0.908])
>>> r2_score(y_true, y_pred, multioutput=[0.3, 0.7])
0.925
>>> y_true = [-2, -2, -2]
>>> y_pred = [-2, -2, -2]
>>> r2_score(y_true, y_pred)
1.0
>>> r2_score(y_true, y_pred, force_finite=False)
nan
>>> y_true = [-2, -2, -2]
>>> y_pred = [-2, -2, -2 + 1e-8]
>>> r2_score(y_true, y_pred)
0.0
>>> r2_score(y_true, y_pred, force_finite=False)
-inf
```
Examples
- See [L1-based models for Sparse Signals](https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_and_elasticnet.html#sphx-glr-auto-examples-linear-model-plot-lasso-and-elasticnet-py) for an example of RĀ² score usage to evaluate Lasso and Elastic Net on sparse signals.
### 3\.4.6.2. Mean absolute error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-error "Link to this heading")
The [`mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error") function computes [mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error), a risk metric corresponding to the expected value of the absolute error loss or l 1\-norm loss.
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the mean absolute error (MAE) estimated over n samples is defined as
MAE
(
y
,
y
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
\|
y
i
ā
y
^
i
\|
.
Here is a small example of usage of the [`mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error") function:
```
>>> from sklearn.metrics import mean_absolute_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_absolute_error(y_true, y_pred)
0.5
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_absolute_error(y_true, y_pred)
0.75
>>> mean_absolute_error(y_true, y_pred, multioutput='raw_values')
array([0.5, 1. ])
>>> mean_absolute_error(y_true, y_pred, multioutput=[0.3, 0.7])
0.85
```
### 3\.4.6.3. Mean squared error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-error "Link to this heading")
The [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error") function computes [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error), a risk metric corresponding to the expected value of the squared (quadratic) error or loss.
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the mean squared error (MSE) estimated over n samples is defined as
MSE
(
y
,
y
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
(
y
i
ā
y
^
i
)
2
.
Here is a small example of usage of the [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error") function:
```
>>> from sklearn.metrics import mean_squared_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_squared_error(y_true, y_pred)
0.375
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_squared_error(y_true, y_pred)
0.7083
```
Examples
- See [Gradient Boosting regression](https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_regression.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-regression-py) for an example of mean squared error usage to evaluate gradient boosting regression.
Taking the square root of the MSE, called the root mean squared error (RMSE), is another common metric that provides a measure in the same units as the target variable. RMSE is available through the [`root_mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.root_mean_squared_error.html#sklearn.metrics.root_mean_squared_error "sklearn.metrics.root_mean_squared_error") function.
### 3\.4.6.4. Mean squared logarithmic error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-logarithmic-error "Link to this heading")
The [`mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_log_error.html#sklearn.metrics.mean_squared_log_error "sklearn.metrics.mean_squared_log_error") function computes a risk metric corresponding to the expected value of the squared logarithmic (quadratic) error or loss.
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the mean squared logarithmic error (MSLE) estimated over n samples is defined as
MSLE
(
y
,
y
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
(
log
e
ā”
(
1
\+
y
i
)
ā
log
e
ā”
(
1
\+
y
^
i
)
)
2
.
Where log e ā” ( x ) means the natural logarithm of x. This metric is best to use when targets having exponential growth, such as population counts, average sales of a commodity over a span of years etc. Note that this metric penalizes an under-predicted estimate greater than an over-predicted estimate.
Here is a small example of usage of the [`mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_log_error.html#sklearn.metrics.mean_squared_log_error "sklearn.metrics.mean_squared_log_error") function:
```
>>> from sklearn.metrics import mean_squared_log_error
>>> y_true = [3, 5, 2.5, 7]
>>> y_pred = [2.5, 5, 4, 8]
>>> mean_squared_log_error(y_true, y_pred)
0.0397
>>> y_true = [[0.5, 1], [1, 2], [7, 6]]
>>> y_pred = [[0.5, 2], [1, 2.5], [8, 8]]
>>> mean_squared_log_error(y_true, y_pred)
0.044
```
The root mean squared logarithmic error (RMSLE) is available through the [`root_mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.root_mean_squared_log_error.html#sklearn.metrics.root_mean_squared_log_error "sklearn.metrics.root_mean_squared_log_error") function.
### 3\.4.6.5. Mean absolute percentage error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error "Link to this heading")
The [`mean_absolute_percentage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html#sklearn.metrics.mean_absolute_percentage_error "sklearn.metrics.mean_absolute_percentage_error") (MAPE), also known as mean absolute percentage deviation (MAPD), is an evaluation metric for regression problems. The idea of this metric is to be sensitive to relative errors. It is for example not changed by a global scaling of the target variable.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value, then the mean absolute percentage error (MAPE) estimated over n samples is defined as
MAPE
(
y
,
y
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
\|
y
i
ā
y
^
i
\|
max
(
Ļµ
,
\|
y
i
\|
)
where Ļµ is an arbitrary small yet strictly positive number to avoid undefined results when y is zero.
The [`mean_absolute_percentage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html#sklearn.metrics.mean_absolute_percentage_error "sklearn.metrics.mean_absolute_percentage_error") function supports multioutput.
Here is a small example of usage of the [`mean_absolute_percentage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html#sklearn.metrics.mean_absolute_percentage_error "sklearn.metrics.mean_absolute_percentage_error") function:
```
>>> from sklearn.metrics import mean_absolute_percentage_error
>>> y_true = [1, 10, 1e6]
>>> y_pred = [0.9, 15, 1.2e6]
>>> mean_absolute_percentage_error(y_true, y_pred)
0.2666
```
In above example, if we had used `mean_absolute_error`, it would have ignored the small magnitude values and only reflected the error in prediction of highest magnitude value. But that problem is resolved in case of MAPE because it calculates relative percentage error with respect to actual output.
Note
The MAPE formula here does not represent the common āpercentageā definition: the percentage in the range \[0, 100\] is converted to a relative value in the range \[0, 1\] by dividing by 100. Thus, an error of 200% corresponds to a relative error of 2. The motivation here is to have a range of values that is more consistent with other error metrics in scikit-learn, such as `accuracy_score`.
To obtain the mean absolute percentage error as per the Wikipedia formula, multiply the `mean_absolute_percentage_error` computed here by 100.
References[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#references-4 "Link to this dropdown")
- [Wikipedia entry for Mean Absolute Percentage Error](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error)
### 3\.4.6.6. Median absolute error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#median-absolute-error "Link to this heading")
The [`median_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.median_absolute_error.html#sklearn.metrics.median_absolute_error "sklearn.metrics.median_absolute_error") is particularly interesting because it is robust to outliers. The loss is calculated by taking the median of all absolute differences between the target and the prediction.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value, then the median absolute error (MedAE) estimated over n samples is defined as
MedAE
(
y
,
y
^
)
\=
median
(
ā£
y
1
ā
y
^
1
ā£
,
ā¦
,
ā£
y
n
ā
y
^
n
ā£
)
.
The [`median_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.median_absolute_error.html#sklearn.metrics.median_absolute_error "sklearn.metrics.median_absolute_error") does not support multioutput.
Here is a small example of usage of the [`median_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.median_absolute_error.html#sklearn.metrics.median_absolute_error "sklearn.metrics.median_absolute_error") function:
```
>>> from sklearn.metrics import median_absolute_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> median_absolute_error(y_true, y_pred)
0.5
```
### 3\.4.6.7. Max error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#max-error "Link to this heading")
The [`max_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.max_error.html#sklearn.metrics.max_error "sklearn.metrics.max_error") function computes the maximum [residual error](https://en.wikipedia.org/wiki/Errors_and_residuals) , a metric that captures the worst case error between the predicted value and the true value. In a perfectly fitted single output regression model, `max_error` would be `0` on the training set and though this would be highly unlikely in the real world, this metric shows the extent of error that the model had when it was fitted.
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the max error is defined as
Max Error
(
y
,
y
^
)
\=
max
(
\|
y
i
ā
y
^
i
\|
)
Here is a small example of usage of the [`max_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.max_error.html#sklearn.metrics.max_error "sklearn.metrics.max_error") function:
```
>>> from sklearn.metrics import max_error
>>> y_true = [3, 2, 7, 1]
>>> y_pred = [9, 2, 7, 1]
>>> max_error(y_true, y_pred)
6.0
```
The [`max_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.max_error.html#sklearn.metrics.max_error "sklearn.metrics.max_error") does not support multioutput.
### 3\.4.6.8. Explained variance score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#explained-variance-score "Link to this heading")
The [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") computes the [explained variance regression score](https://en.wikipedia.org/wiki/Explained_variation).
If y ^ is the estimated target output, y the corresponding (correct) target output, and V a r is [Variance](https://en.wikipedia.org/wiki/Variance), the square of the standard deviation, then the explained variance is estimated as follow:
e
x
p
l
a
i
n
e
d
\_
v
a
r
i
a
n
c
e
(
y
,
y
^
)
\=
1
ā
V
a
r
{
y
ā
y
^
}
V
a
r
{
y
}
The best possible score is 1.0, lower values are worse.
Link to [RĀ² score, the coefficient of determination](https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score)
The difference between the explained variance score and the [RĀ² score, the coefficient of determination](https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score) is that the explained variance score does not account for systematic offset in the prediction. For this reason, the [RĀ² score, the coefficient of determination](https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score) should be preferred in general.
In the particular case where the true target is constant, the Explained Variance score is not finite: it is either `NaN` (perfect predictions) or `-Inf` (imperfect predictions). Such non-finite scores may prevent correct model optimization such as grid-search cross-validation to be performed correctly. For this reason the default behaviour of [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") is to replace them with 1.0 (perfect predictions) or 0.0 (imperfect predictions). You can set the `force_finite` parameter to `False` to prevent this fix from happening and fallback on the original Explained Variance score.
Here is a small example of usage of the [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") function:
```
>>> from sklearn.metrics import explained_variance_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> explained_variance_score(y_true, y_pred)
0.957
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> explained_variance_score(y_true, y_pred, multioutput='raw_values')
array([0.967, 1. ])
>>> explained_variance_score(y_true, y_pred, multioutput=[0.3, 0.7])
0.990
>>> y_true = [-2, -2, -2]
>>> y_pred = [-2, -2, -2]
>>> explained_variance_score(y_true, y_pred)
1.0
>>> explained_variance_score(y_true, y_pred, force_finite=False)
nan
>>> y_true = [-2, -2, -2]
>>> y_pred = [-2, -2, -2 + 1e-8]
>>> explained_variance_score(y_true, y_pred)
0.0
>>> explained_variance_score(y_true, y_pred, force_finite=False)
-inf
```
### 3\.4.6.9. Mean Poisson, Gamma, and Tweedie deviances[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-poisson-gamma-and-tweedie-deviances "Link to this heading")
The [`mean_tweedie_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_tweedie_deviance.html#sklearn.metrics.mean_tweedie_deviance "sklearn.metrics.mean_tweedie_deviance") function computes the [mean Tweedie deviance error](https://en.wikipedia.org/wiki/Tweedie_distribution#The_Tweedie_deviance) with a `power` parameter (p). This is a metric that elicits predicted expectation values of regression targets.
Following special cases exist,
- when `power=0` it is equivalent to [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error").
- when `power=1` it is equivalent to [`mean_poisson_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_poisson_deviance.html#sklearn.metrics.mean_poisson_deviance "sklearn.metrics.mean_poisson_deviance").
- when `power=2` it is equivalent to [`mean_gamma_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_gamma_deviance.html#sklearn.metrics.mean_gamma_deviance "sklearn.metrics.mean_gamma_deviance").
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the mean Tweedie deviance error (D) for power p, estimated over n samples is defined as
D
(
y
,
y
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
{
(
y
i
ā
y
^
i
)
2
,
for
p
\=
0
(Normal)
2
(
y
i
log
ā”
(
y
i
/
y
^
i
)
\+
y
^
i
ā
y
i
)
,
for
p
\=
1
(Poisson)
2
(
log
ā”
(
y
^
i
/
y
i
)
\+
y
i
/
y
^
i
ā
1
)
,
for
p
\=
2
(Gamma)
2
(
max
(
y
i
,
0
)
2
ā
p
(
1
ā
p
)
(
2
ā
p
)
ā
y
i
y
^
i
1
ā
p
1
ā
p
\+
y
^
i
2
ā
p
2
ā
p
)
,
otherwise
Tweedie deviance is a homogeneous function of degree `2-power`. Thus, Gamma distribution with `power=2` means that simultaneously scaling `y_true` and `y_pred` has no effect on the deviance. For Poisson distribution `power=1` the deviance scales linearly, and for Normal distribution (`power=0`), quadratically. In general, the higher `power` the less weight is given to extreme deviations between true and predicted targets.
For instance, letās compare the two predictions 1.5 and 150 that are both 50% larger than their corresponding true value.
The mean squared error (`power=0`) is very sensitive to the prediction difference of the second point,:
```
>>> from sklearn.metrics import mean_tweedie_deviance
>>> mean_tweedie_deviance([1.0], [1.5], power=0)
0.25
>>> mean_tweedie_deviance([100.], [150.], power=0)
2500.0
```
If we increase `power` to 1,:
```
>>> mean_tweedie_deviance([1.0], [1.5], power=1)
0.189
>>> mean_tweedie_deviance([100.], [150.], power=1)
18.9
```
the difference in errors decreases. Finally, by setting, `power=2`:
```
>>> mean_tweedie_deviance([1.0], [1.5], power=2)
0.144
>>> mean_tweedie_deviance([100.], [150.], power=2)
0.144
```
we would get identical errors. The deviance when `power=2` is thus only sensitive to relative errors.
### 3\.4.6.10. Pinball loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss "Link to this heading")
The [`mean_pinball_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_pinball_loss.html#sklearn.metrics.mean_pinball_loss "sklearn.metrics.mean_pinball_loss") function is used to evaluate the predictive performance of [quantile regression](https://en.wikipedia.org/wiki/Quantile_regression) models.
pinball
(
y
,
y
^
)
\=
1
n
samples
ā
i
\=
0
n
samples
ā
1
Ī±
max
(
y
i
ā
y
^
i
,
0
)
\+
(
1
ā
Ī±
)
max
(
y
^
i
ā
y
i
,
0
)
The value of pinball loss is equivalent to half of [`mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error") when the quantile parameter `alpha` is set to 0.5.
Here is a small example of usage of the [`mean_pinball_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_pinball_loss.html#sklearn.metrics.mean_pinball_loss "sklearn.metrics.mean_pinball_loss") function:
```
>>> from sklearn.metrics import mean_pinball_loss
>>> y_true = [1, 2, 3]
>>> mean_pinball_loss(y_true, [0, 2, 3], alpha=0.1)
0.033
>>> mean_pinball_loss(y_true, [1, 2, 4], alpha=0.1)
0.3
>>> mean_pinball_loss(y_true, [0, 2, 3], alpha=0.9)
0.3
>>> mean_pinball_loss(y_true, [1, 2, 4], alpha=0.9)
0.033
>>> mean_pinball_loss(y_true, y_true, alpha=0.1)
0.0
>>> mean_pinball_loss(y_true, y_true, alpha=0.9)
0.0
```
It is possible to build a scorer object with a specific choice of `alpha`:
```
>>> from sklearn.metrics import make_scorer
>>> mean_pinball_loss_95p = make_scorer(mean_pinball_loss, alpha=0.95)
```
Such a scorer can be used to evaluate the generalization performance of a quantile regressor via cross-validation:
```
>>> from sklearn.datasets import make_regression
>>> from sklearn.model_selection import cross_val_score
>>> from sklearn.ensemble import GradientBoostingRegressor
>>>
>>> X, y = make_regression(n_samples=100, random_state=0)
>>> estimator = GradientBoostingRegressor(
... loss="quantile",
... alpha=0.95,
... random_state=0,
... )
>>> cross_val_score(estimator, X, y, cv=5, scoring=mean_pinball_loss_95p)
array([13.6, 9.7, 23.3, 9.5, 10.4])
```
It is also possible to build scorer objects for hyper-parameter tuning. The sign of the loss must be switched to ensure that greater means better as explained in the example linked below.
Examples
- See [Prediction Intervals for Gradient Boosting Regression](https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_quantile.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-quantile-py) for an example of using the pinball loss to evaluate and tune the hyper-parameters of quantile regression models on data with non-symmetric noise and outliers.
### 3\.4.6.11. DĀ² score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#d2-score "Link to this heading")
The DĀ² score computes the fraction of deviance explained. It is a generalization of RĀ², where the squared error is generalized and replaced by a deviance of choice dev ( y , y ^ ) (e.g., Tweedie, pinball or mean absolute error). DĀ² is a form of a *skill score*. It is calculated as
D
2
(
y
,
y
^
)
\=
1
ā
dev
(
y
,
y
^
)
dev
(
y
,
y
null
)
.
Where y null is the optimal prediction of an intercept-only model (e.g., the mean of `y_true` for the Tweedie case, the median for absolute error and the alpha-quantile for pinball loss).
Like RĀ², the best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts y null, disregarding the input features, would get a DĀ² score of 0.0.
DĀ² Tweedie score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#d%C2%B2-tweedie-score "Link to this dropdown")
The [`d2_tweedie_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_tweedie_score.html#sklearn.metrics.d2_tweedie_score "sklearn.metrics.d2_tweedie_score") function implements the special case of DĀ² where dev ( y , y ^ ) is the Tweedie deviance, see [Mean Poisson, Gamma, and Tweedie deviances](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance). It is also known as DĀ² Tweedie and is related to McFaddenās likelihood ratio index.
The argument `power` defines the Tweedie power as for [`mean_tweedie_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_tweedie_deviance.html#sklearn.metrics.mean_tweedie_deviance "sklearn.metrics.mean_tweedie_deviance"). Note that for `power=0`, [`d2_tweedie_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_tweedie_score.html#sklearn.metrics.d2_tweedie_score "sklearn.metrics.d2_tweedie_score") equals [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") (for single targets).
A scorer object with a specific choice of `power` can be built by:
```
>>> from sklearn.metrics import d2_tweedie_score, make_scorer
>>> d2_tweedie_score_15 = make_scorer(d2_tweedie_score, power=1.5)
```
DĀ² pinball score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#d%C2%B2-pinball-score "Link to this dropdown")
The [`d2_pinball_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_pinball_score.html#sklearn.metrics.d2_pinball_score "sklearn.metrics.d2_pinball_score") function implements the special case of DĀ² with the pinball loss, see [Pinball loss](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss), i.e.:
dev
(
y
,
y
^
)
\=
pinball
(
y
,
y
^
)
.
The argument `alpha` defines the slope of the pinball loss as for [`mean_pinball_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_pinball_loss.html#sklearn.metrics.mean_pinball_loss "sklearn.metrics.mean_pinball_loss") ([Pinball loss](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss)). It determines the quantile level `alpha` for which the pinball loss and also DĀ² are optimal. Note that for `alpha=0.5` (the default) [`d2_pinball_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_pinball_score.html#sklearn.metrics.d2_pinball_score "sklearn.metrics.d2_pinball_score") equals [`d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score").
A scorer object with a specific choice of `alpha` can be built by:
```
>>> from sklearn.metrics import d2_pinball_score, make_scorer
>>> d2_pinball_score_08 = make_scorer(d2_pinball_score, alpha=0.8)
```
DĀ² absolute error score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#d%C2%B2-absolute-error-score "Link to this dropdown")
The [`d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score") function implements the special case of the [Mean absolute error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-error):
dev
(
y
,
y
^
)
\=
MAE
(
y
,
y
^
)
.
Here are some usage examples of the [`d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score") function:
```
>>> from sklearn.metrics import d2_absolute_error_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> d2_absolute_error_score(y_true, y_pred)
0.764
>>> y_true = [1, 2, 3]
>>> y_pred = [1, 2, 3]
>>> d2_absolute_error_score(y_true, y_pred)
1.0
>>> y_true = [1, 2, 3]
>>> y_pred = [2, 2, 2]
>>> d2_absolute_error_score(y_true, y_pred)
0.0
```
### 3\.4.6.12. Visual evaluation of regression models[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#visual-evaluation-of-regression-models "Link to this heading")
Among methods to assess the quality of regression models, scikit-learn provides the [`PredictionErrorDisplay`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.PredictionErrorDisplay.html#sklearn.metrics.PredictionErrorDisplay "sklearn.metrics.PredictionErrorDisplay") class. It allows to visually inspect the prediction errors of a model in two different manners.
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_predict.html)
The plot on the left shows the actual values vs predicted values. For a noise-free regression task aiming to predict the (conditional) expectation of `y`, a perfect regression model would display data points on the diagonal defined by predicted equal to actual values. The further away from this optimal line, the larger the error of the model. In a more realistic setting with irreducible noise, that is, when not all the variations of `y` can be explained by features in `X`, then the best model would lead to a cloud of points densely arranged around the diagonal.
Note that the above only holds when the predicted values is the expected value of `y` given `X`. This is typically the case for regression models that minimize the mean squared error objective function or more generally the [mean Tweedie deviance](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance) for any value of its āpowerā parameter.
When plotting the predictions of an estimator that predicts a quantile of `y` given `X`, e.g. [`QuantileRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.QuantileRegressor.html#sklearn.linear_model.QuantileRegressor "sklearn.linear_model.QuantileRegressor") or any other model minimizing the [pinball loss](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss), a fraction of the points are either expected to lie above or below the diagonal depending on the estimated quantile level.
All in all, while intuitive to read, this plot does not really inform us on what to do to obtain a better model.
The right-hand side plot shows the residuals (i.e. the difference between the actual and the predicted values) vs. the predicted values.
This plot makes it easier to visualize if the residuals follow and [homoscedastic or heteroschedastic](https://en.wikipedia.org/wiki/Homoscedasticity_and_heteroscedasticity) distribution.
In particular, if the true distribution of `y|X` is Poisson or Gamma distributed, it is expected that the variance of the residuals of the optimal model would grow with the predicted value of `E[y|X]` (either linearly for Poisson or quadratically for Gamma).
When fitting a linear least squares regression model (see [`LinearRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression "sklearn.linear_model.LinearRegression") and [`Ridge`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge "sklearn.linear_model.Ridge")), we can use this plot to check if some of the [model assumptions](https://en.wikipedia.org/wiki/Ordinary_least_squares#Assumptions) are met, in particular that the residuals should be uncorrelated, their expected value should be null and that their variance should be constant (homoschedasticity).
If this is not the case, and in particular if the residuals plot show some banana-shaped structure, this is a hint that the model is likely mis-specified and that non-linear feature engineering or switching to a non-linear regression model might be useful.
Refer to the example below to see a model evaluation that makes use of this display.
Examples
- See [Effect of transforming the targets in regression model](https://scikit-learn.org/stable/auto_examples/compose/plot_transformed_target.html#sphx-glr-auto-examples-compose-plot-transformed-target-py) for an example on how to use [`PredictionErrorDisplay`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.PredictionErrorDisplay.html#sklearn.metrics.PredictionErrorDisplay "sklearn.metrics.PredictionErrorDisplay") to visualize the prediction quality improvement of a regression model obtained by transforming the target before learning.
## 3\.4.7. Clustering metrics[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#clustering-metrics "Link to this heading")
The [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") module implements several loss, score, and utility functions to measure clustering performance. For more information see the [Clustering performance evaluation](https://scikit-learn.org/stable/modules/clustering.html#clustering-evaluation) section for instance clustering, and [Biclustering evaluation](https://scikit-learn.org/stable/modules/biclustering.html#biclustering-evaluation) for biclustering.
## 3\.4.8. Dummy estimators[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators "Link to this heading")
When doing supervised learning, a simple sanity check consists of comparing oneās estimator against simple rules of thumb. [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html#sklearn.dummy.DummyClassifier "sklearn.dummy.DummyClassifier") implements several such simple strategies for classification:
- `stratified` generates random predictions by respecting the training set class distribution.
- `most_frequent` always predicts the most frequent label in the training set.
- `prior` always predicts the class that maximizes the class prior (like `most_frequent`) and `predict_proba` returns the class prior.
- `uniform` generates predictions uniformly at random.
- `constant` always predicts a constant label that is provided by the user.
A major motivation of this method is F1-scoring, when the positive class is in the minority.
Note that with all these strategies, the `predict` method completely ignores the input data\!
To illustrate [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html#sklearn.dummy.DummyClassifier "sklearn.dummy.DummyClassifier"), first letās create an imbalanced dataset:
```
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> X, y = load_iris(return_X_y=True)
>>> y[y != 1] = -1
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
```
Next, letās compare the accuracy of `SVC` and `most_frequent`:
```
>>> from sklearn.dummy import DummyClassifier
>>> from sklearn.svm import SVC
>>> clf = SVC(kernel='linear', C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test)
0.63
>>> clf = DummyClassifier(strategy='most_frequent', random_state=0)
>>> clf.fit(X_train, y_train)
DummyClassifier(random_state=0, strategy='most_frequent')
>>> clf.score(X_test, y_test)
0.579
```
We see that `SVC` doesnāt do much better than a dummy classifier. Now, letās change the kernel:
```
>>> clf = SVC(kernel='rbf', C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test)
0.94
```
We see that the accuracy was boosted to almost 100%. A cross validation strategy is recommended for a better estimate of the accuracy, if it is not too CPU costly. For more information see the [Cross-validation: evaluating estimator performance](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation) section. Moreover if you want to optimize over the parameter space, it is highly recommended to use an appropriate methodology; see the [Tuning the hyper-parameters of an estimator](https://scikit-learn.org/stable/modules/grid_search.html#grid-search) section for details.
More generally, when the accuracy of a classifier is too close to random, it probably means that something went wrong: features are not helpful, a hyperparameter is not correctly tuned, the classifier is suffering from class imbalance, etcā¦
[`DummyRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html#sklearn.dummy.DummyRegressor "sklearn.dummy.DummyRegressor") also implements four simple rules of thumb for regression:
- `mean` always predicts the mean of the training targets.
- `median` always predicts the median of the training targets.
- `quantile` always predicts a user provided quantile of the training targets.
- `constant` always predicts a constant value that is provided by the user.
In all these strategies, the `predict` method completely ignores the input data.
[previous 3.3. Tuning the decision threshold for class prediction](https://scikit-learn.org/stable/modules/classification_threshold.html "previous page")
[next 3.5. Validation curves: plotting scores to evaluate models](https://scikit-learn.org/stable/modules/learning_curve.html "next page")
On this page
- [3\.4.1. Which scoring function should I use?](https://scikit-learn.org/stable/modules/model_evaluation.html#which-scoring-function-should-i-use)
- [3\.4.2. Scoring API overview](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-api-overview)
- [3\.4.3. The `scoring` parameter: defining model evaluation rules](https://scikit-learn.org/stable/modules/model_evaluation.html#the-scoring-parameter-defining-model-evaluation-rules)
- [3\.4.3.1. String name scorers](https://scikit-learn.org/stable/modules/model_evaluation.html#string-name-scorers)
- [3\.4.3.2. Callable scorers](https://scikit-learn.org/stable/modules/model_evaluation.html#callable-scorers)
- [3\.4.3.2.1. Adapting predefined metrics via `make_scorer`](https://scikit-learn.org/stable/modules/model_evaluation.html#adapting-predefined-metrics-via-make-scorer)
- [3\.4.3.2.2. Creating a custom scorer object](https://scikit-learn.org/stable/modules/model_evaluation.html#creating-a-custom-scorer-object)
- [3\.4.3.3. Using multiple metric evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#using-multiple-metric-evaluation)
- [3\.4.4. Classification metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics)
- [3\.4.4.1. From binary to multiclass and multilabel](https://scikit-learn.org/stable/modules/model_evaluation.html#from-binary-to-multiclass-and-multilabel)
- [3\.4.4.2. Accuracy score](https://scikit-learn.org/stable/modules/model_evaluation.html#accuracy-score)
- [3\.4.4.3. Top-k accuracy score](https://scikit-learn.org/stable/modules/model_evaluation.html#top-k-accuracy-score)
- [3\.4.4.4. Balanced accuracy score](https://scikit-learn.org/stable/modules/model_evaluation.html#balanced-accuracy-score)
- [3\.4.4.5. Cohenās kappa](https://scikit-learn.org/stable/modules/model_evaluation.html#cohen-s-kappa)
- [3\.4.4.6. Confusion matrix](https://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix)
- [3\.4.4.7. Classification report](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-report)
- [3\.4.4.8. Hamming loss](https://scikit-learn.org/stable/modules/model_evaluation.html#hamming-loss)
- [3\.4.4.9. Precision, recall and F-measures](https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-and-f-measures)
- [3\.4.4.9.1. Binary classification](https://scikit-learn.org/stable/modules/model_evaluation.html#binary-classification)
- [3\.4.4.9.2. Multiclass and multilabel classification](https://scikit-learn.org/stable/modules/model_evaluation.html#multiclass-and-multilabel-classification)
- [3\.4.4.10. Jaccard similarity coefficient score](https://scikit-learn.org/stable/modules/model_evaluation.html#jaccard-similarity-coefficient-score)
- [3\.4.4.11. Hinge loss](https://scikit-learn.org/stable/modules/model_evaluation.html#hinge-loss)
- [3\.4.4.12. Log loss](https://scikit-learn.org/stable/modules/model_evaluation.html#log-loss)
- [3\.4.4.13. Matthews correlation coefficient](https://scikit-learn.org/stable/modules/model_evaluation.html#matthews-correlation-coefficient)
- [3\.4.4.14. Multi-label confusion matrix](https://scikit-learn.org/stable/modules/model_evaluation.html#multi-label-confusion-matrix)
- [3\.4.4.15. Receiver operating characteristic (ROC)](https://scikit-learn.org/stable/modules/model_evaluation.html#receiver-operating-characteristic-roc)
- [3\.4.4.15.1. Binary case](https://scikit-learn.org/stable/modules/model_evaluation.html#binary-case)
- [3\.4.4.15.2. Multi-class case](https://scikit-learn.org/stable/modules/model_evaluation.html#multi-class-case)
- [3\.4.4.15.3. Multi-label case](https://scikit-learn.org/stable/modules/model_evaluation.html#multi-label-case)
- [3\.4.4.16. Detection error tradeoff (DET)](https://scikit-learn.org/stable/modules/model_evaluation.html#detection-error-tradeoff-det)
- [3\.4.4.17. Zero one loss](https://scikit-learn.org/stable/modules/model_evaluation.html#zero-one-loss)
- [3\.4.4.18. Brier score loss](https://scikit-learn.org/stable/modules/model_evaluation.html#brier-score-loss)
- [3\.4.4.19. Class likelihood ratios](https://scikit-learn.org/stable/modules/model_evaluation.html#class-likelihood-ratios)
- [3\.4.4.20. DĀ² score for classification](https://scikit-learn.org/stable/modules/model_evaluation.html#d2-score-for-classification)
- [3\.4.5. Multilabel ranking metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#multilabel-ranking-metrics)
- [3\.4.5.1. Coverage error](https://scikit-learn.org/stable/modules/model_evaluation.html#coverage-error)
- [3\.4.5.2. Label ranking average precision](https://scikit-learn.org/stable/modules/model_evaluation.html#label-ranking-average-precision)
- [3\.4.5.3. Ranking loss](https://scikit-learn.org/stable/modules/model_evaluation.html#ranking-loss)
- [3\.4.5.4. Normalized Discounted Cumulative Gain](https://scikit-learn.org/stable/modules/model_evaluation.html#normalized-discounted-cumulative-gain)
- [3\.4.6. Regression metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics)
- [3\.4.6.1. RĀ² score, the coefficient of determination](https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score-the-coefficient-of-determination)
- [3\.4.6.2. Mean absolute error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-error)
- [3\.4.6.3. Mean squared error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-error)
- [3\.4.6.4. Mean squared logarithmic error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-logarithmic-error)
- [3\.4.6.5. Mean absolute percentage error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error)
- [3\.4.6.6. Median absolute error](https://scikit-learn.org/stable/modules/model_evaluation.html#median-absolute-error)
- [3\.4.6.7. Max error](https://scikit-learn.org/stable/modules/model_evaluation.html#max-error)
- [3\.4.6.8. Explained variance score](https://scikit-learn.org/stable/modules/model_evaluation.html#explained-variance-score)
- [3\.4.6.9. Mean Poisson, Gamma, and Tweedie deviances](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-poisson-gamma-and-tweedie-deviances)
- [3\.4.6.10. Pinball loss](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss)
- [3\.4.6.11. DĀ² score](https://scikit-learn.org/stable/modules/model_evaluation.html#d2-score)
- [3\.4.6.12. Visual evaluation of regression models](https://scikit-learn.org/stable/modules/model_evaluation.html#visual-evaluation-of-regression-models)
- [3\.4.7. Clustering metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#clustering-metrics)
- [3\.4.8. Dummy estimators](https://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators)
### This Page
- [Show Source](https://scikit-learn.org/stable/_sources/modules/model_evaluation.rst.txt)
Ā© Copyright 2007 - 2026, scikit-learn developers (BSD License). |
| Readable Markdown | ## 3\.4.1. Which scoring function should I use?[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#which-scoring-function-should-i-use "Link to this heading")
Before we take a closer look into the details of the many scores and [evaluation metrics](https://scikit-learn.org/stable/glossary.html#term-evaluation-metrics), we want to give some guidance, inspired by statistical decision theory, on the choice of **scoring functions** for **supervised learning**, see [\[Gneiting2009\]](https://scikit-learn.org/stable/modules/model_evaluation.html#gneiting2009):
- *Which scoring function should I use?*
- *Which scoring function is a good one for my task?*
In a nutshell, if the scoring function is given, e.g. in a kaggle competition or in a business context, use that one. If you are free to choose, it starts by considering the ultimate goal and application of the prediction. It is useful to distinguish two steps:
- Predicting
- Decision making
**Predicting:** Usually, the response variable Y is a random variable, in the sense that there is *no deterministic* function Y \= g ( X ) of the features X. Instead, there is a probability distribution F of Y. One can aim to predict the whole distribution, known as *probabilistic prediction*, orāmore the focus of scikit-learnāissue a *point prediction* (or point forecast) by choosing a property or functional of that distribution F. Typical examples are the mean (expected value), the median or a quantile of the response variable Y (conditionally on X).
Once that is settled, use a **strictly consistent** scoring function for that (target) functional, see [\[Gneiting2009\]](https://scikit-learn.org/stable/modules/model_evaluation.html#gneiting2009). This means using a scoring function that is aligned with *measuring the distance between predictions* `y_pred` *and the true target functional using observations of* Y, i.e. `y_true`. For classification **strictly proper scoring rules**, see [Wikipedia entry for Scoring rule](https://en.wikipedia.org/wiki/Scoring_rule) and [\[Gneiting2007\]](https://scikit-learn.org/stable/modules/model_evaluation.html#gneiting2007), coincide with strictly consistent scoring functions. The table further below provides examples. One could say that consistent scoring functions act as *truth serum* in that they guarantee *āthat truth telling \[ā¦\] is an optimal strategy in expectationā* [\[Gneiting2014\]](https://scikit-learn.org/stable/modules/model_evaluation.html#gneiting2014).
Once a strictly consistent scoring function is chosen, it is best used for both: as loss function for model training and as metric/score in model evaluation and model comparison.
Note that for regressors, the prediction is done with [predict](https://scikit-learn.org/stable/glossary.html#term-predict) while for classifiers it is usually [predict\_proba](https://scikit-learn.org/stable/glossary.html#term-predict_proba).
**Decision Making:** The most common decisions are done on binary classification tasks, where the result of [predict\_proba](https://scikit-learn.org/stable/glossary.html#term-predict_proba) is turned into a single outcome, e.g., from the predicted probability of rain a decision is made on how to act (whether to take mitigating measures like an umbrella or not). For classifiers, this is what [predict](https://scikit-learn.org/stable/glossary.html#term-predict) returns. See also [Tuning the decision threshold for class prediction](https://scikit-learn.org/stable/modules/classification_threshold.html#tunedthresholdclassifiercv). There are many scoring functions which measure different aspects of such a decision, most of them are covered with or derived from the [`metrics.confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix "sklearn.metrics.confusion_matrix").
**List of strictly consistent scoring functions:** Here, we list some of the most relevant statistical functionals and corresponding strictly consistent scoring functions for tasks in practice. Note that the list is not complete and that there are more of them. For further criteria on how to select a specific one, see [\[Fissler2022\]](https://scikit-learn.org/stable/modules/model_evaluation.html#fissler2022).
| functional | scoring or loss function | response `y` | prediction |
|---|---|---|---|
| **Classification** | | | |
| mean | [Brier score](https://scikit-learn.org/stable/modules/model_evaluation.html#brier-score-loss) 1 | multi-class | `predict_proba` |
| mean | [log loss](https://scikit-learn.org/stable/modules/model_evaluation.html#log-loss) | multi-class | `predict_proba` |
| mode | [zero-one loss](https://scikit-learn.org/stable/modules/model_evaluation.html#zero-one-loss) 2 | multi-class | `predict`, categorical |
| **Regression** | | | |
| mean | [squared error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-error) 3 | all reals | `predict`, all reals |
| mean | [Poisson deviance](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance) | non-negative | `predict`, strictly positive |
| mean | [Gamma deviance](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance) | strictly positive | `predict`, strictly positive |
| mean | [Tweedie deviance](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance) | depends on `power` | `predict`, depends on `power` |
| median | [absolute error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-error) | all reals | `predict`, all reals |
| quantile | [pinball loss](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss) | all reals | `predict`, all reals |
| mode | no consistent one exists | reals | |
1 The Brier score is just a different name for the squared error in case of classification with one-hot encoded targets.
2 The zero-one loss is only consistent but not strictly consistent for the mode. The zero-one loss is equivalent to one minus the accuracy score, meaning it gives different score values but the same ranking.
3 RĀ² gives the same ranking as squared error.
**Fictitious Example:** Letās make the above arguments more tangible. Consider a setting in network reliability engineering, such as maintaining stable internet or Wi-Fi connections. As provider of the network, you have access to the dataset of log entries of network connections containing network load over time and many interesting features. Your goal is to improve the reliability of the connections. In fact, you promise your customers that on at least 99% of all days there are no connection discontinuities larger than 1 minute. Therefore, you are interested in a prediction of the 99% quantile (of longest connection interruption duration per day) in order to know in advance when to add more bandwidth and thereby satisfy your customers. So the *target functional* is the 99% quantile. From the table above, you choose the pinball loss as scoring function (fair enough, not much choice given), for model training (e.g. `HistGradientBoostingRegressor(loss="quantile", quantile=0.99)`) as well as model evaluation (`mean_pinball_loss(..., alpha=0.99)` - we apologize for the different argument names, `quantile` and `alpha`) be it in grid search for finding hyperparameters or in comparing to other models like `QuantileRegressor(quantile=0.99)`.
References
\[[Gneiting2014](https://scikit-learn.org/stable/modules/model_evaluation.html#id4)\]
T. Gneiting and M. Katzfuss. [Probabilistic Forecasting](https://doi.org/10.1146/annurev-statistics-062713-085831). In: Annual Review of Statistics and Its Application 1.1 (2014), pp. 125ā151.
## 3\.4.2. Scoring API overview[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-api-overview "Link to this heading")
There are 3 different APIs for evaluating the quality of a modelās predictions:
- **Estimator score method**: Estimators have a `score` method providing a default evaluation criterion for the problem they are designed to solve. Most commonly this is [accuracy](https://scikit-learn.org/stable/modules/model_evaluation.html#accuracy-score) for classifiers and the [coefficient of determination](https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score) (R 2) for regressors. Details for each estimator can be found in its documentation.
- **Scoring parameter**: Model-evaluation tools that use [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation) (such as [`model_selection.GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV "sklearn.model_selection.GridSearchCV"), [`model_selection.validation_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.validation_curve.html#sklearn.model_selection.validation_curve "sklearn.model_selection.validation_curve") and [`linear_model.LogisticRegressionCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV "sklearn.linear_model.LogisticRegressionCV")) rely on an internal *scoring* strategy. This can be specified using the `scoring` parameter of that tool and is discussed in the section [The scoring parameter: defining model evaluation rules](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter).
- **Metric functions**: The [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") module implements functions assessing prediction error for specific purposes. These metrics are detailed in sections on [Classification metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics), [Multilabel ranking metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#multilabel-ranking-metrics), [Regression metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics) and [Clustering metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#clustering-metrics).
Finally, [Dummy estimators](https://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators) are useful to get a baseline value of those metrics for random predictions.
## 3\.4.3. The `scoring` parameter: defining model evaluation rules[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#the-scoring-parameter-defining-model-evaluation-rules "Link to this heading")
Model selection and evaluation tools that internally use [cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation) (such as [`model_selection.GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV "sklearn.model_selection.GridSearchCV"), [`model_selection.validation_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.validation_curve.html#sklearn.model_selection.validation_curve "sklearn.model_selection.validation_curve") and [`linear_model.LogisticRegressionCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV "sklearn.linear_model.LogisticRegressionCV")) take a `scoring` parameter that controls what metric they apply to the estimators evaluated.
They can be specified in several ways:
- `None`: the estimatorās default evaluation criterion (i.e., the metric used in the estimatorās `score` method) is used.
- [String name](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-string-names): common metrics can be passed via a string name.
- [Callable](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-callable): more complex metrics can be passed via a custom metric callable (e.g., function).
Some tools do also accept multiple metric evaluation. See [Using multiple metric evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html#multimetric-scoring) for details.
### 3\.4.3.1. String name scorers[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#string-name-scorers "Link to this heading")
For the most common use cases, you can designate a scorer object with the `scoring` parameter via a string name; the table below shows all possible values. All scorer objects follow the convention that **higher return values are better than lower return values**. Thus metrics which measure the distance between the model and the data, like [`metrics.mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error"), are available as āneg\_mean\_squared\_errorā which return the negated value of the metric.
| Scoring string name | Function | Comment |
|---|---|---|
| **Classification** | | |
| āaccuracyā | [`metrics.accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score") | |
| ābalanced\_accuracyā | [`metrics.balanced_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html#sklearn.metrics.balanced_accuracy_score "sklearn.metrics.balanced_accuracy_score") | |
| ātop\_k\_accuracyā | [`metrics.top_k_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html#sklearn.metrics.top_k_accuracy_score "sklearn.metrics.top_k_accuracy_score") | |
| āaverage\_precisionā | [`metrics.average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") | |
| āneg\_brier\_scoreā | [`metrics.brier_score_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss "sklearn.metrics.brier_score_loss") | requires `predict_proba` support |
| āf1ā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | for binary targets |
| āf1\_microā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | micro-averaged |
| āf1\_macroā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | macro-averaged |
| āf1\_weightedā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | weighted average |
| āf1\_samplesā | [`metrics.f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score") | by multilabel sample |
| āneg\_log\_lossā | [`metrics.log_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html#sklearn.metrics.log_loss "sklearn.metrics.log_loss") | requires `predict_proba` support |
| āprecisionā etc. | [`metrics.precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score") | suffixes apply as with āf1ā |
| ārecallā etc. | [`metrics.recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score") | suffixes apply as with āf1ā |
| ājaccardā etc. | [`metrics.jaccard_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html#sklearn.metrics.jaccard_score "sklearn.metrics.jaccard_score") | suffixes apply as with āf1ā |
| āroc\_aucā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| āroc\_auc\_ovrā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| āroc\_auc\_ovoā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| āroc\_auc\_ovr\_weightedā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| āroc\_auc\_ovo\_weightedā | [`metrics.roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") | |
| ād2\_log\_loss\_scoreā | [`metrics.d2_log_loss_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_log_loss_score.html#sklearn.metrics.d2_log_loss_score "sklearn.metrics.d2_log_loss_score") | requires `predict_proba` support |
| ād2\_brier\_scoreā | [`metrics.d2_brier_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_brier_score.html#sklearn.metrics.d2_brier_score "sklearn.metrics.d2_brier_score") | requires `predict_proba` support |
| **Clustering** | | |
| āadjusted\_mutual\_info\_scoreā | [`metrics.adjusted_mutual_info_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_mutual_info_score.html#sklearn.metrics.adjusted_mutual_info_score "sklearn.metrics.adjusted_mutual_info_score") | |
| āadjusted\_rand\_scoreā | [`metrics.adjusted_rand_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html#sklearn.metrics.adjusted_rand_score "sklearn.metrics.adjusted_rand_score") | |
| ācompleteness\_scoreā | [`metrics.completeness_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.completeness_score.html#sklearn.metrics.completeness_score "sklearn.metrics.completeness_score") | |
| āfowlkes\_mallows\_scoreā | [`metrics.fowlkes_mallows_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fowlkes_mallows_score.html#sklearn.metrics.fowlkes_mallows_score "sklearn.metrics.fowlkes_mallows_score") | |
| āhomogeneity\_scoreā | [`metrics.homogeneity_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.homogeneity_score.html#sklearn.metrics.homogeneity_score "sklearn.metrics.homogeneity_score") | |
| āmutual\_info\_scoreā | [`metrics.mutual_info_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mutual_info_score.html#sklearn.metrics.mutual_info_score "sklearn.metrics.mutual_info_score") | |
| ānormalized\_mutual\_info\_scoreā | [`metrics.normalized_mutual_info_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.normalized_mutual_info_score.html#sklearn.metrics.normalized_mutual_info_score "sklearn.metrics.normalized_mutual_info_score") | |
| ārand\_scoreā | [`metrics.rand_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.rand_score.html#sklearn.metrics.rand_score "sklearn.metrics.rand_score") | |
| āv\_measure\_scoreā | [`metrics.v_measure_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.v_measure_score.html#sklearn.metrics.v_measure_score "sklearn.metrics.v_measure_score") | |
| **Regression** | | |
| āexplained\_varianceā | [`metrics.explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") | |
| āneg\_max\_errorā | [`metrics.max_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.max_error.html#sklearn.metrics.max_error "sklearn.metrics.max_error") | |
| āneg\_mean\_absolute\_errorā | [`metrics.mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error") | |
| āneg\_mean\_squared\_errorā | [`metrics.mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error") | |
| āneg\_root\_mean\_squared\_errorā | [`metrics.root_mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.root_mean_squared_error.html#sklearn.metrics.root_mean_squared_error "sklearn.metrics.root_mean_squared_error") | |
| āneg\_mean\_squared\_log\_errorā | [`metrics.mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_log_error.html#sklearn.metrics.mean_squared_log_error "sklearn.metrics.mean_squared_log_error") | |
| āneg\_root\_mean\_squared\_log\_errorā | [`metrics.root_mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.root_mean_squared_log_error.html#sklearn.metrics.root_mean_squared_log_error "sklearn.metrics.root_mean_squared_log_error") | |
| āneg\_median\_absolute\_errorā | [`metrics.median_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.median_absolute_error.html#sklearn.metrics.median_absolute_error "sklearn.metrics.median_absolute_error") | |
| ār2ā | [`metrics.r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") | |
| āneg\_mean\_poisson\_devianceā | [`metrics.mean_poisson_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_poisson_deviance.html#sklearn.metrics.mean_poisson_deviance "sklearn.metrics.mean_poisson_deviance") | |
| āneg\_mean\_gamma\_devianceā | [`metrics.mean_gamma_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_gamma_deviance.html#sklearn.metrics.mean_gamma_deviance "sklearn.metrics.mean_gamma_deviance") | |
| āneg\_mean\_absolute\_percentage\_errorā | [`metrics.mean_absolute_percentage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html#sklearn.metrics.mean_absolute_percentage_error "sklearn.metrics.mean_absolute_percentage_error") | |
| ād2\_absolute\_error\_scoreā | [`metrics.d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score") | |
Usage examples:
```
>>> from sklearn import svm, datasets
>>> from sklearn.model_selection import cross_val_score
>>> X, y = datasets.load_iris(return_X_y=True)
>>> clf = svm.SVC(random_state=0)
>>> cross_val_score(clf, X, y, cv=5, scoring='recall_macro')
array([0.96, 0.96, 0.96, 0.93, 1. ])
```
Note
If a wrong scoring name is passed, an `InvalidParameterError` is raised. You can retrieve the names of all available scorers by calling [`get_scorer_names`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.get_scorer_names.html#sklearn.metrics.get_scorer_names "sklearn.metrics.get_scorer_names").
### 3\.4.3.2. Callable scorers[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#callable-scorers "Link to this heading")
For more complex use cases and more flexibility, you can pass a callable to the `scoring` parameter. This can be done by:
- [Adapting predefined metrics via make\_scorer](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-adapt-metric)
- [Creating a custom scorer object](https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-custom) (most flexible)
#### 3\.4.3.2.1. Adapting predefined metrics via `make_scorer`[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#adapting-predefined-metrics-via-make-scorer "Link to this heading")
The following metric functions are not implemented as named scorers, sometimes because they require additional parameters, such as [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score"). They cannot be passed to the `scoring` parameters; instead their callable needs to be passed to [`make_scorer`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer "sklearn.metrics.make_scorer") together with the value of the user-settable parameters.
| Function | Parameter | Example usage |
|---|---|---|
| **Classification** | | |
| `metrics.fbeta_score` | `beta` | `make_scorer(fbeta_score, beta=2)` |
| **Regression** | | |
| `metrics.mean_tweedie_deviance` | `power` | `make_scorer(mean_tweedie_deviance, power=1.5)` |
| `metrics.mean_pinball_loss` | `alpha` | `make_scorer(mean_pinball_loss, alpha=0.95)` |
| `metrics.d2_tweedie_score` | `power` | `make_scorer(d2_tweedie_score, power=1.5)` |
| `metrics.d2_pinball_score` | `alpha` | `make_scorer(d2_pinball_score, alpha=0.95)` |
One typical use case is to wrap an existing metric function from the library with non-default values for its parameters, such as the `beta` parameter for the [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score") function:
```
>>> from sklearn.metrics import fbeta_score, make_scorer
>>> ftwo_scorer = make_scorer(fbeta_score, beta=2)
>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.svm import LinearSVC
>>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]},
... scoring=ftwo_scorer, cv=5)
```
The module [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") also exposes a set of simple functions measuring a prediction error given ground truth and prediction:
- functions ending with `_score` return a value to maximize, the higher the better.
- functions ending with `_error`, `_loss`, or `_deviance` return a value to minimize, the lower the better. When converting into a scorer object using [`make_scorer`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer "sklearn.metrics.make_scorer"), set the `greater_is_better` parameter to `False` (`True` by default; see the parameter description below).
#### 3\.4.3.2.2. Creating a custom scorer object[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#creating-a-custom-scorer-object "Link to this heading")
You can create your own custom scorer object using [`make_scorer`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer "sklearn.metrics.make_scorer").
You can build a completely custom scorer object from a simple python function using [`make_scorer`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer "sklearn.metrics.make_scorer"), which can take several parameters:
- the python function you want to use (`my_custom_loss_func` in the example below)
- whether the python function returns a score (`greater_is_better=True`, the default) or a loss (`greater_is_better=False`). If a loss, the output of the python function is negated by the scorer object, conforming to the cross validation convention that scorers return higher values for better models.
- for classification metrics only: whether the python function you provided requires continuous decision certainties. If the scoring function only accepts probability estimates (e.g. `metrics.log_loss`), then one needs to set the parameter `response_method="predict_proba"`. Some scoring functions do not necessarily require probability estimates but rather non-thresholded decision values (e.g. `metrics.roc_auc_score`). In this case, one can provide a list (e.g., `response_method=["decision_function", "predict_proba"]`), and scorer will use the first available method, in the order given in the list, to compute the scores.
- any additional parameters of the scoring function, such as `beta` or `labels`.
Here is an example of building custom scorers, and of using the `greater_is_better` parameter:
```
>>> import numpy as np
>>> def my_custom_loss_func(y_true, y_pred):
... diff = np.abs(y_true - y_pred).max()
... return float(np.log1p(diff))
...
>>> # score will negate the return value of my_custom_loss_func,
>>> # which will be np.log(2), 0.693, given the values for X
>>> # and y defined below.
>>> score = make_scorer(my_custom_loss_func, greater_is_better=False)
>>> X = [[1], [1]]
>>> y = [0, 1]
>>> from sklearn.dummy import DummyClassifier
>>> clf = DummyClassifier(strategy='most_frequent', random_state=0)
>>> clf = clf.fit(X, y)
>>> my_custom_loss_func(y, clf.predict(X))
0.69
>>> score(clf, X, y)
-0.69
```
While defining the custom scoring function alongside the calling function should work out of the box with the default joblib backend (loky), importing it from another module will be a more robust approach and work independently of the joblib backend.
For example, to use `n_jobs` greater than 1 in the example below, `custom_scoring_function` function is saved in a user-created module (`custom_scorer_module.py`) and imported:
```
>>> from custom_scorer_module import custom_scoring_function
>>> cross_val_score(model,
... X_train,
... y_train,
... scoring=make_scorer(custom_scoring_function, greater_is_better=False),
... cv=5,
... n_jobs=-1)
```
### 3\.4.3.3. Using multiple metric evaluation[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#using-multiple-metric-evaluation "Link to this heading")
Scikit-learn also permits evaluation of multiple metrics in `GridSearchCV`, `RandomizedSearchCV` and `cross_validate`.
There are three ways to specify multiple scoring metrics for the `scoring` parameter:
- As an iterable of string metrics:
```
>>> scoring = ['accuracy', 'precision']
```
- As a `dict` mapping the scorer name to the scoring function:
```
>>> from sklearn.metrics import accuracy_score
>>> from sklearn.metrics import make_scorer
>>> scoring = {'accuracy': make_scorer(accuracy_score),
... 'prec': 'precision'}
```
Note that the dict values can either be scorer functions or one of the predefined metric strings.
- As a callable that returns a dictionary of scores:
```
>>> from sklearn.model_selection import cross_validate
>>> from sklearn.metrics import confusion_matrix
>>> # A sample toy binary classification dataset
>>> X, y = datasets.make_classification(n_classes=2, random_state=0)
>>> svm = LinearSVC(random_state=0)
>>> def confusion_matrix_scorer(clf, X, y):
... y_pred = clf.predict(X)
... cm = confusion_matrix(y, y_pred)
... return {'tn': cm[0, 0], 'fp': cm[0, 1],
... 'fn': cm[1, 0], 'tp': cm[1, 1]}
>>> cv_results = cross_validate(svm, X, y, cv=5,
... scoring=confusion_matrix_scorer)
>>> # Getting the test set true positive scores
>>> print(cv_results['test_tp'])
[10 9 8 7 8]
>>> # Getting the test set false negative scores
>>> print(cv_results['test_fn'])
[0 1 2 3 2]
```
## 3\.4.4. Classification metrics[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics "Link to this heading")
The [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") module implements several loss, score, and utility functions to measure classification performance. Some metrics might require probability estimates of the positive class, confidence values, or binary decisions values. Most implementations allow each sample to provide a weighted contribution to the overall score, through the `sample_weight` parameter.
Some of these are restricted to the binary classification case:
Others also work in the multiclass case:
| | |
|---|---|
| [`balanced_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html#sklearn.metrics.balanced_accuracy_score "sklearn.metrics.balanced_accuracy_score")(y\_true, y\_pred, \*\[, ...\]) | Compute the balanced accuracy. |
| [`cohen_kappa_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html#sklearn.metrics.cohen_kappa_score "sklearn.metrics.cohen_kappa_score")(y1, y2, \*\[, labels, ...\]) | Compute Cohen's kappa: a statistic that measures inter-annotator agreement. |
| [`confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix "sklearn.metrics.confusion_matrix")(y\_true, y\_pred, \*\[, ...\]) | Compute confusion matrix to evaluate the accuracy of a classification. |
| [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss")(y\_true, pred\_decision, \*\[, ...\]) | Average hinge loss (non-regularized). |
| [`matthews_corrcoef`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html#sklearn.metrics.matthews_corrcoef "sklearn.metrics.matthews_corrcoef")(y\_true, y\_pred, \*\[, ...\]) | Compute the Matthews correlation coefficient (MCC). |
| [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score")(y\_true, y\_score, \*\[, average, ...\]) | Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. |
| [`top_k_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html#sklearn.metrics.top_k_accuracy_score "sklearn.metrics.top_k_accuracy_score")(y\_true, y\_score, \*\[, ...\]) | Top-k Accuracy classification score. |
Some also work in the multilabel case:
| | |
|---|---|
| [`accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score")(y\_true, y\_pred, \*\[, ...\]) | Accuracy classification score. |
| [`classification_report`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html#sklearn.metrics.classification_report "sklearn.metrics.classification_report")(y\_true, y\_pred, \*\[, ...\]) | Build a text report showing the main classification metrics. |
| [`f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the F1 score, also known as balanced F-score or F-measure. |
| [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score")(y\_true, y\_pred, \*, beta\[, ...\]) | Compute the F-beta score. |
| [`hamming_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hamming_loss.html#sklearn.metrics.hamming_loss "sklearn.metrics.hamming_loss")(y\_true, y\_pred, \*\[, sample\_weight\]) | Compute the average Hamming loss. |
| [`jaccard_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html#sklearn.metrics.jaccard_score "sklearn.metrics.jaccard_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Jaccard similarity coefficient score. |
| [`log_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html#sklearn.metrics.log_loss "sklearn.metrics.log_loss")(y\_true, y\_pred, \*\[, normalize, ...\]) | Log loss, aka logistic loss or cross-entropy loss. |
| [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix")(y\_true, y\_pred, \*) | Compute a confusion matrix for each class or sample. |
| [`precision_recall_fscore_support`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html#sklearn.metrics.precision_recall_fscore_support "sklearn.metrics.precision_recall_fscore_support")(y\_true, ...) | Compute precision, recall, F-measure and support for each class. |
| [`precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the precision. |
| [`recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the recall. |
| [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score")(y\_true, y\_score, \*\[, average, ...\]) | Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. |
| [`zero_one_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.zero_one_loss.html#sklearn.metrics.zero_one_loss "sklearn.metrics.zero_one_loss")(y\_true, y\_pred, \*\[, ...\]) | Zero-one classification loss. |
| [`d2_log_loss_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_log_loss_score.html#sklearn.metrics.d2_log_loss_score "sklearn.metrics.d2_log_loss_score")(y\_true, y\_pred, \*\[, ...\]) | D 2 score function, fraction of log loss explained. |
And some work with binary and multilabel (but not multiclass) problems:
In the following sub-sections, we will describe each of those functions, preceded by some notes on common API and metric definition.
### 3\.4.4.1. From binary to multiclass and multilabel[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#from-binary-to-multiclass-and-multilabel "Link to this heading")
Some metrics are essentially defined for binary classification tasks (e.g. [`f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score"), [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score")). In these cases, by default only the positive label is evaluated, assuming by default that the positive class is labelled `1` (though this may be configurable through the `pos_label` parameter).
In extending a binary metric to multiclass or multilabel problems, the data is treated as a collection of binary problems, one for each class. There are then a number of ways to average binary metric calculations across the set of classes, each of which may be useful in some scenario. Where available, you should select among these using the `average` parameter.
- `"macro"` simply calculates the mean of the binary metrics, giving equal weight to each class. In problems where infrequent classes are nonetheless important, macro-averaging may be a means of highlighting their performance. On the other hand, the assumption that all classes are equally important is often untrue, such that macro-averaging will over-emphasize the typically low performance on an infrequent class.
- `"weighted"` accounts for class imbalance by computing the average of binary metrics in which each classās score is weighted by its presence in the true data sample.
- `"micro"` gives each sample-class pair an equal contribution to the overall metric (except as a result of sample-weight). Rather than summing the metric per class, this sums the dividends and divisors that make up the per-class metrics to calculate an overall quotient. Micro-averaging may be preferred in multilabel settings, including multiclass classification where a majority class is to be ignored.
- `"samples"` applies only to multilabel problems. It does not calculate a per-class measure, instead calculating the metric over the true and predicted classes for each sample in the evaluation data, and returning their (`sample_weight`\-weighted) average.
- Selecting `average=None` will return an array with the score for each class.
While multiclass data is provided to the metric, like binary targets, as an array of class labels, multilabel data is specified as an indicator matrix, in which cell `[i, j]` has value 1 if sample `i` has label `j` and value 0 otherwise.
### 3\.4.4.2. Accuracy score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#accuracy-score "Link to this heading")
The [`accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score") function computes the [accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision), either the fraction (default) or the count (normalize=False) of correct predictions.
In multilabel classification, the function returns the subset accuracy. If the entire set of predicted labels for a sample strictly match with the true set of labels, then the subset accuracy is 1.0; otherwise it is 0.0.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value, then the fraction of correct predictions over n samples is defined as
accuracy ( y , y ^ ) \= 1 n samples ā i \= 0 n samples ā 1 1 ( y ^ i \= y i )
where 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function).
```
>>> import numpy as np
>>> from sklearn.metrics import accuracy_score
>>> y_pred = [0, 2, 1, 3]
>>> y_true = [0, 1, 2, 3]
>>> accuracy_score(y_true, y_pred)
0.5
>>> accuracy_score(y_true, y_pred, normalize=False)
2.0
```
In the multilabel case with binary label indicators:
```
>>> accuracy_score(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5
```
Examples
- See [Test with permutations the significance of a classification score](https://scikit-learn.org/stable/auto_examples/model_selection/plot_permutation_tests_for_classification.html#sphx-glr-auto-examples-model-selection-plot-permutation-tests-for-classification-py) for an example of accuracy score usage using permutations of the dataset.
### 3\.4.4.3. Top-k accuracy score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#top-k-accuracy-score "Link to this heading")
The [`top_k_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.top_k_accuracy_score.html#sklearn.metrics.top_k_accuracy_score "sklearn.metrics.top_k_accuracy_score") function is a generalization of [`accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score"). The difference is that a prediction is considered correct as long as the true label is associated with one of the `k` highest predicted scores. [`accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score "sklearn.metrics.accuracy_score") is the special case of `k = 1`.
The function covers the binary and multiclass classification cases but not the multilabel case.
If f ^ i , j is the predicted class for the i\-th sample corresponding to the j\-th largest predicted score and y i is the corresponding true value, then the fraction of correct predictions over n samples is defined as
top-k accuracy ( y , f ^ ) \= 1 n samples ā i \= 0 n samples ā 1 ā j \= 1 k 1 ( f ^ i , j \= y i )
where k is the number of guesses allowed and 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function).
```
>>> import numpy as np
>>> from sklearn.metrics import top_k_accuracy_score
>>> y_true = np.array([0, 1, 2, 2])
>>> y_score = np.array([[0.5, 0.2, 0.2],
... [0.3, 0.4, 0.2],
... [0.2, 0.4, 0.3],
... [0.7, 0.2, 0.1]])
>>> top_k_accuracy_score(y_true, y_score, k=2)
0.75
>>> # Not normalizing gives the number of "correctly" classified samples
>>> top_k_accuracy_score(y_true, y_score, k=2, normalize=False)
3.0
```
### 3\.4.4.4. Balanced accuracy score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#balanced-accuracy-score "Link to this heading")
The [`balanced_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html#sklearn.metrics.balanced_accuracy_score "sklearn.metrics.balanced_accuracy_score") function computes the [balanced accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision), which avoids inflated performance estimates on imbalanced datasets. It is the macro-average of recall scores per class or, equivalently, raw accuracy where each sample is weighted according to the inverse prevalence of its true class. Thus for balanced datasets, the score is equal to accuracy.
In the binary case, balanced accuracy is equal to the arithmetic mean of [sensitivity](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) (true positive rate) and [specificity](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) (true negative rate), or the area under the ROC curve with binary predictions rather than scores:
balanced-accuracy \= 1 2 ( T P T P \+ F N \+ T N T N \+ F P )
If the classifier performs equally well on either class, this term reduces to the conventional accuracy (i.e., the number of correct predictions divided by the total number of predictions).
In contrast, if the conventional accuracy is above chance only because the classifier takes advantage of an imbalanced test set, then the balanced accuracy, as appropriate, will drop to 1 n \_ c l a s s e s.
The score ranges from 0 to 1, or when `adjusted=True` is used, it is rescaled to the range 1 1 ā n \_ c l a s s e s to 1, inclusive, with performance at random scoring 0.
If y i is the true value of the i\-th sample, and w i is the corresponding sample weight, then we adjust the sample weight to:
w ^ i \= w i ā j 1 ( y j \= y i ) w j
where 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function). Given predicted y ^ i for sample i, balanced accuracy is defined as:
balanced-accuracy ( y , y ^ , w ) \= 1 ā w ^ i ā i 1 ( y ^ i \= y i ) w ^ i
With `adjusted=True`, balanced accuracy reports the relative increase from balanced-accuracy ( y , 0 , w ) \= 1 n \_ c l a s s e s. In the binary case, this is also known as [Youdenās J statistic](https://en.wikipedia.org/wiki/Youden%27s_J_statistic), or *informedness*.
Note
The multiclass definition here seems the most reasonable extension of the metric used in binary classification, though there is no certain consensus in the literature:
- Our definition: [\[Mosley2013\]](https://scikit-learn.org/stable/modules/model_evaluation.html#mosley2013), [\[Kelleher2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#kelleher2015) and [\[Guyon2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#guyon2015), where [\[Guyon2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#guyon2015) adopt the adjusted version to ensure that random predictions have a score of 0 and perfect predictions have a score of 1.
- Class balanced accuracy as described in [\[Mosley2013\]](https://scikit-learn.org/stable/modules/model_evaluation.html#mosley2013): the minimum between the precision and the recall for each class is computed. Those values are then averaged over the total number of classes to get the balanced accuracy.
- Balanced Accuracy as described in [\[Urbanowicz2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#urbanowicz2015): the average of sensitivity and specificity is computed for each class and then averaged over total number of classes.
References
\[Guyon2015\] ([1](https://scikit-learn.org/stable/modules/model_evaluation.html#id15),[2](https://scikit-learn.org/stable/modules/model_evaluation.html#id16))
I. Guyon, K. Bennett, G. Cawley, H.J. Escalante, S. Escalera, T.K. Ho, N. MaciĆ , B. Ray, M. Saeed, A.R. Statnikov, E. Viegas, [Design of the 2015 ChaLearn AutoML Challenge](https://ieeexplore.ieee.org/document/7280767), IJCNN 2015.
### 3\.4.4.5. Cohenās kappa[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#cohen-s-kappa "Link to this heading")
The function [`cohen_kappa_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html#sklearn.metrics.cohen_kappa_score "sklearn.metrics.cohen_kappa_score") computes [Cohenās kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa) statistic. This measure is intended to compare labelings by different human annotators, not a classifier versus a ground truth.
The kappa score is a number between -1 and 1. Scores above .8 are generally considered good agreement; zero or lower means no agreement (practically random labels).
Kappa scores can be computed for binary or multiclass problems, but not for multilabel problems (except by manually computing a per-label score) and not for more than two annotators.
```
>>> from sklearn.metrics import cohen_kappa_score
>>> labeling1 = [2, 0, 2, 2, 0, 1]
>>> labeling2 = [0, 0, 2, 2, 0, 2]
>>> cohen_kappa_score(labeling1, labeling2)
0.4285714285714286
```
### 3\.4.4.6. Confusion matrix[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix "Link to this heading")
The [`confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix "sklearn.metrics.confusion_matrix") function evaluates classification accuracy by computing the [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) with each row corresponding to the true class (Wikipedia and other references may use different convention for axes).
By definition, entry i , j in a confusion matrix is the number of observations actually in group i, but predicted to be in group j. Here is an example:
```
>>> from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
[0, 0, 1],
[1, 0, 2]])
```
[`ConfusionMatrixDisplay`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay "sklearn.metrics.ConfusionMatrixDisplay") can be used to visually represent a confusion matrix as shown in the [Evaluate the performance of a classifier with Confusion Matrix](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py) example, which creates the following figure:
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html)
The parameter `normalize` allows to report ratios instead of counts. The confusion matrix can be normalized in 3 different ways: `'pred'`, `'true'`, and `'all'` which will divide the counts by the sum of each columns, rows, or the entire matrix, respectively.
```
>>> y_true = [0, 0, 0, 1, 1, 1, 1, 1]
>>> y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
>>> confusion_matrix(y_true, y_pred, normalize='all')
array([[0.25 , 0.125],
[0.25 , 0.375]])
```
For binary problems, we can get counts of true negatives, false positives, false negatives and true positives as follows:
```
>>> y_true = [0, 0, 0, 1, 1, 1, 1, 1]
>>> y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
>>> tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel().tolist()
>>> tn, fp, fn, tp
(2, 1, 2, 3)
```
With [`confusion_matrix_at_thresholds`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix_at_thresholds.html#sklearn.metrics.confusion_matrix_at_thresholds "sklearn.metrics.confusion_matrix_at_thresholds") we can get true negatives, false positives, false negatives and true positives for different thresholds:
```
>>> from sklearn.metrics import confusion_matrix_at_thresholds
>>> y_true = np.array([0., 0., 1., 1.])
>>> y_score = np.array([0.1, 0.4, 0.35, 0.8])
>>> tns, fps, fns, tps, thresholds = confusion_matrix_at_thresholds(y_true, y_score)
>>> tns
array([2., 1., 1., 0.])
>>> fps
array([0., 1., 1., 2.])
>>> fns
array([1., 1., 0., 0.])
>>> tps
array([1., 1., 2., 2.])
>>> thresholds
array([0.8, 0.4, 0.35, 0.1])
```
Note that the thresholds consist of distinct `y_score` values, in decreasing order.
Examples
- See [Evaluate the performance of a classifier with Confusion Matrix](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py) for an example of using a confusion matrix to evaluate classifier output quality.
- See [Recognizing hand-written digits](https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html#sphx-glr-auto-examples-classification-plot-digits-classification-py) for an example of using a confusion matrix to classify hand-written digits.
- See [Classification of text documents using sparse features](https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#sphx-glr-auto-examples-text-plot-document-classification-20newsgroups-py) for an example of using a confusion matrix to classify text documents.
### 3\.4.4.7. Classification report[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-report "Link to this heading")
The [`classification_report`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html#sklearn.metrics.classification_report "sklearn.metrics.classification_report") function builds a text report showing the main classification metrics. Here is a small example with custom `target_names` and inferred labels:
```
>>> from sklearn.metrics import classification_report
>>> y_true = [0, 1, 2, 2, 0]
>>> y_pred = [0, 0, 2, 1, 0]
>>> target_names = ['class 0', 'class 1', 'class 2']
>>> print(classification_report(y_true, y_pred, target_names=target_names))
precision recall f1-score support
class 0 0.67 1.00 0.80 2
class 1 0.00 0.00 0.00 1
class 2 1.00 0.50 0.67 2
accuracy 0.60 5
macro avg 0.56 0.50 0.49 5
weighted avg 0.67 0.60 0.59 5
```
Examples
- See [Recognizing hand-written digits](https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html#sphx-glr-auto-examples-classification-plot-digits-classification-py) for an example of classification report usage for hand-written digits.
- See [Custom refit strategy of a grid search with cross-validation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html#sphx-glr-auto-examples-model-selection-plot-grid-search-digits-py) for an example of classification report usage for grid search with nested cross-validation.
### 3\.4.4.8. Hamming loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#hamming-loss "Link to this heading")
The [`hamming_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hamming_loss.html#sklearn.metrics.hamming_loss "sklearn.metrics.hamming_loss") computes the average Hamming loss or [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) between two sets of samples.
If y ^ i , j is the predicted value for the j\-th label of a given sample i, y i , j is the corresponding true value, n samples is the number of samples and n labels is the number of labels, then the Hamming loss L H a m m i n g is defined as:
L H a m m i n g ( y , y ^ ) \= 1 n samples ā n labels ā i \= 0 n samples ā 1 ā j \= 0 n labels ā 1 1 ( y ^ i , j ā y i , j )
where 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function).
The equation above does not hold true in the case of multiclass classification. Please refer to the note below for more information.
```
>>> from sklearn.metrics import hamming_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> hamming_loss(y_true, y_pred)
0.25
```
In the multilabel case with binary label indicators:
```
>>> hamming_loss(np.array([[0, 1], [1, 1]]), np.zeros((2, 2)))
0.75
```
Note
In multiclass classification, the Hamming loss corresponds to the Hamming distance between `y_true` and `y_pred` which is similar to the [Zero one loss](https://scikit-learn.org/stable/modules/model_evaluation.html#zero-one-loss) function. However, while zero-one loss penalizes prediction sets that do not strictly match true sets, the Hamming loss penalizes individual labels. Thus the Hamming loss, upper bounded by the zero-one loss, is always between zero and one, inclusive; and predicting a proper subset or superset of the true labels will give a Hamming loss between zero and one, exclusive.
### 3\.4.4.9. Precision, recall and F-measures[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-and-f-measures "Link to this heading")
Intuitively, [precision](https://en.wikipedia.org/wiki/Precision_and_recall#Precision) is the ability of the classifier not to label as positive a sample that is negative, and [recall](https://en.wikipedia.org/wiki/Precision_and_recall#Recall) is the ability of the classifier to find all the positive samples.
The [F-measure](https://en.wikipedia.org/wiki/F1_score) (F Ī² and F 1 measures) can be interpreted as a weighted harmonic mean of the precision and recall. A F Ī² measure reaches its best value at 1 and its worst score at 0. With Ī² \= 1, F Ī² and F 1 are equivalent, and the recall and the precision are equally important.
The [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve "sklearn.metrics.precision_recall_curve") computes a precision-recall curve from the ground truth label and a score given by the classifier by varying a decision threshold.
The [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") function computes the [average precision](https://en.wikipedia.org/w/index.php?title=Information_retrieval&oldid=793358396#Average_precision) (AP) from prediction scores. The value is between 0 and 1 and higher is better. AP is defined as
AP \= ā n ( R n ā R n ā 1 ) P n
where P n and R n are the precision and recall at the nth threshold. With random predictions, the AP is the fraction of positive samples.
References [\[Manning2008\]](https://scikit-learn.org/stable/modules/model_evaluation.html#manning2008) and [\[Everingham2010\]](https://scikit-learn.org/stable/modules/model_evaluation.html#everingham2010) present alternative variants of AP that interpolate the precision-recall curve. Currently, [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") does not implement any interpolated variant. References [\[Davis2006\]](https://scikit-learn.org/stable/modules/model_evaluation.html#davis2006) and [\[Flach2015\]](https://scikit-learn.org/stable/modules/model_evaluation.html#flach2015) describe why a linear interpolation of points on the precision-recall curve provides an overly-optimistic measure of classifier performance. This linear interpolation is used when computing area under the curve with the trapezoidal rule in [`auc`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.auc.html#sklearn.metrics.auc "sklearn.metrics.auc"). [\[Chen2024\]](https://scikit-learn.org/stable/modules/model_evaluation.html#chen2024) benchmarks different interpolation strategies to demonstrate the effects.
Several functions allow you to analyze the precision, recall and F-measures score:
| | |
|---|---|
| [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score")(y\_true, y\_score, \*) | Compute average precision (AP) from prediction scores. |
| [`f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the F1 score, also known as balanced F-score or F-measure. |
| [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score")(y\_true, y\_pred, \*, beta\[, ...\]) | Compute the F-beta score. |
| [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve "sklearn.metrics.precision_recall_curve")(y\_true, y\_score, \*\[, ...\]) | Compute precision-recall pairs for different probability thresholds. |
| [`precision_recall_fscore_support`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html#sklearn.metrics.precision_recall_fscore_support "sklearn.metrics.precision_recall_fscore_support")(y\_true, ...) | Compute precision, recall, F-measure and support for each class. |
| [`precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the precision. |
| [`recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score")(y\_true, y\_pred, \*\[, labels, ...\]) | Compute the recall. |
Note that the [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve "sklearn.metrics.precision_recall_curve") function is restricted to the binary case. The [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") function supports multiclass and multilabel formats by computing each class score in a One-vs-the-rest (OvR) fashion and averaging them or not depending of its `average` argument value.
The [`PrecisionRecallDisplay.from_estimator`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.PrecisionRecallDisplay.html#sklearn.metrics.PrecisionRecallDisplay.from_estimator "sklearn.metrics.PrecisionRecallDisplay.from_estimator") and [`PrecisionRecallDisplay.from_predictions`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.PrecisionRecallDisplay.html#sklearn.metrics.PrecisionRecallDisplay.from_predictions "sklearn.metrics.PrecisionRecallDisplay.from_predictions") functions will plot the precision-recall curve as follows.
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html#plot-the-precision-recall-curve)
Examples
- See [Custom refit strategy of a grid search with cross-validation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html#sphx-glr-auto-examples-model-selection-plot-grid-search-digits-py) for an example of [`precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score") and [`recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score") usage to estimate parameters using grid search with nested cross-validation.
- See [Precision-Recall](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html#sphx-glr-auto-examples-model-selection-plot-precision-recall-py) for an example of [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve "sklearn.metrics.precision_recall_curve") usage to evaluate classifier output quality.
References
\[[Chen2024](https://scikit-learn.org/stable/modules/model_evaluation.html#id29)\]
W. Chen, C. Miao, Z. Zhang, C.S. Fung, R. Wang, Y. Chen, Y. Qian, L. Cheng, K.Y. Yip, S.K Tsui, Q. Cao, [Commonly used software tools produce conflicting and overly-optimistic AUPRC values](https://doi.org/10.1186/s13059-024-03266-y), Genome Biology 2024.
#### 3\.4.4.9.1. Binary classification[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#binary-classification "Link to this heading")
In a binary classification task, the terms āāpositiveāā and āānegativeāā refer to the classifierās prediction, and the terms āātrueāā and āāfalseāā refer to whether that prediction corresponds to the external judgment (sometimes known as the āāobservationāā). Given these definitions, we can formulate the following table:
In this context, we can define the notions of precision and recall:
precision \= tp tp \+ fp ,
recall \= tp tp \+ fn ,
(Sometimes recall is also called āāsensitivityāā)
F-measure is the weighted harmonic mean of precision and recall, with precisionās contribution to the mean weighted by some parameter Ī²:
F Ī² \= ( 1 \+ Ī² 2 ) precision Ć recall Ī² 2 precision \+ recall
To avoid division by zero when precision and recall are zero, Scikit-Learn calculates F-measure with this otherwise-equivalent formula:
F Ī² \= ( 1 \+ Ī² 2 ) tp ( 1 \+ Ī² 2 ) tp \+ fp \+ Ī² 2 fn
Note that this formula is still undefined when there are no true positives, false positives, or false negatives. By default, F-1 for a set of exclusively true negatives is calculated as 0, however this behavior can be changed using the `zero_division` parameter. Here are some small examples in binary classification:
```
>>> from sklearn import metrics
>>> y_pred = [0, 1, 0, 0]
>>> y_true = [0, 1, 0, 1]
>>> metrics.precision_score(y_true, y_pred)
1.0
>>> metrics.recall_score(y_true, y_pred)
0.5
>>> metrics.f1_score(y_true, y_pred)
0.66
>>> metrics.fbeta_score(y_true, y_pred, beta=0.5)
0.83
>>> metrics.fbeta_score(y_true, y_pred, beta=1)
0.66
>>> metrics.fbeta_score(y_true, y_pred, beta=2)
0.55
>>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5)
(array([0.66, 1. ]), array([1. , 0.5]), array([0.71, 0.83]), array([2, 2]))
>>> import numpy as np
>>> from sklearn.metrics import precision_recall_curve
>>> from sklearn.metrics import average_precision_score
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> precision, recall, threshold = precision_recall_curve(y_true, y_scores)
>>> precision
array([0.5 , 0.66, 0.5 , 1. , 1. ])
>>> recall
array([1. , 1. , 0.5, 0.5, 0. ])
>>> threshold
array([0.1 , 0.35, 0.4 , 0.8 ])
>>> average_precision_score(y_true, y_scores)
0.83
```
#### 3\.4.4.9.2. Multiclass and multilabel classification[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multiclass-and-multilabel-classification "Link to this heading")
In a multiclass and multilabel classification task, the notions of precision, recall, and F-measures can be applied to each label independently. There are a few ways to combine results across labels, specified by the `average` argument to the [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score"), [`f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score "sklearn.metrics.f1_score"), [`fbeta_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score "sklearn.metrics.fbeta_score"), [`precision_recall_fscore_support`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html#sklearn.metrics.precision_recall_fscore_support "sklearn.metrics.precision_recall_fscore_support"), [`precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score "sklearn.metrics.precision_score") and [`recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score "sklearn.metrics.recall_score") functions, as described [above](https://scikit-learn.org/stable/modules/model_evaluation.html#average).
Note the following behaviors when averaging:
- If all labels are included, āmicroā-averaging in a multiclass setting will produce precision, recall and F that are all identical to accuracy.
- āweightedā averaging may produce a F-score that is not between precision and recall.
- āmacroā averaging for F-measures is calculated as the arithmetic mean over per-label/class F-measures, not the harmonic mean over the arithmetic precision and recall means. Both calculations can be seen in the literature but are not equivalent, see [\[OB2019\]](https://scikit-learn.org/stable/modules/model_evaluation.html#ob2019) for details.
To make this more explicit, consider the following notation:
- y the set of *true* ( s a m p l e , l a b e l ) pairs
- y ^ the set of *predicted* ( s a m p l e , l a b e l ) pairs
- L the set of labels
- S the set of samples
- y s the subset of y with sample s, i.e. y s := { ( s ā² , l ) ā y \| s ā² \= s }
- y l the subset of y with label l
- similarly, y ^ s and y ^ l are subsets of y ^
- P ( A , B ) := \| A ā© B \| \| B \| for some sets A and B
- R ( A , B ) := \| A ā© B \| \| A \| (Conventions vary on handling A \= ā
; this implementation uses R ( A , B ) := 0, and similar for P.)
- F Ī² ( A , B ) := ( 1 \+ Ī² 2 ) P ( A , B ) Ć R ( A , B ) Ī² 2 P ( A , B ) \+ R ( A , B )
Then the metrics are defined as:
| `average` | Precision | Recall | F\_beta |
|---|---|---|---|
| `"micro"` | P ( y , y ^ ) | | |
```
>>> from sklearn import metrics
>>> y_true = [0, 1, 2, 0, 1, 2]
>>> y_pred = [0, 2, 1, 0, 0, 1]
>>> metrics.precision_score(y_true, y_pred, average='macro')
0.22
>>> metrics.recall_score(y_true, y_pred, average='micro')
0.33
>>> metrics.f1_score(y_true, y_pred, average='weighted')
0.267
>>> metrics.fbeta_score(y_true, y_pred, average='macro', beta=0.5)
0.238
>>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5, average=None)
(array([0.667, 0., 0.]), array([1., 0., 0.]), array([0.714, 0., 0.]), array([2, 2, 2]))
```
For multiclass classification with a ānegative classā, it is possible to exclude some labels:
```
>>> metrics.recall_score(y_true, y_pred, labels=[1, 2], average='micro')
... # excluding 0, no labels were correctly recalled
0.0
```
Similarly, labels not present in the data sample may be accounted for in macro-averaging.
```
>>> metrics.precision_score(y_true, y_pred, labels=[0, 1, 2, 3], average='macro')
0.166
```
References
### 3\.4.4.10. Jaccard similarity coefficient score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#jaccard-similarity-coefficient-score "Link to this heading")
The [`jaccard_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html#sklearn.metrics.jaccard_score "sklearn.metrics.jaccard_score") function computes the average of [Jaccard similarity coefficients](https://en.wikipedia.org/wiki/Jaccard_index), also called the Jaccard index, between pairs of label sets.
The Jaccard similarity coefficient with a ground truth label set y and predicted label set y ^, is defined as
J ( y , y ^ ) \= \| y ā© y ^ \| \| y āŖ y ^ \| .
The [`jaccard_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html#sklearn.metrics.jaccard_score "sklearn.metrics.jaccard_score") (like [`precision_recall_fscore_support`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html#sklearn.metrics.precision_recall_fscore_support "sklearn.metrics.precision_recall_fscore_support")) applies natively to binary targets. By computing it set-wise it can be extended to apply to multilabel and multiclass through the use of `average` (see [above](https://scikit-learn.org/stable/modules/model_evaluation.html#average)).
In the binary case:
```
>>> import numpy as np
>>> from sklearn.metrics import jaccard_score
>>> y_true = np.array([[0, 1, 1],
... [1, 1, 0]])
>>> y_pred = np.array([[1, 1, 1],
... [1, 0, 0]])
>>> jaccard_score(y_true[0], y_pred[0])
0.6666
```
In the 2D comparison case (e.g. image similarity):
```
>>> jaccard_score(y_true, y_pred, average="micro")
0.6
```
In the multilabel case with binary label indicators:
```
>>> jaccard_score(y_true, y_pred, average='samples')
0.5833
>>> jaccard_score(y_true, y_pred, average='macro')
0.6666
>>> jaccard_score(y_true, y_pred, average=None)
array([0.5, 0.5, 1. ])
```
Multiclass problems are binarized and treated like the corresponding multilabel problem:
```
>>> y_pred = [0, 2, 1, 2]
>>> y_true = [0, 1, 2, 2]
>>> jaccard_score(y_true, y_pred, average=None)
array([1. , 0. , 0.33])
>>> jaccard_score(y_true, y_pred, average='macro')
0.44
>>> jaccard_score(y_true, y_pred, average='micro')
0.33
```
### 3\.4.4.11. Hinge loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#hinge-loss "Link to this heading")
The [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss") function computes the average distance between the model and the data using [hinge loss](https://en.wikipedia.org/wiki/Hinge_loss), a one-sided metric that considers only prediction errors. (Hinge loss is used in maximal margin classifiers such as support vector machines.)
If the true label y i of a binary classification task is encoded as y i \= { ā 1 , \+ 1 } for every sample i; and w i is the corresponding predicted decision (an array of shape (`n_samples`,) as output by the `decision_function` method), then the hinge loss is defined as:
L Hinge ( y , w ) \= 1 n samples ā i \= 0 n samples ā 1 max { 1 ā w i y i , 0 }
If there are more than two labels, [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss") uses a multiclass variant due to Crammer & Singer. [Here](https://jmlr.csail.mit.edu/papers/volume2/crammer01a/crammer01a.pdf) is the paper describing it.
In this case the predicted decision is an array of shape (`n_samples`, `n_labels`). If w i , y i is the predicted decision for the true label y i of the i\-th sample; and w ^ i , y i \= max { w i , y j \| y j ā y i } is the maximum of the predicted decisions for all the other labels, then the multi-class hinge loss is defined by:
L Hinge ( y , w ) \= 1 n samples ā i \= 0 n samples ā 1 max { 1 \+ w ^ i , y i ā w i , y i , 0 }
Here is a small example demonstrating the use of the [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss") function with an svm classifier in a binary class problem:
```
>>> from sklearn import svm
>>> from sklearn.metrics import hinge_loss
>>> X = [[0], [1]]
>>> y = [-1, 1]
>>> est = svm.LinearSVC(random_state=0)
>>> est.fit(X, y)
LinearSVC(random_state=0)
>>> pred_decision = est.decision_function([[-2], [3], [0.5]])
>>> pred_decision
array([-2.18, 2.36, 0.09])
>>> hinge_loss([-1, 1, 1], pred_decision)
0.3
```
Here is an example demonstrating the use of the [`hinge_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html#sklearn.metrics.hinge_loss "sklearn.metrics.hinge_loss") function with an svm classifier in a multiclass problem:
```
>>> X = np.array([[0], [1], [2], [3]])
>>> Y = np.array([0, 1, 2, 3])
>>> labels = np.array([0, 1, 2, 3])
>>> est = svm.LinearSVC()
>>> est.fit(X, Y)
LinearSVC()
>>> pred_decision = est.decision_function([[-1], [2], [3]])
>>> y_true = [0, 2, 3]
>>> hinge_loss(y_true, pred_decision, labels=labels)
0.56
```
### 3\.4.4.12. Log loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#log-loss "Link to this heading")
Log loss, also called logistic regression loss or cross-entropy loss, is defined on probability estimates. It is commonly used in (multinomial) logistic regression and neural networks, as well as in some variants of expectation-maximization, and can be used to evaluate the probability outputs (`predict_proba`) of a classifier instead of its discrete predictions.
For binary classification with a true label y ā { 0 , 1 } and a probability estimate p ^ ā Pr ( y \= 1 ), the log loss per sample is the negative log-likelihood of the classifier given the true label:
L log ( y , p ^ ) \= ā log ā” Pr ( y \| p ^ ) \= ā ( y log ā” ( p ^ ) \+ ( 1 ā y ) log ā” ( 1 ā p ^ ) )
This extends to the multiclass case as follows. Let the true labels for a set of samples be encoded as a 1-of-K binary indicator matrix Y, i.e., y i , k \= 1 if sample i has label k taken from a set of K labels. Let P ^ be a matrix of probability estimates, with elements p ^ i , k ā Pr ( y i , k \= 1 ). Then the log loss of the whole set is
L log ( Y , P ^ ) \= ā log ā” Pr ( Y \| P ^ ) \= ā 1 N ā i \= 0 N ā 1 ā k \= 0 K ā 1 y i , k log ā” p ^ i , k
To see how this generalizes the binary log loss given above, note that in the binary case, p ^ i , 0 \= 1 ā p ^ i , 1 and y i , 0 \= 1 ā y i , 1, so expanding the inner sum over y i , k ā { 0 , 1 } gives the binary log loss.
The [`log_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html#sklearn.metrics.log_loss "sklearn.metrics.log_loss") function computes log loss given a list of ground-truth labels and a probability matrix, as returned by an estimatorās `predict_proba` method.
```
>>> from sklearn.metrics import log_loss
>>> y_true = [0, 0, 1, 1]
>>> y_pred = [[.9, .1], [.8, .2], [.3, .7], [.01, .99]]
>>> log_loss(y_true, y_pred)
0.1738
```
The first `[.9, .1]` in `y_pred` denotes 90% probability that the first sample has label 0. The log loss is non-negative.
### 3\.4.4.13. Matthews correlation coefficient[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#matthews-correlation-coefficient "Link to this heading")
The [`matthews_corrcoef`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html#sklearn.metrics.matthews_corrcoef "sklearn.metrics.matthews_corrcoef") function computes the [Matthewās correlation coefficient (MCC)](https://en.wikipedia.org/wiki/Matthews_correlation_coefficient) for binary classes. Quoting Wikipedia:
> āThe Matthews correlation coefficient is used in machine learning as a measure of the quality of binary (two-class) classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient.ā
In the binary (two-class) case, t p, t n, f p and f n are respectively the number of true positives, true negatives, false positives and false negatives, the MCC is defined as
M C C \= t p Ć t n ā f p Ć f n ( t p \+ f p ) ( t p \+ f n ) ( t n \+ f p ) ( t n \+ f n ) .
In the multiclass case, the Matthews correlation coefficient can be [defined](http://rk.kvl.dk/introduction/index.html) in terms of a [`confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix "sklearn.metrics.confusion_matrix") C for K classes. To simplify the definition consider the following intermediate variables:
- t k \= ā i K C i k the number of times class k truly occurred,
- p k \= ā i K C k i the number of times class k was predicted,
- c \= ā k K C k k the total number of samples correctly predicted,
- s \= ā i K ā j K C i j the total number of samples.
Then the multiclass MCC is defined as:
M C C \= c Ć s ā ā k K p k Ć t k ( s 2 ā ā k K p k 2 ) Ć ( s 2 ā ā k K t k 2 )
When there are more than two labels, the value of the MCC will no longer range between -1 and +1. Instead the minimum value will be somewhere between -1 and 0 depending on the number and distribution of ground truth labels. The maximum value is always +1. For additional information, see [\[WikipediaMCC2021\]](https://scikit-learn.org/stable/modules/model_evaluation.html#wikipediamcc2021).
Here is a small example illustrating the usage of the [`matthews_corrcoef`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html#sklearn.metrics.matthews_corrcoef "sklearn.metrics.matthews_corrcoef") function:
```
>>> from sklearn.metrics import matthews_corrcoef
>>> y_true = [+1, +1, +1, -1]
>>> y_pred = [+1, -1, +1, +1]
>>> matthews_corrcoef(y_true, y_pred)
-0.33
```
References
### 3\.4.4.14. Multi-label confusion matrix[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multi-label-confusion-matrix "Link to this heading")
The [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix") function computes class-wise (default) or sample-wise (samplewise=True) multilabel confusion matrix to evaluate the accuracy of a classification. multilabel\_confusion\_matrix also treats multiclass data as if it were multilabel, as this is a transformation commonly applied to evaluate multiclass problems with binary classification metrics (such as precision, recall, etc.).
When calculating class-wise multilabel confusion matrix C, the count of true negatives for class i is C i , 0 , 0, false negatives is C i , 1 , 0, true positives is C i , 1 , 1 and false positives is C i , 0 , 1.
Here is an example demonstrating the use of the [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix") function with [multilabel indicator matrix](https://scikit-learn.org/stable/glossary.html#term-multilabel-indicator-matrix) input:
```
>>> import numpy as np
>>> from sklearn.metrics import multilabel_confusion_matrix
>>> y_true = np.array([[1, 0, 1],
... [0, 1, 0]])
>>> y_pred = np.array([[1, 0, 0],
... [0, 1, 1]])
>>> multilabel_confusion_matrix(y_true, y_pred)
array([[[1, 0],
[0, 1]],
[[1, 0],
[0, 1]],
[[0, 1],
[1, 0]]])
```
Or a confusion matrix can be constructed for each sampleās labels:
```
>>> multilabel_confusion_matrix(y_true, y_pred, samplewise=True)
array([[[1, 0],
[1, 1]],
[[1, 1],
[0, 1]]])
```
Here is an example demonstrating the use of the [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix") function with [multiclass](https://scikit-learn.org/stable/glossary.html#term-multiclass) input:
```
>>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
>>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
>>> multilabel_confusion_matrix(y_true, y_pred,
... labels=["ant", "bird", "cat"])
array([[[3, 1],
[0, 2]],
[[5, 0],
[1, 0]],
[[2, 1],
[1, 2]]])
```
Here are some examples demonstrating the use of the [`multilabel_confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html#sklearn.metrics.multilabel_confusion_matrix "sklearn.metrics.multilabel_confusion_matrix") function to calculate recall (or sensitivity), specificity, fall out and miss rate for each class in a problem with multilabel indicator matrix input.
Calculating [recall](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) (also called the true positive rate or the sensitivity) for each class:
```
>>> y_true = np.array([[0, 0, 1],
... [0, 1, 0],
... [1, 1, 0]])
>>> y_pred = np.array([[0, 1, 0],
... [0, 0, 1],
... [1, 1, 0]])
>>> mcm = multilabel_confusion_matrix(y_true, y_pred)
>>> tn = mcm[:, 0, 0]
>>> tp = mcm[:, 1, 1]
>>> fn = mcm[:, 1, 0]
>>> fp = mcm[:, 0, 1]
>>> tp / (tp + fn)
array([1. , 0.5, 0. ])
```
Calculating [specificity](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) (also called the true negative rate) for each class:
```
>>> tn / (tn + fp)
array([1. , 0. , 0.5])
```
Calculating [fall out](https://en.wikipedia.org/wiki/False_positive_rate) (also called the false positive rate) for each class:
```
>>> fp / (fp + tn)
array([0. , 1. , 0.5])
```
Calculating [miss rate](https://en.wikipedia.org/wiki/False_positives_and_false_negatives) (also called the false negative rate) for each class:
```
>>> fn / (fn + tp)
array([0. , 0.5, 1. ])
```
### 3\.4.4.15. Receiver operating characteristic (ROC)[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#receiver-operating-characteristic-roc "Link to this heading")
The function [`roc_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve "sklearn.metrics.roc_curve") computes the [receiver operating characteristic curve, or ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic). Quoting Wikipedia :
> āA receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied. It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate) vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold settings. TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate.ā
This function requires the true binary value and the target scores, which can either be probability estimates of the positive class, confidence values, or binary decisions. Here is a small example of how to use the [`roc_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve "sklearn.metrics.roc_curve") function:
```
>>> import numpy as np
>>> from sklearn.metrics import roc_curve
>>> y = np.array([1, 1, 2, 2])
>>> scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, tpr, thresholds = roc_curve(y, scores, pos_label=2)
>>> fpr
array([0. , 0. , 0.5, 0.5, 1. ])
>>> tpr
array([0. , 0.5, 0.5, 1. , 1. ])
>>> thresholds
array([ inf, 0.8 , 0.4 , 0.35, 0.1 ])
```
Compared to metrics such as the subset accuracy, the Hamming loss, or the F1 score, ROC doesnāt require optimizing a threshold for each label.
The [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") function, denoted by ROC-AUC or AUROC, computes the area under the ROC curve. By doing so, the curve information is summarized in one number.
The following figure shows the ROC curve and ROC-AUC score for a classifier aimed to distinguish the virginica flower from the rest of the species in the [Iris plants dataset](https://scikit-learn.org/stable/datasets/toy_dataset.html#iris-dataset):
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html)
For more information see the [Wikipedia article on AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve).
#### 3\.4.4.15.1. Binary case[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#binary-case "Link to this heading")
In the **binary case**, you can either provide the probability estimates, using the `classifier.predict_proba()` method, or the non-thresholded decision values given by the `classifier.decision_function()` method. In the case of providing the probability estimates, the probability of the class with the āgreater labelā should be provided. The āgreater labelā corresponds to `classifier.classes_[1]` and thus `classifier.predict_proba(X)[:, 1]`. Therefore, the `y_score` parameter is of size (n\_samples,).
```
>>> from sklearn.datasets import load_breast_cancer
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.metrics import roc_auc_score
>>> X, y = load_breast_cancer(return_X_y=True)
>>> clf = LogisticRegression().fit(X, y)
>>> clf.classes_
array([0, 1])
```
We can use the probability estimates corresponding to `clf.classes_[1]`.
```
>>> y_score = clf.predict_proba(X)[:, 1]
>>> roc_auc_score(y, y_score)
0.99
```
Otherwise, we can use the non-thresholded decision values
```
>>> roc_auc_score(y, clf.decision_function(X))
0.99
```
#### 3\.4.4.15.2. Multi-class case[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multi-class-case "Link to this heading")
The [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") function can also be used in **multi-class classification**. Two averaging strategies are currently supported: the one-vs-one algorithm computes the average of the pairwise ROC AUC scores, and the one-vs-rest algorithm computes the average of the ROC AUC scores for each class against all other classes. In both cases, the predicted labels are provided in an array with values from 0 to `n_classes`, and the scores correspond to the probability estimates that a sample belongs to a particular class. The OvO and OvR algorithms support weighting uniformly (`average='macro'`) and by prevalence (`average='weighted'`).
Computes the average AUC of all possible pairwise combinations of classes. [\[HT2001\]](https://scikit-learn.org/stable/modules/model_evaluation.html#ht2001) defines a multiclass AUC metric weighted uniformly:
1 c ( c ā 1 ) ā j \= 1 c ā k \> j c ( AUC ( j \| k ) \+ AUC ( k \| j ) )
where c is the number of classes and AUC ( j \| k ) is the AUC with class j as the positive class and class k as the negative class. In general, AUC ( j \| k ) ā AUC ( k \| j ) in the multiclass case. This algorithm is used by setting the keyword argument `multiclass` to `'ovo'` and `average` to `'macro'`.
The [\[HT2001\]](https://scikit-learn.org/stable/modules/model_evaluation.html#ht2001) multiclass AUC metric can be extended to be weighted by the prevalence:
1 c ( c ā 1 ) ā j \= 1 c ā k \> j c p ( j āŖ k ) ( AUC ( j \| k ) \+ AUC ( k \| j ) )
where c is the number of classes. This algorithm is used by setting the keyword argument `multiclass` to `'ovo'` and `average` to `'weighted'`. The `'weighted'` option returns a prevalence-weighted average as described in [\[FC2009\]](https://scikit-learn.org/stable/modules/model_evaluation.html#fc2009).
Computes the AUC of each class against the rest [\[PD2000\]](https://scikit-learn.org/stable/modules/model_evaluation.html#pd2000). The algorithm is functionally the same as the multilabel case. To enable this algorithm set the keyword argument `multiclass` to `'ovr'`. Additionally to `'macro'` [\[F2006\]](https://scikit-learn.org/stable/modules/model_evaluation.html#f2006) and `'weighted'` [\[F2001\]](https://scikit-learn.org/stable/modules/model_evaluation.html#f2001) averaging, OvR supports `'micro'` averaging.
In applications where a high false positive rate is not tolerable the parameter `max_fpr` of [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") can be used to summarize the ROC curve up to the given limit.
The following figure shows the micro-averaged ROC curve and its corresponding ROC-AUC score for a classifier aimed to distinguish the different species in the [Iris plants dataset](https://scikit-learn.org/stable/datasets/toy_dataset.html#iris-dataset):
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html)
#### 3\.4.4.15.3. Multi-label case[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multi-label-case "Link to this heading")
In **multi-label classification**, the [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score "sklearn.metrics.roc_auc_score") function is extended by averaging over the labels as [above](https://scikit-learn.org/stable/modules/model_evaluation.html#average). In this case, you should provide a `y_score` of shape `(n_samples, n_classes)`. Thus, when using the probability estimates, one needs to select the probability of the class with the greater label for each output.
```
>>> from sklearn.datasets import make_multilabel_classification
>>> from sklearn.multioutput import MultiOutputClassifier
>>> X, y = make_multilabel_classification(random_state=0)
>>> inner_clf = LogisticRegression(random_state=0)
>>> clf = MultiOutputClassifier(inner_clf).fit(X, y)
>>> y_score = np.transpose([y_pred[:, 1] for y_pred in clf.predict_proba(X)])
>>> roc_auc_score(y, y_score, average=None)
array([0.828, 0.851, 0.94, 0.87, 0.95])
```
And the decision values do not require such processing.
```
>>> from sklearn.linear_model import RidgeClassifierCV
>>> clf = RidgeClassifierCV().fit(X, y)
>>> y_score = clf.decision_function(X)
>>> roc_auc_score(y, y_score, average=None)
array([0.82, 0.85, 0.93, 0.87, 0.94])
```
Examples
- See [Multiclass Receiver Operating Characteristic (ROC)](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#sphx-glr-auto-examples-model-selection-plot-roc-py) for an example of using ROC to evaluate the quality of the output of a classifier.
- See [Receiver Operating Characteristic (ROC) with cross validation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossval.html#sphx-glr-auto-examples-model-selection-plot-roc-crossval-py) for an example of using ROC to evaluate classifier output quality, using cross-validation.
- See [Species distribution modeling](https://scikit-learn.org/stable/auto_examples/applications/plot_species_distribution_modeling.html#sphx-glr-auto-examples-applications-plot-species-distribution-modeling-py) for an example of using ROC to model species distribution.
References
### 3\.4.4.16. Detection error tradeoff (DET)[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#detection-error-tradeoff-det "Link to this heading")
The function [`det_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.det_curve.html#sklearn.metrics.det_curve "sklearn.metrics.det_curve") computes the detection error tradeoff curve (DET) curve [\[WikipediaDET2017\]](https://scikit-learn.org/stable/modules/model_evaluation.html#wikipediadet2017). Quoting Wikipedia:
> āA detection error tradeoff (DET) graph is a graphical plot of error rates for binary classification systems, plotting false reject rate vs. false accept rate. The x- and y-axes are scaled non-linearly by their standard normal deviates (or just by logarithmic transformation), yielding tradeoff curves that are more linear than ROC curves, and use most of the image area to highlight the differences of importance in the critical operating region.ā
DET curves are a variation of receiver operating characteristic (ROC) curves where False Negative Rate is plotted on the y-axis instead of True Positive Rate. DET curves are commonly plotted in normal deviate scale by transformation with Ļ ā 1 (with Ļ being the cumulative distribution function). The resulting performance curves explicitly visualize the tradeoff of error types for given classification algorithms. See [\[Martin1997\]](https://scikit-learn.org/stable/modules/model_evaluation.html#martin1997) for examples and further motivation.
This figure compares the ROC and DET curves of two example classifiers on the same classification task:
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_det.html)
- DET curves form a linear curve in normal deviate scale if the detection scores are normally (or close-to normally) distributed. It was shown by [\[Navratil2007\]](https://scikit-learn.org/stable/modules/model_evaluation.html#navratil2007) that the reverse is not necessarily true and even more general distributions are able to produce linear DET curves.
- The normal deviate scale transformation spreads out the points such that a comparatively larger space of plot is occupied. Therefore curves with similar classification performance might be easier to distinguish on a DET plot.
- With False Negative Rate being āinverseā to True Positive Rate the point of perfection for DET curves is the origin (in contrast to the top left corner for ROC curves).
DET curves are intuitive to read and hence allow quick visual assessment of a classifierās performance. Additionally DET curves can be consulted for threshold analysis and operating point selection. This is particularly helpful if a comparison of error types is required.
On the other hand DET curves do not provide their metric as a single number. Therefore for either automated evaluation or comparison to other classification tasks metrics like the derived area under ROC curve might be better suited.
Examples
- See [Detection error tradeoff (DET) curve](https://scikit-learn.org/stable/auto_examples/model_selection/plot_det.html#sphx-glr-auto-examples-model-selection-plot-det-py) for an example comparison between receiver operating characteristic (ROC) curves and Detection error tradeoff (DET) curves.
References
\[[Navratil2007](https://scikit-learn.org/stable/modules/model_evaluation.html#id43)\]
J. Navratil and D. Klusacek, [āOn Linear DETsā](https://ieeexplore.ieee.org/document/4218079), 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ā07, Honolulu, HI, 2007, pp. IV-229-IV-232.
### 3\.4.4.17. Zero one loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#zero-one-loss "Link to this heading")
The [`zero_one_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.zero_one_loss.html#sklearn.metrics.zero_one_loss "sklearn.metrics.zero_one_loss") function computes the sum or the average of the 0-1 classification loss (L 0 ā 1) over n samples. By default, the function normalizes over the sample. To get the sum of the L 0 ā 1, set `normalize` to `False`.
In multilabel classification, the [`zero_one_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.zero_one_loss.html#sklearn.metrics.zero_one_loss "sklearn.metrics.zero_one_loss") scores a subset as one if its labels strictly match the predictions, and as a zero if there are any errors. By default, the function returns the percentage of imperfectly predicted subsets. To get the count of such subsets instead, set `normalize` to `False`.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value, then the 0-1 loss L 0 ā 1 is defined as:
L 0 ā 1 ( y , y ^ ) \= 1 n samples ā i \= 0 n samples ā 1 1 ( y ^ i ā y i )
where 1 ( x ) is the [indicator function](https://en.wikipedia.org/wiki/Indicator_function). The zero-one loss can also be computed as zero-one loss \= 1 ā accuracy.
```
>>> from sklearn.metrics import zero_one_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> zero_one_loss(y_true, y_pred)
0.25
>>> zero_one_loss(y_true, y_pred, normalize=False)
1.0
```
In the multilabel case with binary label indicators, where the first label set \[0,1\] has an error:
```
>>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5
>>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2)), normalize=False)
1.0
```
Examples
- See [Recursive feature elimination with cross-validation](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html#sphx-glr-auto-examples-feature-selection-plot-rfe-with-cross-validation-py) for an example of zero one loss usage to perform recursive feature elimination with cross-validation.
### 3\.4.4.18. Brier score loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#brier-score-loss "Link to this heading")
The [`brier_score_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss "sklearn.metrics.brier_score_loss") function computes the [Brier score](https://en.wikipedia.org/wiki/Brier_score) for binary and multiclass probabilistic predictions and is equivalent to the mean squared error. Quoting Wikipedia:
> āThe Brier score is a strictly proper scoring rule that measures the accuracy of probabilistic predictions. \[ā¦\] \[It\] is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes or classes.ā
Let the true labels for a set of N data points be encoded as a 1-of-K binary indicator matrix Y, i.e., y i , k \= 1 if sample i has label k taken from a set of K labels. Let P ^ be a matrix of probability estimates with elements p ^ i , k ā Pr ( y i , k \= 1 ). Following the original definition by [\[Brier1950\]](https://scikit-learn.org/stable/modules/model_evaluation.html#brier1950), the Brier score is given by:
B S ( Y , P ^ ) \= 1 N ā i \= 0 N ā 1 ā k \= 0 K ā 1 ( y i , k ā p ^ i , k ) 2
The Brier score lies in the interval \[ 0 , 2 \] and the lower the value the better the probability estimates are (the mean squared difference is smaller). Actually, the Brier score is a strictly proper scoring rule, meaning that it achieves the best score only when the estimated probabilities equal the true ones.
Note that in the binary case, the Brier score is usually divided by two and ranges between \[ 0 , 1 \]. For binary targets y i ā { 0 , 1 } and probability estimates p ^ i ā Pr ( y i \= 1 ) for the positive class, the Brier score is then equal to:
B S ( y , p ^ ) \= 1 N ā i \= 0 N ā 1 ( y i ā p ^ i ) 2
The [`brier_score_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html#sklearn.metrics.brier_score_loss "sklearn.metrics.brier_score_loss") function computes the Brier score given the ground-truth labels and predicted probabilities, as returned by an estimatorās `predict_proba` method. The `scale_by_half` parameter controls which of the two above definitions to follow.
```
>>> import numpy as np
>>> from sklearn.metrics import brier_score_loss
>>> y_true = np.array([0, 1, 1, 0])
>>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"])
>>> y_prob = np.array([0.1, 0.9, 0.8, 0.4])
>>> brier_score_loss(y_true, y_prob)
0.055
>>> brier_score_loss(y_true, 1 - y_prob, pos_label=0)
0.055
>>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham")
0.055
>>> brier_score_loss(
... ["eggs", "ham", "spam"],
... [[0.8, 0.1, 0.1], [0.2, 0.7, 0.1], [0.2, 0.2, 0.6]],
... labels=["eggs", "ham", "spam"],
... )
0.146
```
The Brier score can be used to assess how well a classifier is calibrated. However, a lower Brier score loss does not always mean a better calibration. This is because, by analogy with the bias-variance decomposition of the mean squared error, the Brier score loss can be decomposed as the sum of calibration loss and refinement loss [\[Bella2012\]](https://scikit-learn.org/stable/modules/model_evaluation.html#bella2012). Calibration loss is defined as the mean squared deviation from empirical probabilities derived from the slope of ROC segments. Refinement loss can be defined as the expected optimal loss as measured by the area under the optimal cost curve. Refinement loss can change independently from calibration loss, thus a lower Brier score loss does not necessarily mean a better calibrated model. āOnly when refinement loss remains the same does a lower Brier score loss always mean better calibrationā [\[Bella2012\]](https://scikit-learn.org/stable/modules/model_evaluation.html#bella2012), [\[Flach2008\]](https://scikit-learn.org/stable/modules/model_evaluation.html#flach2008).
Examples
- See [Probability calibration of classifiers](https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration.html#sphx-glr-auto-examples-calibration-plot-calibration-py) for an example of Brier score loss usage to perform probability calibration of classifiers.
References
\[Bella2012\] ([1](https://scikit-learn.org/stable/modules/model_evaluation.html#id48),[2](https://scikit-learn.org/stable/modules/model_evaluation.html#id49))
Bella, Ferri, HernĆ”ndez-Orallo, and RamĆrez-Quintana [āCalibration of Machine Learning Modelsā](http://dmip.webs.upv.es/papers/BFHRHandbook2010.pdf) in Khosrow-Pour, M. āMachine learning: concepts, methodologies, tools and applications.ā Hershey, PA: Information Science Reference (2012).
### 3\.4.4.19. Class likelihood ratios[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#class-likelihood-ratios "Link to this heading")
The [`class_likelihood_ratios`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.class_likelihood_ratios.html#sklearn.metrics.class_likelihood_ratios "sklearn.metrics.class_likelihood_ratios") function computes the [positive and negative likelihood ratios](https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing) L R Ā± for binary classes, which can be interpreted as the ratio of post-test to pre-test odds as explained below. As a consequence, this metric is invariant w.r.t. the class prevalence (the number of samples in the positive class divided by the total number of samples) and **can be extrapolated between populations regardless of any possible class imbalance.**
The L R Ā± metrics are therefore very useful in settings where the data available to learn and evaluate a classifier is a study population with nearly balanced classes, such as a case-control study, while the target application, i.e. the general population, has very low prevalence.
The positive likelihood ratio L R \+ is the probability of a classifier to correctly predict that a sample belongs to the positive class divided by the probability of predicting the positive class for a sample belonging to the negative class:
L R \+ \= PR ( P \+ \| T \+ ) PR ( P \+ \| T ā ) .
The notation here refers to predicted (P) or true (T) label and the sign \+ and ā refer to the positive and negative class, respectively, e.g. P \+ stands for āpredicted positiveā.
Analogously, the negative likelihood ratio L R ā is the probability of a sample of the positive class being classified as belonging to the negative class divided by the probability of a sample of the negative class being correctly classified:
L R ā \= PR ( P ā \| T \+ ) PR ( P ā \| T ā ) .
For classifiers above chance L R \+ above 1 **higher is better**, while L R ā ranges from 0 to 1 and **lower is better**. Values of L R Ā± ā 1 correspond to chance level.
Notice that probabilities differ from counts, for instance PR ( P \+ \| T \+ ) is not equal to the number of true positive counts `tp` (see [the wikipedia page](https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing) for the actual formulas).
Examples
- [Class Likelihood Ratios to measure classification performance](https://scikit-learn.org/stable/auto_examples/model_selection/plot_likelihood_ratios.html#sphx-glr-auto-examples-model-selection-plot-likelihood-ratios-py)
Both class likelihood ratios are interpretable in terms of an odds ratio (pre-test and post-tests):
post-test odds \= Likelihood ratio Ć pre-test odds .
Odds are in general related to probabilities via
odds \= probability 1 ā probability ,
or equivalently
probability \= odds 1 \+ odds .
On a given population, the pre-test probability is given by the prevalence. By converting odds to probabilities, the likelihood ratios can be translated into a probability of truly belonging to either class before and after a classifier prediction:
post-test odds \= Likelihood ratio Ć pre-test probability 1 ā pre-test probability ,
post-test probability \= post-test odds 1 \+ post-test odds .
The positive likelihood ratio (`LR+`) is undefined when f p \= 0, meaning the classifier does not misclassify any negative labels as positives. This condition can either indicate a perfect identification of all the negative cases or, if there are also no true positive predictions (t p \= 0), that the classifier does not predict the positive class at all. In the first case, `LR+` can be interpreted as `np.inf`, in the second case (for instance, with highly imbalanced data) it can be interpreted as `np.nan`.
The negative likelihood ratio (`LR-`) is undefined when t n \= 0. Such divergence is invalid, as L R ā \> 1\.0 would indicate an increase in the odds of a sample belonging to the positive class after being classified as negative, as if the act of classifying caused the positive condition. This includes the case of a [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html#sklearn.dummy.DummyClassifier "sklearn.dummy.DummyClassifier") that always predicts the positive class (i.e. when t n \= f n \= 0).
Both class likelihood ratios (`LR+ and LR-`) are undefined when t p \= f n \= 0, which means that no samples of the positive class were present in the test set. This can happen when cross-validating on highly imbalanced data and also leads to a division by zero.
If a division by zero occurs and `raise_warning` is set to `True` (default), [`class_likelihood_ratios`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.class_likelihood_ratios.html#sklearn.metrics.class_likelihood_ratios "sklearn.metrics.class_likelihood_ratios") raises an `UndefinedMetricWarning` and returns `np.nan` by default to avoid pollution when averaging over cross-validation folds. Users can set return values in case of a division by zero with the `replace_undefined_by` param.
For a worked-out demonstration of the [`class_likelihood_ratios`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.class_likelihood_ratios.html#sklearn.metrics.class_likelihood_ratios "sklearn.metrics.class_likelihood_ratios") function, see the example below.
- [Wikipedia entry for Likelihood ratios in diagnostic testing](https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing)
- Brenner, H., & Gefeller, O. (1997). Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Statistics in medicine, 16(9), 981-991.
### 3\.4.4.20. DĀ² score for classification[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#d2-score-for-classification "Link to this heading")
The DĀ² score computes the fraction of deviance explained. It is a generalization of RĀ², where the squared error is generalized and replaced by a classification deviance of choice dev ( y , y ^ ) (e.g., Log loss, Brier score,). DĀ² is a form of a *skill score*. It is calculated as
D 2 ( y , y ^ ) \= 1 ā dev ( y , y ^ ) dev ( y , y null ) .
Where y null is the optimal prediction of an intercept-only model (e.g., the per-class proportion of `y_true` in the case of the Log loss and Brier score).
Like RĀ², the best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts y null, disregarding the input features, would get a DĀ² score of 0.0.
The [`d2_log_loss_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_log_loss_score.html#sklearn.metrics.d2_log_loss_score "sklearn.metrics.d2_log_loss_score") function implements the special case of DĀ² with the log loss, see [Log loss](https://scikit-learn.org/stable/modules/model_evaluation.html#log-loss), i.e.:
dev ( y , y ^ ) \= log\_loss ( y , y ^ ) .
Here are some usage examples of the [`d2_log_loss_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_log_loss_score.html#sklearn.metrics.d2_log_loss_score "sklearn.metrics.d2_log_loss_score") function:
```
>>> from sklearn.metrics import d2_log_loss_score
>>> y_true = [1, 1, 2, 3]
>>> y_pred = [
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... ]
>>> d2_log_loss_score(y_true, y_pred)
0.0
>>> y_true = [1, 2, 3]
>>> y_pred = [
... [0.98, 0.01, 0.01],
... [0.01, 0.98, 0.01],
... [0.01, 0.01, 0.98],
... ]
>>> d2_log_loss_score(y_true, y_pred)
0.981
>>> y_true = [1, 2, 3]
>>> y_pred = [
... [0.1, 0.6, 0.3],
... [0.1, 0.6, 0.3],
... [0.4, 0.5, 0.1],
... ]
>>> d2_log_loss_score(y_true, y_pred)
-0.552
```
The [`d2_brier_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_brier_score.html#sklearn.metrics.d2_brier_score "sklearn.metrics.d2_brier_score") function implements the special case of DĀ² with the Brier score, see [Brier score loss](https://scikit-learn.org/stable/modules/model_evaluation.html#brier-score-loss), i.e.:
dev ( y , y ^ ) \= brier\_score\_loss ( y , y ^ ) .
This is also referred to as the Brier Skill Score (BSS).
Here are some usage examples of the [`d2_brier_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_brier_score.html#sklearn.metrics.d2_brier_score "sklearn.metrics.d2_brier_score") function:
```
>>> from sklearn.metrics import d2_brier_score
>>> y_true = [1, 1, 2, 3]
>>> y_pred = [
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... [0.5, 0.25, 0.25],
... ]
>>> d2_brier_score(y_true, y_pred)
0.0
>>> y_true = [1, 2, 3]
>>> y_pred = [
... [0.98, 0.01, 0.01],
... [0.01, 0.98, 0.01],
... [0.01, 0.01, 0.98],
... ]
>>> d2_brier_score(y_true, y_pred)
0.9991
>>> y_true = [1, 2, 3]
>>> y_pred = [
... [0.1, 0.6, 0.3],
... [0.1, 0.6, 0.3],
... [0.4, 0.5, 0.1],
... ]
>>> d2_brier_score(y_true, y_pred)
-0.370...
```
## 3\.4.5. Multilabel ranking metrics[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#multilabel-ranking-metrics "Link to this heading")
In multilabel learning, each sample can have any number of ground truth labels associated with it. The goal is to give high scores and better rank to the ground truth labels.
### 3\.4.5.1. Coverage error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#coverage-error "Link to this heading")
The [`coverage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.coverage_error.html#sklearn.metrics.coverage_error "sklearn.metrics.coverage_error") function computes the average number of labels that have to be included in the final prediction such that all true labels are predicted. This is useful if you want to know how many top-scored-labels you have to predict in average without missing any true one. The best value of this metric is thus the average number of true labels.
Note
Our implementationās score is 1 greater than the one given in Tsoumakas et al., 2010. This extends it to handle the degenerate case in which an instance has 0 true labels.
Formally, given a binary indicator matrix of the ground truth labels y ā { 0 , 1 } n samples Ć n labels and the score associated with each label f ^ ā R n samples Ć n labels, the coverage is defined as
c o v e r a g e ( y , f ^ ) \= 1 n samples ā i \= 0 n samples ā 1 max j : y i j \= 1 rank i j
with rank i j \= \| { k : f ^ i k ā„ f ^ i j } \|. Given the rank definition, ties in `y_scores` are broken by giving the maximal rank that would have been assigned to all tied values.
Here is a small example of usage of this function:
```
>>> import numpy as np
>>> from sklearn.metrics import coverage_error
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> coverage_error(y_true, y_score)
2.5
```
### 3\.4.5.2. Label ranking average precision[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#label-ranking-average-precision "Link to this heading")
The [`label_ranking_average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.label_ranking_average_precision_score.html#sklearn.metrics.label_ranking_average_precision_score "sklearn.metrics.label_ranking_average_precision_score") function implements label ranking average precision (LRAP). This metric is linked to the [`average_precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score "sklearn.metrics.average_precision_score") function, but is based on the notion of label ranking instead of precision and recall.
Label ranking average precision (LRAP) averages over the samples the answer to the following question: for each ground truth label, what fraction of higher-ranked labels were true labels? This performance measure will be higher if you are able to give better rank to the labels associated with each sample. The obtained score is always strictly greater than 0, and the best value is 1. If there is exactly one relevant label per sample, label ranking average precision is equivalent to the [mean reciprocal rank](https://en.wikipedia.org/wiki/Mean_reciprocal_rank).
Formally, given a binary indicator matrix of the ground truth labels y ā { 0 , 1 } n samples Ć n labels and the score associated with each label f ^ ā R n samples Ć n labels, the average precision is defined as
L R A P ( y , f ^ ) \= 1 n samples ā i \= 0 n samples ā 1 1 \| \| y i \| \| 0 ā j : y i j \= 1 \| L i j \| rank i j
where L i j \= { k : y i k \= 1 , f ^ i k ā„ f ^ i j }, rank i j \= \| { k : f ^ i k ā„ f ^ i j } \|, \| ā
\| computes the cardinality of the set (i.e., the number of elements in the set), and \| \| ā
\| \| 0 is the ā 0 ānormā (which computes the number of nonzero elements in a vector).
Here is a small example of usage of this function:
```
>>> import numpy as np
>>> from sklearn.metrics import label_ranking_average_precision_score
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> label_ranking_average_precision_score(y_true, y_score)
0.416
```
### 3\.4.5.3. Ranking loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#ranking-loss "Link to this heading")
The [`label_ranking_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.label_ranking_loss.html#sklearn.metrics.label_ranking_loss "sklearn.metrics.label_ranking_loss") function computes the ranking loss which averages over the samples the number of label pairs that are incorrectly ordered, i.e. true labels have a lower score than false labels, weighted by the inverse of the number of ordered pairs of false and true labels. The lowest achievable ranking loss is zero.
Formally, given a binary indicator matrix of the ground truth labels y ā { 0 , 1 } n samples Ć n labels and the score associated with each label f ^ ā R n samples Ć n labels, the ranking loss is defined as
r a n k i n g \_ l o s s ( y , f ^ ) \= 1 n samples ā i \= 0 n samples ā 1 1 \| \| y i \| \| 0 ( n labels ā \| \| y i \| \| 0 ) \| { ( k , l ) : f ^ i k ā¤ f ^ i l , y i k \= 1 , y i l \= 0 } \|
where \| ā
\| computes the cardinality of the set (i.e., the number of elements in the set) and \| \| ā
\| \| 0 is the ā 0 ānormā (which computes the number of nonzero elements in a vector).
Here is a small example of usage of this function:
```
>>> import numpy as np
>>> from sklearn.metrics import label_ranking_loss
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> label_ranking_loss(y_true, y_score)
0.75
>>> # With the following prediction, we have perfect and minimal loss
>>> y_score = np.array([[1.0, 0.1, 0.2], [0.1, 0.2, 0.9]])
>>> label_ranking_loss(y_true, y_score)
0.0
```
- Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In Data mining and knowledge discovery handbook (pp. 667-685). Springer US.
### 3\.4.5.4. Normalized Discounted Cumulative Gain[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#normalized-discounted-cumulative-gain "Link to this heading")
Discounted Cumulative Gain (DCG) and Normalized Discounted Cumulative Gain (NDCG) are ranking metrics implemented in [`dcg_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.dcg_score.html#sklearn.metrics.dcg_score "sklearn.metrics.dcg_score") and [`ndcg_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html#sklearn.metrics.ndcg_score "sklearn.metrics.ndcg_score") ; they compare a predicted order to ground-truth scores, such as the relevance of answers to a query.
From the Wikipedia page for Discounted Cumulative Gain:
āDiscounted cumulative gain (DCG) is a measure of ranking quality. In information retrieval, it is often used to measure effectiveness of web search engine algorithms or related applications. Using a graded relevance scale of documents in a search-engine result set, DCG measures the usefulness, or gain, of a document based on its position in the result list. The gain is accumulated from the top of the result list to the bottom, with the gain of each result discounted at lower ranks.ā
DCG orders the true targets (e.g. relevance of query answers) in the predicted order, then multiplies them by a logarithmic decay and sums the result. The sum can be truncated after the first K results, in which case we call it DCG@K. NDCG, or NDCG@K is DCG divided by the DCG obtained by a perfect prediction, so that it is always between 0 and 1. Usually, NDCG is preferred to DCG.
Compared with the ranking loss, NDCG can take into account relevance scores, rather than a ground-truth ranking. So if the ground-truth consists only of an ordering, the ranking loss should be preferred; if the ground-truth consists of actual usefulness scores (e.g. 0 for irrelevant, 1 for relevant, 2 for very relevant), NDCG can be used.
For one sample, given the vector of continuous ground-truth values for each target y ā R M, where M is the number of outputs, and the prediction y ^, which induces the ranking function f, the DCG score is
ā r \= 1 min ( K , M ) y f ( r ) log ā” ( 1 \+ r )
and the NDCG score is the DCG score divided by the DCG score obtained for y.
- [Wikipedia entry for Discounted Cumulative Gain](https://en.wikipedia.org/wiki/Discounted_cumulative_gain)
- Jarvelin, K., & Kekalainen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422-446.
- Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013, May). A theoretical analysis of NDCG ranking measures. In Proceedings of the 26th Annual Conference on Learning Theory (COLT 2013)
- McSherry, F., & Najork, M. (2008, March). Computing information retrieval performance measures efficiently in the presence of tied scores. In European conference on information retrieval (pp. 414-421). Springer, Berlin, Heidelberg.
## 3\.4.6. Regression metrics[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics "Link to this heading")
The [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") module implements several loss, score, and utility functions to measure regression performance. Some of those have been enhanced to handle the multioutput case: [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error"), [`mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error"), [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score"), [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score"), [`mean_pinball_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_pinball_loss.html#sklearn.metrics.mean_pinball_loss "sklearn.metrics.mean_pinball_loss"), [`d2_pinball_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_pinball_score.html#sklearn.metrics.d2_pinball_score "sklearn.metrics.d2_pinball_score") and [`d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score").
These functions have a `multioutput` keyword argument which specifies the way the scores or losses for each individual target should be averaged. The default is `'uniform_average'`, which specifies a uniformly weighted mean over outputs. If an `ndarray` of shape `(n_outputs,)` is passed, then its entries are interpreted as weights and an according weighted average is returned. If `multioutput` is `'raw_values'`, then all unaltered individual scores or losses will be returned in an array of shape `(n_outputs,)`.
The [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") and [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") accept an additional value `'variance_weighted'` for the `multioutput` parameter. This option leads to a weighting of each individual score by the variance of the corresponding target variable. This setting quantifies the globally captured unscaled variance. If the target variables are of different scale, then this score puts more importance on explaining the higher variance variables.
### 3\.4.6.1. RĀ² score, the coefficient of determination[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score-the-coefficient-of-determination "Link to this heading")
The [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") function computes the [coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination), usually denoted as R 2.
It represents the proportion of variance (of y) that has been explained by the independent variables in the model. It provides an indication of goodness of fit and therefore a measure of how well unseen samples are likely to be predicted by the model, through the proportion of explained variance.
As such variance is dataset dependent, R 2 may not be meaningfully comparable across different datasets. Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected (average) value of y, disregarding the input features, would get an R 2 score of 0.0.
Note: when the prediction residuals have zero mean, the R 2 score and the [Explained variance score](https://scikit-learn.org/stable/modules/model_evaluation.html#explained-variance-score) are identical.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value for total n samples, the estimated R 2 is defined as:
R 2 ( y , y ^ ) \= 1 ā ā i \= 1 n ( y i ā y ^ i ) 2 ā i \= 1 n ( y i ā y ĀÆ ) 2
where y ĀÆ \= 1 n ā i \= 1 n y i and ā i \= 1 n ( y i ā y ^ i ) 2 \= ā i \= 1 n Ļµ i 2.
Note that [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") calculates unadjusted R 2 without correcting for bias in sample variance of y.
In the particular case where the true target is constant, the R 2 score is not finite: it is either `NaN` (perfect predictions) or `-Inf` (imperfect predictions). Such non-finite scores may prevent correct model optimization such as grid-search cross-validation to be performed correctly. For this reason the default behaviour of [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") is to replace them with 1.0 (perfect predictions) or 0.0 (imperfect predictions). If `force_finite` is set to `False`, this score falls back on the original R 2 definition.
Here is a small example of usage of the [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") function:
```
>>> from sklearn.metrics import r2_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> r2_score(y_true, y_pred)
0.948
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> r2_score(y_true, y_pred, multioutput='variance_weighted')
0.938
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> r2_score(y_true, y_pred, multioutput='uniform_average')
0.936
>>> r2_score(y_true, y_pred, multioutput='raw_values')
array([0.965, 0.908])
>>> r2_score(y_true, y_pred, multioutput=[0.3, 0.7])
0.925
>>> y_true = [-2, -2, -2]
>>> y_pred = [-2, -2, -2]
>>> r2_score(y_true, y_pred)
1.0
>>> r2_score(y_true, y_pred, force_finite=False)
nan
>>> y_true = [-2, -2, -2]
>>> y_pred = [-2, -2, -2 + 1e-8]
>>> r2_score(y_true, y_pred)
0.0
>>> r2_score(y_true, y_pred, force_finite=False)
-inf
```
Examples
- See [L1-based models for Sparse Signals](https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_and_elasticnet.html#sphx-glr-auto-examples-linear-model-plot-lasso-and-elasticnet-py) for an example of RĀ² score usage to evaluate Lasso and Elastic Net on sparse signals.
### 3\.4.6.2. Mean absolute error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-error "Link to this heading")
The [`mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error") function computes [mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error), a risk metric corresponding to the expected value of the absolute error loss or l 1\-norm loss.
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the mean absolute error (MAE) estimated over n samples is defined as
MAE ( y , y ^ ) \= 1 n samples ā i \= 0 n samples ā 1 \| y i ā y ^ i \| .
Here is a small example of usage of the [`mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error") function:
```
>>> from sklearn.metrics import mean_absolute_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_absolute_error(y_true, y_pred)
0.5
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_absolute_error(y_true, y_pred)
0.75
>>> mean_absolute_error(y_true, y_pred, multioutput='raw_values')
array([0.5, 1. ])
>>> mean_absolute_error(y_true, y_pred, multioutput=[0.3, 0.7])
0.85
```
### 3\.4.6.3. Mean squared error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-error "Link to this heading")
The [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error") function computes [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error), a risk metric corresponding to the expected value of the squared (quadratic) error or loss.
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the mean squared error (MSE) estimated over n samples is defined as
MSE ( y , y ^ ) \= 1 n samples ā i \= 0 n samples ā 1 ( y i ā y ^ i ) 2 .
Here is a small example of usage of the [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error") function:
```
>>> from sklearn.metrics import mean_squared_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_squared_error(y_true, y_pred)
0.375
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_squared_error(y_true, y_pred)
0.7083
```
Examples
- See [Gradient Boosting regression](https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_regression.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-regression-py) for an example of mean squared error usage to evaluate gradient boosting regression.
Taking the square root of the MSE, called the root mean squared error (RMSE), is another common metric that provides a measure in the same units as the target variable. RMSE is available through the [`root_mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.root_mean_squared_error.html#sklearn.metrics.root_mean_squared_error "sklearn.metrics.root_mean_squared_error") function.
### 3\.4.6.4. Mean squared logarithmic error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-logarithmic-error "Link to this heading")
The [`mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_log_error.html#sklearn.metrics.mean_squared_log_error "sklearn.metrics.mean_squared_log_error") function computes a risk metric corresponding to the expected value of the squared logarithmic (quadratic) error or loss.
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the mean squared logarithmic error (MSLE) estimated over n samples is defined as
MSLE ( y , y ^ ) \= 1 n samples ā i \= 0 n samples ā 1 ( log e ā” ( 1 \+ y i ) ā log e ā” ( 1 \+ y ^ i ) ) 2 .
Where log e ā” ( x ) means the natural logarithm of x. This metric is best to use when targets having exponential growth, such as population counts, average sales of a commodity over a span of years etc. Note that this metric penalizes an under-predicted estimate greater than an over-predicted estimate.
Here is a small example of usage of the [`mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_log_error.html#sklearn.metrics.mean_squared_log_error "sklearn.metrics.mean_squared_log_error") function:
```
>>> from sklearn.metrics import mean_squared_log_error
>>> y_true = [3, 5, 2.5, 7]
>>> y_pred = [2.5, 5, 4, 8]
>>> mean_squared_log_error(y_true, y_pred)
0.0397
>>> y_true = [[0.5, 1], [1, 2], [7, 6]]
>>> y_pred = [[0.5, 2], [1, 2.5], [8, 8]]
>>> mean_squared_log_error(y_true, y_pred)
0.044
```
The root mean squared logarithmic error (RMSLE) is available through the [`root_mean_squared_log_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.root_mean_squared_log_error.html#sklearn.metrics.root_mean_squared_log_error "sklearn.metrics.root_mean_squared_log_error") function.
### 3\.4.6.5. Mean absolute percentage error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error "Link to this heading")
The [`mean_absolute_percentage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html#sklearn.metrics.mean_absolute_percentage_error "sklearn.metrics.mean_absolute_percentage_error") (MAPE), also known as mean absolute percentage deviation (MAPD), is an evaluation metric for regression problems. The idea of this metric is to be sensitive to relative errors. It is for example not changed by a global scaling of the target variable.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value, then the mean absolute percentage error (MAPE) estimated over n samples is defined as
MAPE ( y , y ^ ) \= 1 n samples ā i \= 0 n samples ā 1 \| y i ā y ^ i \| max ( Ļµ , \| y i \| )
where Ļµ is an arbitrary small yet strictly positive number to avoid undefined results when y is zero.
The [`mean_absolute_percentage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html#sklearn.metrics.mean_absolute_percentage_error "sklearn.metrics.mean_absolute_percentage_error") function supports multioutput.
Here is a small example of usage of the [`mean_absolute_percentage_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html#sklearn.metrics.mean_absolute_percentage_error "sklearn.metrics.mean_absolute_percentage_error") function:
```
>>> from sklearn.metrics import mean_absolute_percentage_error
>>> y_true = [1, 10, 1e6]
>>> y_pred = [0.9, 15, 1.2e6]
>>> mean_absolute_percentage_error(y_true, y_pred)
0.2666
```
In above example, if we had used `mean_absolute_error`, it would have ignored the small magnitude values and only reflected the error in prediction of highest magnitude value. But that problem is resolved in case of MAPE because it calculates relative percentage error with respect to actual output.
Note
The MAPE formula here does not represent the common āpercentageā definition: the percentage in the range \[0, 100\] is converted to a relative value in the range \[0, 1\] by dividing by 100. Thus, an error of 200% corresponds to a relative error of 2. The motivation here is to have a range of values that is more consistent with other error metrics in scikit-learn, such as `accuracy_score`.
To obtain the mean absolute percentage error as per the Wikipedia formula, multiply the `mean_absolute_percentage_error` computed here by 100.
### 3\.4.6.6. Median absolute error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#median-absolute-error "Link to this heading")
The [`median_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.median_absolute_error.html#sklearn.metrics.median_absolute_error "sklearn.metrics.median_absolute_error") is particularly interesting because it is robust to outliers. The loss is calculated by taking the median of all absolute differences between the target and the prediction.
If y ^ i is the predicted value of the i\-th sample and y i is the corresponding true value, then the median absolute error (MedAE) estimated over n samples is defined as
MedAE ( y , y ^ ) \= median ( ā£ y 1 ā y ^ 1 ā£ , ā¦ , ā£ y n ā y ^ n ā£ ) .
The [`median_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.median_absolute_error.html#sklearn.metrics.median_absolute_error "sklearn.metrics.median_absolute_error") does not support multioutput.
Here is a small example of usage of the [`median_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.median_absolute_error.html#sklearn.metrics.median_absolute_error "sklearn.metrics.median_absolute_error") function:
```
>>> from sklearn.metrics import median_absolute_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> median_absolute_error(y_true, y_pred)
0.5
```
### 3\.4.6.7. Max error[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#max-error "Link to this heading")
The [`max_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.max_error.html#sklearn.metrics.max_error "sklearn.metrics.max_error") function computes the maximum [residual error](https://en.wikipedia.org/wiki/Errors_and_residuals) , a metric that captures the worst case error between the predicted value and the true value. In a perfectly fitted single output regression model, `max_error` would be `0` on the training set and though this would be highly unlikely in the real world, this metric shows the extent of error that the model had when it was fitted.
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the max error is defined as
Max Error ( y , y ^ ) \= max ( \| y i ā y ^ i \| )
Here is a small example of usage of the [`max_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.max_error.html#sklearn.metrics.max_error "sklearn.metrics.max_error") function:
```
>>> from sklearn.metrics import max_error
>>> y_true = [3, 2, 7, 1]
>>> y_pred = [9, 2, 7, 1]
>>> max_error(y_true, y_pred)
6.0
```
The [`max_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.max_error.html#sklearn.metrics.max_error "sklearn.metrics.max_error") does not support multioutput.
### 3\.4.6.8. Explained variance score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#explained-variance-score "Link to this heading")
The [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") computes the [explained variance regression score](https://en.wikipedia.org/wiki/Explained_variation).
If y ^ is the estimated target output, y the corresponding (correct) target output, and V a r is [Variance](https://en.wikipedia.org/wiki/Variance), the square of the standard deviation, then the explained variance is estimated as follow:
e x p l a i n e d \_ v a r i a n c e ( y , y ^ ) \= 1 ā V a r { y ā y ^ } V a r { y }
The best possible score is 1.0, lower values are worse.
In the particular case where the true target is constant, the Explained Variance score is not finite: it is either `NaN` (perfect predictions) or `-Inf` (imperfect predictions). Such non-finite scores may prevent correct model optimization such as grid-search cross-validation to be performed correctly. For this reason the default behaviour of [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") is to replace them with 1.0 (perfect predictions) or 0.0 (imperfect predictions). You can set the `force_finite` parameter to `False` to prevent this fix from happening and fallback on the original Explained Variance score.
Here is a small example of usage of the [`explained_variance_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.explained_variance_score.html#sklearn.metrics.explained_variance_score "sklearn.metrics.explained_variance_score") function:
```
>>> from sklearn.metrics import explained_variance_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> explained_variance_score(y_true, y_pred)
0.957
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> explained_variance_score(y_true, y_pred, multioutput='raw_values')
array([0.967, 1. ])
>>> explained_variance_score(y_true, y_pred, multioutput=[0.3, 0.7])
0.990
>>> y_true = [-2, -2, -2]
>>> y_pred = [-2, -2, -2]
>>> explained_variance_score(y_true, y_pred)
1.0
>>> explained_variance_score(y_true, y_pred, force_finite=False)
nan
>>> y_true = [-2, -2, -2]
>>> y_pred = [-2, -2, -2 + 1e-8]
>>> explained_variance_score(y_true, y_pred)
0.0
>>> explained_variance_score(y_true, y_pred, force_finite=False)
-inf
```
### 3\.4.6.9. Mean Poisson, Gamma, and Tweedie deviances[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-poisson-gamma-and-tweedie-deviances "Link to this heading")
The [`mean_tweedie_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_tweedie_deviance.html#sklearn.metrics.mean_tweedie_deviance "sklearn.metrics.mean_tweedie_deviance") function computes the [mean Tweedie deviance error](https://en.wikipedia.org/wiki/Tweedie_distribution#The_Tweedie_deviance) with a `power` parameter (p). This is a metric that elicits predicted expectation values of regression targets.
Following special cases exist,
- when `power=0` it is equivalent to [`mean_squared_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error "sklearn.metrics.mean_squared_error").
- when `power=1` it is equivalent to [`mean_poisson_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_poisson_deviance.html#sklearn.metrics.mean_poisson_deviance "sklearn.metrics.mean_poisson_deviance").
- when `power=2` it is equivalent to [`mean_gamma_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_gamma_deviance.html#sklearn.metrics.mean_gamma_deviance "sklearn.metrics.mean_gamma_deviance").
If y ^ i is the predicted value of the i\-th sample, and y i is the corresponding true value, then the mean Tweedie deviance error (D) for power p, estimated over n samples is defined as
D ( y , y ^ ) \= 1 n samples ā i \= 0 n samples ā 1 { ( y i ā y ^ i ) 2 , for p \= 0 (Normal) 2 ( y i log ā” ( y i / y ^ i ) \+ y ^ i ā y i ) , for p \= 1 (Poisson) 2 ( log ā” ( y ^ i / y i ) \+ y i / y ^ i ā 1 ) , for p \= 2 (Gamma) 2 ( max ( y i , 0 ) 2 ā p ( 1 ā p ) ( 2 ā p ) ā y i y ^ i 1 ā p 1 ā p \+ y ^ i 2 ā p 2 ā p ) , otherwise
Tweedie deviance is a homogeneous function of degree `2-power`. Thus, Gamma distribution with `power=2` means that simultaneously scaling `y_true` and `y_pred` has no effect on the deviance. For Poisson distribution `power=1` the deviance scales linearly, and for Normal distribution (`power=0`), quadratically. In general, the higher `power` the less weight is given to extreme deviations between true and predicted targets.
For instance, letās compare the two predictions 1.5 and 150 that are both 50% larger than their corresponding true value.
The mean squared error (`power=0`) is very sensitive to the prediction difference of the second point,:
```
>>> from sklearn.metrics import mean_tweedie_deviance
>>> mean_tweedie_deviance([1.0], [1.5], power=0)
0.25
>>> mean_tweedie_deviance([100.], [150.], power=0)
2500.0
```
If we increase `power` to 1,:
```
>>> mean_tweedie_deviance([1.0], [1.5], power=1)
0.189
>>> mean_tweedie_deviance([100.], [150.], power=1)
18.9
```
the difference in errors decreases. Finally, by setting, `power=2`:
```
>>> mean_tweedie_deviance([1.0], [1.5], power=2)
0.144
>>> mean_tweedie_deviance([100.], [150.], power=2)
0.144
```
we would get identical errors. The deviance when `power=2` is thus only sensitive to relative errors.
### 3\.4.6.10. Pinball loss[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss "Link to this heading")
The [`mean_pinball_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_pinball_loss.html#sklearn.metrics.mean_pinball_loss "sklearn.metrics.mean_pinball_loss") function is used to evaluate the predictive performance of [quantile regression](https://en.wikipedia.org/wiki/Quantile_regression) models.
pinball ( y , y ^ ) \= 1 n samples ā i \= 0 n samples ā 1 Ī± max ( y i ā y ^ i , 0 ) \+ ( 1 ā Ī± ) max ( y ^ i ā y i , 0 )
The value of pinball loss is equivalent to half of [`mean_absolute_error`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error "sklearn.metrics.mean_absolute_error") when the quantile parameter `alpha` is set to 0.5.
Here is a small example of usage of the [`mean_pinball_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_pinball_loss.html#sklearn.metrics.mean_pinball_loss "sklearn.metrics.mean_pinball_loss") function:
```
>>> from sklearn.metrics import mean_pinball_loss
>>> y_true = [1, 2, 3]
>>> mean_pinball_loss(y_true, [0, 2, 3], alpha=0.1)
0.033
>>> mean_pinball_loss(y_true, [1, 2, 4], alpha=0.1)
0.3
>>> mean_pinball_loss(y_true, [0, 2, 3], alpha=0.9)
0.3
>>> mean_pinball_loss(y_true, [1, 2, 4], alpha=0.9)
0.033
>>> mean_pinball_loss(y_true, y_true, alpha=0.1)
0.0
>>> mean_pinball_loss(y_true, y_true, alpha=0.9)
0.0
```
It is possible to build a scorer object with a specific choice of `alpha`:
```
>>> from sklearn.metrics import make_scorer
>>> mean_pinball_loss_95p = make_scorer(mean_pinball_loss, alpha=0.95)
```
Such a scorer can be used to evaluate the generalization performance of a quantile regressor via cross-validation:
```
>>> from sklearn.datasets import make_regression
>>> from sklearn.model_selection import cross_val_score
>>> from sklearn.ensemble import GradientBoostingRegressor
>>>
>>> X, y = make_regression(n_samples=100, random_state=0)
>>> estimator = GradientBoostingRegressor(
... loss="quantile",
... alpha=0.95,
... random_state=0,
... )
>>> cross_val_score(estimator, X, y, cv=5, scoring=mean_pinball_loss_95p)
array([13.6, 9.7, 23.3, 9.5, 10.4])
```
It is also possible to build scorer objects for hyper-parameter tuning. The sign of the loss must be switched to ensure that greater means better as explained in the example linked below.
Examples
- See [Prediction Intervals for Gradient Boosting Regression](https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_quantile.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-quantile-py) for an example of using the pinball loss to evaluate and tune the hyper-parameters of quantile regression models on data with non-symmetric noise and outliers.
### 3\.4.6.11. DĀ² score[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#d2-score "Link to this heading")
The DĀ² score computes the fraction of deviance explained. It is a generalization of RĀ², where the squared error is generalized and replaced by a deviance of choice dev ( y , y ^ ) (e.g., Tweedie, pinball or mean absolute error). DĀ² is a form of a *skill score*. It is calculated as
D 2 ( y , y ^ ) \= 1 ā dev ( y , y ^ ) dev ( y , y null ) .
Where y null is the optimal prediction of an intercept-only model (e.g., the mean of `y_true` for the Tweedie case, the median for absolute error and the alpha-quantile for pinball loss).
Like RĀ², the best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts y null, disregarding the input features, would get a DĀ² score of 0.0.
The [`d2_tweedie_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_tweedie_score.html#sklearn.metrics.d2_tweedie_score "sklearn.metrics.d2_tweedie_score") function implements the special case of DĀ² where dev ( y , y ^ ) is the Tweedie deviance, see [Mean Poisson, Gamma, and Tweedie deviances](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance). It is also known as DĀ² Tweedie and is related to McFaddenās likelihood ratio index.
The argument `power` defines the Tweedie power as for [`mean_tweedie_deviance`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_tweedie_deviance.html#sklearn.metrics.mean_tweedie_deviance "sklearn.metrics.mean_tweedie_deviance"). Note that for `power=0`, [`d2_tweedie_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_tweedie_score.html#sklearn.metrics.d2_tweedie_score "sklearn.metrics.d2_tweedie_score") equals [`r2_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score "sklearn.metrics.r2_score") (for single targets).
A scorer object with a specific choice of `power` can be built by:
```
>>> from sklearn.metrics import d2_tweedie_score, make_scorer
>>> d2_tweedie_score_15 = make_scorer(d2_tweedie_score, power=1.5)
```
The [`d2_pinball_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_pinball_score.html#sklearn.metrics.d2_pinball_score "sklearn.metrics.d2_pinball_score") function implements the special case of DĀ² with the pinball loss, see [Pinball loss](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss), i.e.:
dev ( y , y ^ ) \= pinball ( y , y ^ ) .
The argument `alpha` defines the slope of the pinball loss as for [`mean_pinball_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_pinball_loss.html#sklearn.metrics.mean_pinball_loss "sklearn.metrics.mean_pinball_loss") ([Pinball loss](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss)). It determines the quantile level `alpha` for which the pinball loss and also DĀ² are optimal. Note that for `alpha=0.5` (the default) [`d2_pinball_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_pinball_score.html#sklearn.metrics.d2_pinball_score "sklearn.metrics.d2_pinball_score") equals [`d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score").
A scorer object with a specific choice of `alpha` can be built by:
```
>>> from sklearn.metrics import d2_pinball_score, make_scorer
>>> d2_pinball_score_08 = make_scorer(d2_pinball_score, alpha=0.8)
```
The [`d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score") function implements the special case of the [Mean absolute error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-error):
dev ( y , y ^ ) \= MAE ( y , y ^ ) .
Here are some usage examples of the [`d2_absolute_error_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.d2_absolute_error_score.html#sklearn.metrics.d2_absolute_error_score "sklearn.metrics.d2_absolute_error_score") function:
```
>>> from sklearn.metrics import d2_absolute_error_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> d2_absolute_error_score(y_true, y_pred)
0.764
>>> y_true = [1, 2, 3]
>>> y_pred = [1, 2, 3]
>>> d2_absolute_error_score(y_true, y_pred)
1.0
>>> y_true = [1, 2, 3]
>>> y_pred = [2, 2, 2]
>>> d2_absolute_error_score(y_true, y_pred)
0.0
```
### 3\.4.6.12. Visual evaluation of regression models[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#visual-evaluation-of-regression-models "Link to this heading")
Among methods to assess the quality of regression models, scikit-learn provides the [`PredictionErrorDisplay`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.PredictionErrorDisplay.html#sklearn.metrics.PredictionErrorDisplay "sklearn.metrics.PredictionErrorDisplay") class. It allows to visually inspect the prediction errors of a model in two different manners.
[](https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_predict.html)
The plot on the left shows the actual values vs predicted values. For a noise-free regression task aiming to predict the (conditional) expectation of `y`, a perfect regression model would display data points on the diagonal defined by predicted equal to actual values. The further away from this optimal line, the larger the error of the model. In a more realistic setting with irreducible noise, that is, when not all the variations of `y` can be explained by features in `X`, then the best model would lead to a cloud of points densely arranged around the diagonal.
Note that the above only holds when the predicted values is the expected value of `y` given `X`. This is typically the case for regression models that minimize the mean squared error objective function or more generally the [mean Tweedie deviance](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-tweedie-deviance) for any value of its āpowerā parameter.
When plotting the predictions of an estimator that predicts a quantile of `y` given `X`, e.g. [`QuantileRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.QuantileRegressor.html#sklearn.linear_model.QuantileRegressor "sklearn.linear_model.QuantileRegressor") or any other model minimizing the [pinball loss](https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss), a fraction of the points are either expected to lie above or below the diagonal depending on the estimated quantile level.
All in all, while intuitive to read, this plot does not really inform us on what to do to obtain a better model.
The right-hand side plot shows the residuals (i.e. the difference between the actual and the predicted values) vs. the predicted values.
This plot makes it easier to visualize if the residuals follow and [homoscedastic or heteroschedastic](https://en.wikipedia.org/wiki/Homoscedasticity_and_heteroscedasticity) distribution.
In particular, if the true distribution of `y|X` is Poisson or Gamma distributed, it is expected that the variance of the residuals of the optimal model would grow with the predicted value of `E[y|X]` (either linearly for Poisson or quadratically for Gamma).
When fitting a linear least squares regression model (see [`LinearRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression "sklearn.linear_model.LinearRegression") and [`Ridge`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge "sklearn.linear_model.Ridge")), we can use this plot to check if some of the [model assumptions](https://en.wikipedia.org/wiki/Ordinary_least_squares#Assumptions) are met, in particular that the residuals should be uncorrelated, their expected value should be null and that their variance should be constant (homoschedasticity).
If this is not the case, and in particular if the residuals plot show some banana-shaped structure, this is a hint that the model is likely mis-specified and that non-linear feature engineering or switching to a non-linear regression model might be useful.
Refer to the example below to see a model evaluation that makes use of this display.
Examples
- See [Effect of transforming the targets in regression model](https://scikit-learn.org/stable/auto_examples/compose/plot_transformed_target.html#sphx-glr-auto-examples-compose-plot-transformed-target-py) for an example on how to use [`PredictionErrorDisplay`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.PredictionErrorDisplay.html#sklearn.metrics.PredictionErrorDisplay "sklearn.metrics.PredictionErrorDisplay") to visualize the prediction quality improvement of a regression model obtained by transforming the target before learning.
## 3\.4.7. Clustering metrics[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#clustering-metrics "Link to this heading")
The [`sklearn.metrics`](https://scikit-learn.org/stable/api/sklearn.metrics.html#module-sklearn.metrics "sklearn.metrics") module implements several loss, score, and utility functions to measure clustering performance. For more information see the [Clustering performance evaluation](https://scikit-learn.org/stable/modules/clustering.html#clustering-evaluation) section for instance clustering, and [Biclustering evaluation](https://scikit-learn.org/stable/modules/biclustering.html#biclustering-evaluation) for biclustering.
## 3\.4.8. Dummy estimators[\#](https://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators "Link to this heading")
When doing supervised learning, a simple sanity check consists of comparing oneās estimator against simple rules of thumb. [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html#sklearn.dummy.DummyClassifier "sklearn.dummy.DummyClassifier") implements several such simple strategies for classification:
- `stratified` generates random predictions by respecting the training set class distribution.
- `most_frequent` always predicts the most frequent label in the training set.
- `prior` always predicts the class that maximizes the class prior (like `most_frequent`) and `predict_proba` returns the class prior.
- `uniform` generates predictions uniformly at random.
- `constant` always predicts a constant label that is provided by the user.
A major motivation of this method is F1-scoring, when the positive class is in the minority.
Note that with all these strategies, the `predict` method completely ignores the input data\!
To illustrate [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html#sklearn.dummy.DummyClassifier "sklearn.dummy.DummyClassifier"), first letās create an imbalanced dataset:
```
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> X, y = load_iris(return_X_y=True)
>>> y[y != 1] = -1
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
```
Next, letās compare the accuracy of `SVC` and `most_frequent`:
```
>>> from sklearn.dummy import DummyClassifier
>>> from sklearn.svm import SVC
>>> clf = SVC(kernel='linear', C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test)
0.63
>>> clf = DummyClassifier(strategy='most_frequent', random_state=0)
>>> clf.fit(X_train, y_train)
DummyClassifier(random_state=0, strategy='most_frequent')
>>> clf.score(X_test, y_test)
0.579
```
We see that `SVC` doesnāt do much better than a dummy classifier. Now, letās change the kernel:
```
>>> clf = SVC(kernel='rbf', C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test)
0.94
```
We see that the accuracy was boosted to almost 100%. A cross validation strategy is recommended for a better estimate of the accuracy, if it is not too CPU costly. For more information see the [Cross-validation: evaluating estimator performance](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation) section. Moreover if you want to optimize over the parameter space, it is highly recommended to use an appropriate methodology; see the [Tuning the hyper-parameters of an estimator](https://scikit-learn.org/stable/modules/grid_search.html#grid-search) section for details.
More generally, when the accuracy of a classifier is too close to random, it probably means that something went wrong: features are not helpful, a hyperparameter is not correctly tuned, the classifier is suffering from class imbalance, etcā¦
[`DummyRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html#sklearn.dummy.DummyRegressor "sklearn.dummy.DummyRegressor") also implements four simple rules of thumb for regression:
- `mean` always predicts the mean of the training targets.
- `median` always predicts the median of the training targets.
- `quantile` always predicts a user provided quantile of the training targets.
- `constant` always predicts a constant value that is provided by the user.
In all these strategies, the `predict` method completely ignores the input data. |
| Shard | 148 (laksa) |
| Root Hash | 6052685795207125548 |
| Unparsed URL | org,scikit-learn!/stable/modules/model_evaluation.html s443 |