Accuracy

Description

Defined as: mean(response == truth)

Usage

ACC(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
response = as.factor(sample(c(1,2,3), n, replace = TRUE))
ACC(truth, response)

Adjusted coefficient of determination

Description

Defined as: 1 - (1 - rsq) * (p / (n - p - 1L)). Adjusted R-squared is only defined for normal linear regression.

Usage

ARSQ(truth, response, n, p)

Arguments

Examples

n = 20
p = 5
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
ARSQ(truth, response, n, p)

Area under the curve

Description

Integral over the graph that results from computing fpr and tpr for many different thresholds.

Usage

AUC(probabilities, truth, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
AUC(probabilities, truth, negative, positive)

Balanced accuracy

Description

Mean of true positive rate and true negative rate.

Usage

BAC(truth, response, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
BAC(truth, response, negative, positive)

Balanced error rate

Description

Mean of misclassification error rates on all individual classes.

Usage

BER(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
response = as.factor(sample(c(1,2,3), n, replace = TRUE))
BER(truth, response)

Brier score

Description

The Brier score is defined as the quadratic difference between the probability and the value (1,0) for the class. That means we use the numeric representation 1 and 0 for our target classes. It is similiar to the mean squared error in regression. multiclass.brier is the sum over all one vs. all comparisons and for a binary classifcation 2 * brier.

Usage

Brier(probabilities, truth, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
Brier(probabilities, truth, negative, positive)

Brier scaled

Description

Brier score scaled to [0,1], see http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575184/.

Usage

BrierScaled(probabilities, truth, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
BrierScaled(probabilities, truth, negative, positive)

Explained variance

Description

Similar to RSQ (R-squared). Defined as explained_sum_of_squares / total_sum_of_squares.

Usage

EXPVAR(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
EXPVAR(truth, response)

F1 measure

Description

Defined as: 2 * tp/ (sum(truth == positive) + sum(response == positive))

Usage

F1(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
F1(truth, response, positive)

False discovery rate

Description

Defined as: fp / (tp + fp)

Usage

FDR(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
FDR(truth, response, positive)

False negatives

Description

Sum of misclassified observations in the negative class. Also called misses.

Usage

FN(truth, response, negative)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
negative = 0
FN(truth, response, negative)

False negative rate

Description

Percentage of misclassified observations in the negative class.

Usage

FNR(truth, response, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
FNR(truth, response, negative, positive)

False positives

Description

Sum of misclassified observations in the positive class. Also called false alarms.

Usage

FP(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
FP(truth, response, positive)

False positive rate

Description

Percentage of misclassified observations in the positive class. Also called false alarm rate or fall-out.

Usage

FPR(truth, response, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
FPR(truth, response, negative, positive)

G-mean

Description

Geometric mean of recall and specificity.

Usage

GMEAN(truth, response, negative, positive)

Arguments

References

He, H. & Garcia, E. A. (2009) *Learning from Imbalanced Data.* IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9. pp. 1263-1284.

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
GMEAN(truth, response, negative, positive)

Geometric mean of precision and recall.

Description

Defined as: sqrt(ppv * tpr)

Usage

GPR(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
GPR(truth, response, positive)

Cohen's kappa

Description

Defined as: 1 - (1 - p0) / (1 - pe). With: p0 = 'observed frequency of agreement' and pe = 'expected agremeent frequency under independence

Usage

KAPPA(truth, response)

Arguments

Kendall's tau

Description

Defined as: Kendall's tau correlation between truth and response. Only looks at the order. See Rosset et al.: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.1398&rep=rep1&type=pdf.

Usage

KendallTau(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
KendallTau(truth, response)

Logarithmic Scoring Rule

Description

Defined as: mean(log(p_i)), where p_i is the predicted probability of the true class of observation i. This scoring rule is the same as the negative logloss, self-information or surprisal. See: Bickel, J. E. (2007). Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Analysis, 4(2), 49-65.

Usage

LSR(probabilities, truth)

Arguments

Logarithmic loss

Description

Defined as: -mean(log(p_i)), where p_i is the predicted probability of the true class of observation i. Inspired by https://www.kaggle.com/wiki/MultiClassLogLoss.

Usage

Logloss(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
Logloss(probabilities, truth)

Mean of absolute errors

Description

Defined as: mean(abs(response - truth))

Usage

MAE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MAE(truth, response)

Mean absolute percentage error

Description

Defined as the abs(truth_i - response_i) / truth_i. Won't work if any truth value is equal to zero. In this case the output will be NA.

Usage

MAPE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MAPE(truth, response)

Matthews correlation coefficient

Description

Defined as (tp * tn - fp * fn) / sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)), denominator set to 1 if 0.

Usage

MCC(truth, response, negative, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
negative = 0
MCC(truth, response, negative, positive)

Median of absolute errors

Description

Defined as: median(abs(response - truth)).

Usage

MEDAE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MEDAE(truth, response)

Median of squared errors

Description

Defined as: median((response - truth)^2).

Usage

MEDSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MEDSE(truth, response)

Mean misclassification error

Description

Defined as: mean(response != truth)

Usage

MMCE(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
response = as.factor(sample(c(1,2,3), n, replace = TRUE))
MMCE(truth, response)

Mean of squared errors

Description

Defined as: mean((response - truth)^2)

Usage

MSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
MSE(truth, response)

Mean squared logarithmic error

Description

Defined as: mean((log(response + 1, exp(1)) - log(truth + 1, exp(1)))^2). This is mostly used for count data, note that all predicted and actual target values must be greater or equal '-1' to compute the mean squared logarithmic error.

Usage

MSLE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = abs(rnorm(n))
response = abs(rnorm(n))
MSLE(truth, response)

Accuracy (multilabel)

Description

Averaged proportion of correctly predicted labels with respect to the total number of labels for each instance, following the definition by Charte and Charte: https: / /journal.r-project.org / archive / 2015 - 2 / charte-charte.pdf. Fractions where the denominator becomes 0 are replaced with 1 before computing the average across all instances.

Usage

MultilabelACC(truth, response)

Arguments

F1 measure (multilabel)

Description

Harmonic mean of precision and recall on a per instance basis (Micro-F1), following the definition by Montanes et al.: http: / /www.sciencedirect.com / science / article / pii / S0031320313004019. Fractions where the denominator becomes 0 are replaced with 1 before computing the average across all instances.

Usage

MultilabelF1(truth, response)

Arguments

Hamming loss

Description

Proportion of labels that are predicted incorrectly, following the definition by Charte and Charte: https://journal.r-project.org/archive/2015-2/charte-charte.pdf.

Usage

MultilabelHamloss(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = matrix(sample(c(0,1), 60, replace = TRUE), 20, 3)
response = matrix(sample(c(0,1), 60, replace = TRUE), 20, 3)
MultilabelHamloss(truth, response)

Positive predictive value (multilabel)

Description

Also called precision. Averaged ratio of correctly predicted labels for each instance, following the definition by Charte and Charte: https: / /journal.r-project.org / archive / 2015 - 2 / charte-charte.pdf. Fractions where the denominator becomes 0 are ignored in the average calculation.

Usage

MultilabelPPV(truth, response)

Arguments

Subset-0-1 loss

Description

Proportion of observations where the complete multilabel set (all 0-1-labels) is predicted incorrectly, following the definition by Charte and Charte: https://journal.r-project.org/archive/2015-2/charte-charte.pdf.

Usage

MultilabelSubset01(truth, response)

Arguments

Examples

n = 20
set.seed(122)
truth = matrix(sample(c(0,1), 60, replace = TRUE), 20, 3)
response = matrix(sample(c(0,1), 60, replace = TRUE), 20, 3)
MultilabelSubset01(truth, response)

TPR (multilabel)

Description

Also called recall. Averaged proportion of predicted labels which are relevant for each instance, following the definition by Charte and Charte: https: / /journal.r-project.org / archive / 2015 - 2 / charte-charte.pdf. Fractions where the denominator becomes 0 are ignored in the average calculation.

Usage

MultilabelTPR(truth, response)

Arguments

Negative predictive value

Description

Defined as: tn / (tn + fn).

Usage

NPV(truth, response, negative)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
negative = 0
NPV(truth, response, negative)

Positive predictive value

Description

Defined as: tp / (tp + fp). Also called precision. If the denominator is 0, PPV is set to be either 1 or 0 depending on whether the highest probability prediction is positive (1) or negative (0).

Usage

PPV(truth, response, positive, probabilities = NULL)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
PPV(truth, response, positive, probabilities = NULL)

Quadratic Scoring Rule

Description

Defined as: 1 - (1/n) sum_i sum_j (y_ij - p_ij)^2, where y_ij = 1 if observation i has class j (else 0), and p_ij is the predicted probablity of observation i for class j. This scoring rule is the same as 1 - multiclass.brier. See: Bickel, J. E. (2007). Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Analysis, 4(2), 49-65.

Usage

QSR(probabilities, truth)

Arguments

Relative absolute error

Description

Defined as sum_of_absolute_errors / mean_absolute_deviation. Undefined for single instances and when every truth value is identical. In this case the output will be NA.

Usage

RAE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
RAE(truth, response)

Root mean squared error

Description

The RMSE is aggregated as sqrt(mean(rmse.vals.on.test.sets^2))

Usage

RMSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
RMSE(truth, response)

Root mean squared logarithmic error

Description

Definition taken from: https: / /www.kaggle.com / wiki / RootMeanSquaredLogarithmicError. This is mostly used for count data, note that all predicted and actual target values must be greater or equal '-1' to compute the root mean squared logarithmic error.

Usage

RMSLE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = abs(rnorm(n))
response = abs(rnorm(n))
RMSLE(truth, response)

Root relative squared error

Description

Defined as sqrt (sum_of_squared_errors / total_sum_of_squares). Undefined for single instances and when every truth value is identical. In this case the output will be NA.

Usage

RRSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
RRSE(truth, response)

Coefficient of determination

Description

Also called R-squared, which is 1 - residual_sum_of_squares / total_sum_of_squares.

Usage

RSQ(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
RSQ(truth, response)

Sum of absolute errors

Description

Defined as: sum(abs(response - truth))"

Usage

SAE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
SAE(truth, response)

Sum of squared errors

Description

Defined as: sum((response - truth)^2)

Usage

SSE(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
SSE(truth, response)

Spherical Scoring Rule

Description

Defined as: mean(p_i(sum_j(p_ij))), where p_i is the predicted probability of the true class of observation i and p_ij is the predicted probablity of observation i for class j. See: Bickel, J. E. (2007). Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Analysis, 4(2), 49-65.

Usage

SSR(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
SSR(probabilities, truth)

Spearman's rho

Description

Defined as: Spearman's rho correlation between truth and response. Only looks at the order. See Rosset et al.: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.1398&rep=rep1&type=pdf.

Usage

SpearmanRho(truth, response)

Arguments

Examples

n = 20
set.seed(123)
truth = rnorm(n)
response = rnorm(n)
SpearmanRho(truth, response)

True negatives

Description

Sum of correctly classified observations in the negative class. Also called correct rejections.

Usage

TN(truth, response, negative)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
negative = 0
TN(truth, response, negative)

True negative rate

Description

Percentage of correctly classified observations in the negative class. Also called specificity.

Usage

TNR(truth, response, negative)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
negative = 0
TNR(truth, response, negative)

True positives

Description

Sum of all correctly classified observations in the positive class.

Usage

TP(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
TP(truth, response, positive)

True positive rate

Description

Percentage of correctly classified observations in the positive class. Also called hit rate or recall or sensitivity.

Usage

TPR(truth, response, positive)

Arguments

Examples

n = 20
set.seed(125)
truth = as.factor(sample(c(1,0), n, replace = TRUE))
probabilities = runif(n)
response = as.factor(as.numeric(probabilities > 0.5))
positive = 1
TPR(truth, response, positive)

Mean quadratic weighted kappa

Description

Defined as: 1 - sum(weights * conf.mat) / sum(weights * expected.mat), the weight matrix measures seriousness of disagreement with the squared euclidean metric.

Usage

WKAPPA(truth, response)

Arguments

List all measures

Description

Lists all measures that are available in the package with their corresponding task.

Usage

listAllMeasures()

Value

Dataframe with all available measures and the correspoding task

Examples

listAllMeasures()

Weighted average 1 vs. 1 multiclass AUC

Description

Computes AUC of c(c - 1) binary classifiers while considering the a priori distribution of the classes. See Ferri et al.: https://www.math.ucdavis.edu/~saito/data/roc/ferri-class-perf-metrics.pdf.

Usage

multiclass.AU1P(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.AU1P(probabilities, truth)

Average 1 vs. 1 multiclass AUC

Description

Computes AUC of c(c - 1) binary classifiers (all possible pairwise combinations) while considering uniform distribution of the classes. See Ferri et al.: https://www.math.ucdavis.edu/~saito/data/roc/ferri-class-perf-metrics.pdf.

Usage

multiclass.AU1U(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.AU1U(probabilities, truth)

Weighted average 1 vs. rest multiclass AUC

Description

Computes the AUC treating a c-dimensional classifier as c two-dimensional classifiers, taking into account the prior probability of each class. See Ferri et al.: https://www.math.ucdavis.edu/~saito/data/roc/ferri-class-perf-metrics.pdf.

Usage

multiclass.AUNP(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.AUNP(probabilities, truth)

Average 1 vs. rest multiclass AUC

Description

Computes the AUC treating a c-dimensional classifier as c two-dimensional classifiers, where classes are assumed to have uniform distribution, in order to have a measure which is independent of class distribution change. See Ferri et al.: https://www.math.ucdavis.edu/~saito/data/roc/ferri-class-perf-metrics.pdf.

Usage

multiclass.AUNU(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.AUNU(probabilities, truth)

Multiclass Brier score

Description

Defined as: (1/n) sum_i sum_j (y_ij - p_ij)^2, where y_ij = 1 if observation i has class j (else 0), and p_ij is the predicted probability of observation i for class j. From http://docs.lib.noaa.gov/rescue/mwr/078/mwr-078-01-0001.pdf.

Usage

multiclass.Brier(probabilities, truth)

Arguments

Examples

n = 20
set.seed(122)
truth = as.factor(sample(c(1,2,3), n, replace = TRUE))
probabilities = matrix(runif(60), 20, 3)
probabilities = probabilities/rowSums(probabilities)
colnames(probabilities) = c(1,2,3)
multiclass.Brier(probabilities, truth)