sigr: Format Significance Summaries for Reports

Description

Succinctly format significance summaries of various models and tests (F-test, Chi-Sq-test, Fisher-test, T-test, and rank-significance). The main purpose is unified reporting and planning of experimental results, working around issue such as the difficulty of extracting model summary facts (such as with 'lm'/'glm'). This package also includes empirical tests, such as bootstrap estimates.

Details

To learn more about sigr, please start with the vignette: vignette('sigrFormatting','sigr')

Author(s)

Maintainer: John Mount jmount@win-vector.com

Authors:

• Nina Zumel nzumel@win-vector.com

Other contributors:

• Win-Vector LLC [copyright holder]

See Also

Useful links:

• https://github.com/WinVector/sigr/

• https://winvector.github.io/sigr/

• Report bugs at https://github.com/WinVector/sigr/issues

Compute the distribution of differences of replacement samples of two Binomial or Bernoulli experiments.

Description

Assuming max(nA, nB) %% min(nA, nB) == 0: compute the distribution of differences of weighted sums between max(1, nB/nA)*sum(a) and max(1, nA/nB)*sum(b) where a is a 0/1 vector of length nA with each item 1 with independent probability (kA+kB)/(nA+nB), and b is a 0/1 vector of length nB with each item 1 with independent probability (kA+kB)/(nA+nB). Then return the significance of a direct two-sided test that the absolute value of this difference is at least as large as the test_rate_difference (if supplied) or the empirically observed rate difference abs(nB*kA - nA*kB)/(nA*nB). The idea is: under this scaling differences in success rates between the two processes are easily observed as differences in counts returned by the scaled processes. The method can be used to get the exact probability of a given difference under the null hypothesis that both the A and B processes have the same success rate (kA+kB)/(nA+nB). When nA and nB don't divide evenly into to each other two calculations are run with the larger process is alternately padded and truncated to look like a larger or smaller experiment that meets the above conditions. This gives us a good range of significances.

Usage

Bernoulli_diff_stat(kA, nA, kB, nB, test_rate_difference, common_rate)

Arguments

Details

Note the intent is that we are measuring the results of an A/B test with max(nA, nB) %% min(nA, nB) == 0 (no padding needed), or max(nA,nB) >> min(nA,nB) (padding is small effect).

The idea of converting a rate problem into a counting problem follows from reading Wald's Sequential Analysis.

For very small p-values the calculation is sensitive to rounding in the observed ratio-difference, as an arbitrarily small change in test-rate can move an entire set of observed differences in or out of the significance calculation.

Value

Bernoulli difference test statistic.

Examples

Bernoulli_diff_stat(2000, 5000, 100, 200)
Bernoulli_diff_stat(2000, 5000, 100, 200, 0.1)
Bernoulli_diff_stat(2000, 5000, 100, 199)
Bernoulli_diff_stat(2000, 5000, 100, 199, 0.1)
Bernoulli_diff_stat(100, 200, 2000, 5000)

# sigr adjusts experiment sizes when lengths
# don't divide into each other.
Bernoulli_diff_stat(100, 199, 2000, 5000)
Bernoulli_diff_stat(100, 199, 2000, 5000)$pValue

Wrap TInterval (test of Binomial/Bernoulli rate).

Description

Wrap TInterval (test of Binomial/Bernoulli rate).

Usage

TInterval(x, ...)

Arguments

See Also

TIntervalS, TInterval.numeric, TInterval.data.frame

Student-T tolerance-style interval around an estimate of a mean from a data.frame.

Description

Student-T tolerance-style interval around an estimate of a mean from a data.frame.

Usage

## S3 method for class 'data.frame'
TInterval(x, ColumnName, ..., conf.level = 0.95, na.rm = FALSE)

Arguments

Value

wrapped stat

See Also

TInterval, TIntervalS, TInterval.numeric, TInterval.data.frame

Examples

set.seed(2018)
d <- data.frame(x = rnorm(100) + 3.2)
TInterval(d, "x")

Student-T tolerance-style interval around an estimate of a mean from observations.

Description

Student-T tolerance-style interval around an estimate of a mean from observations.

Usage

## S3 method for class 'numeric'
TInterval(x, ..., conf.level = 0.95, na.rm = FALSE)

Arguments

Value

wrapped stat

See Also

TInterval, TIntervalS, TInterval.numeric, TInterval.data.frame

Examples

set.seed(2018)
d <- rnorm(100) + 3.2
TInterval(d)

Student-T tolerance-style interval around an estimate of a mean from summary.

Description

Student-T tolerance-style interval around an estimate of a mean from summary.

Usage

TIntervalS(
  sample_size,
  sample_mean,
  sample_var,
  ...,
  nNA = 0,
  conf.level = 0.95
)

Arguments

Value

wrapped stat

See Also

TInterval, TIntervalS, TInterval.numeric, TInterval.data.frame

Examples

set.seed(2018)
d <- rnorm(100) + 3.2
TIntervalS(length(d), mean(d), stats::var(d))

Add ROC derived columns.

Description

Add ROC columns derived from sensitivity and specificity.

Usage

add_ROC_derived_columns(d, positive_prevalence)

Arguments

Value

extended data frame with more columns

Examples

d <- data.frame(pred = 1:4, truth = c(TRUE,FALSE,TRUE,TRUE))
roc <- build_ROC_curve(d$pred, d$truth)
add_ROC_derived_columns(roc, mean(d$truth))

as.character

Description

as.character

Usage

## S3 method for class 'sigr_statistic'
as.character(x, ...)

Arguments

Value

formatted string

Examples

as.character(wrapSignificance(1/300))

calculate ROC curve.

Description

Based on: https://blog.revolutionanalytics.com/2016/08/roc-curves-in-two-lines-of-code.html

Usage

build_ROC_curve(modelPredictions, yValues, ..., na.rm = FALSE, yTarget = TRUE)

Arguments

Value

the ROC graph of Score (model score), Sensitivity, and Specificity. Guaranteed to have the (0, 0) and (1, 1) (1-Specificity,Sensitivity) endpoints.

Examples

sigr::build_ROC_curve(1:4, c(TRUE,FALSE,TRUE,TRUE))

calculate AUC.

Description

Based on: https://blog.revolutionanalytics.com/2016/08/roc-curves-in-two-lines-of-code.html

Usage

calcAUC(modelPredictions, yValues, ..., na.rm = FALSE, yTarget = TRUE)

Arguments

Value

area under curve

Examples

sigr::calcAUC(1:4, c(TRUE,FALSE,TRUE,TRUE)) # should be 2/3

Calculate deviance.

Description

Calculate deviance.

Usage

calcDeviance(pred, y, na.rm = FALSE, eps = 1e-06)

Arguments

Value

deviance

Examples

sigr::calcDeviance(1:4,c(TRUE,FALSE,TRUE,TRUE))

Calculate sum of squared error.

Description

Calculate sum of squared error.

Usage

calcSSE(pred, y, na.rm = FALSE)

Arguments

Value

sum of squared error

Examples

sigr::calcSSE(1:4,c(1,0,1,1))

Check a few things we expect to be true for the utility frame.

Description

Utility to inspect a utility frame for some debugging.

Usage

check_utility_calc(
  values,
  ...,
  orig_score = NULL,
  orig_outcome = NULL,
  constant_utilities = FALSE
)

Arguments

Value

NULL if okay, else a string describing the problem.

Examples

d <- data.frame(
  predicted_probability = c(0, 0.5, 0.5, 0.5),
  made_purchase = c(FALSE, TRUE, FALSE, FALSE),
  false_positive_value = -5,    # acting on any predicted positive costs $5
  true_positive_value = 95,     # revenue on a true positive is $100 minus action cost
  true_negative_value = 0.001,  # true negatives have no value in our application
                                # but just give ourselves a small reward for being right
  false_negative_value = -0.01  # adding a small notional tax for false negatives,
                                # don't want our competitor getting these accounts.
  )

values <- model_utility(d, 'predicted_probability', 'made_purchase')
check_utility_calc(values,
                   orig_score = d$predicted_probability,
                   orig_outcome = d$made_purchase)

Studentized estimate of how often a difference is below zero.

Description

Studentized estimate of how often a difference is below zero.

Usage

estimateDifferenceZeroCrossing(resampledDiffs, na.rm = FALSE)

Arguments

Value

estimated probability of seeing a re-sampled difference below zero.

Examples

set.seed(2352)
resampledDiffs <- rnorm(10)+1
estimateDifferenceZeroCrossing(resampledDiffs)

Find area matching polynomial curve.

Description

Based on https://win-vector.com/2020/09/13/why-working-with-auc-is-more-powerful-than-one-might-think/

Usage

find_AUC_q(
  modelPredictions,
  yValues,
  ...,
  na.rm = FALSE,
  yTarget = TRUE,
  n_points = 101
)

Arguments

Value

q that such that curve 1 - (1 - (1-ideal_roc$Specificity)^q)^(1/q) matches area

Examples

d <- data.frame(pred = 1:4, truth = c(TRUE,FALSE,TRUE,TRUE))
q <- find_AUC_q(d$pred, d$truth)
roc <- build_ROC_curve(d$pred, d$truth)
ideal_roc <- data.frame(Specificity = seq(0, 1, length.out = 101))
ideal_roc$Sensitivity <- sensitivity_from_specificity_q(ideal_roc$Specificity, q)
# library(ggplot2)
# ggplot(mapping = aes(x = 1 - Specificity, y = Sensitivity)) +
#   geom_line(data = roc, color = "DarkBlue") +
#   geom_line(data  = ideal_roc, color = "Orange") +
#   theme(aspect.ratio=1) +
#   ggtitle("example actual and ideal curve")

Find area matching polynomial curve.

Description

Based on https://win-vector.com/2020/09/13/why-working-with-auc-is-more-powerful-than-one-might-think/

Usage

find_area_q(area, ..., n_points = 101)

Arguments

Value

q that such that curve 1 - (1 - (1-Specificity)^q)^(1/q) matches area

Examples

find_area_q(0.75)

Find beta-1 shape parameters matching the conditional distributions.

Description

Based on doi:10.1177/0272989X15582210. Fits a Beta(a, 1) distribuiton on positive examples and an Beta(1, b) distribution on negative examples.

Usage

find_matching_a1_1b(
  modelPredictions,
  yValues,
  ...,
  yTarget = TRUE,
  step_size = 0.001
)

find_ROC_matching_ab1(
  modelPredictions,
  yValues,
  ...,
  yTarget = TRUE,
  step_size = 0.001
)

Arguments

Value

beta curve shape parameters

Examples

d <- rbind(
  data.frame(x = rbeta(1000, shape1 = 6, shape2 = 4), y = TRUE),
  data.frame(x = rbeta(1000, shape1 = 2, shape2 = 5), y = FALSE)
)
find_ROC_matching_ab1(modelPredictions = d$x, yValues = d$y)
# should be near
# shape1_pos shape2_pos shape1_neg shape2_neg          a          b
#   3.985017   1.000000   1.000000   1.746613   3.985017   1.746613
#
# # How to land what you want as variables
# unpack[a, b] <-
#    find_matching_a1_1b(modelPredictions = d$x, yValues = d$y)

Find beta shape parameters matching the conditional distributions.

Description

Based on https://win-vector.com/2020/09/13/why-working-with-auc-is-more-powerful-than-one-might-think/. Used to find one beta distribution on positive examples, and another on negative examples.

Usage

find_matching_conditional_betas(modelPredictions, yValues, ..., yTarget = TRUE)

find_ROC_matching_ab(modelPredictions, yValues, ..., yTarget = TRUE)

Arguments

Value

beta curve shape parameters

Examples

d <- rbind(
  data.frame(x = rbeta(1000, shape1 = 6, shape2 = 4), y = TRUE),
  data.frame(x = rbeta(1000, shape1 = 2, shape2 = 3), y = FALSE)
)
find_matching_conditional_betas(modelPredictions = d$x, yValues = d$y)
# should be near
# shape1_pos shape2_pos shape1_neg shape2_neg
# 6          4          2          3
#
# # How to land all as variables
# unpack[shape1_pos, shape2_pos, shape1_neg, shape2_neg] <-
#    find_ROC_matching_ab(modelPredictions = d$x, yValues = d$y)

Fit beta parameters from data.

Description

Fit shape1, shape2 using the method of moments.

Usage

fit_beta_shapes(x)

Arguments

Value

beta shape1, shape2 parameters in a named list

Examples

x <- rbeta(1000, shape1 = 3, shape2 = 5.5)
fit_beta_shapes(x) # should often be near [3, 5.5]

Format

Description

Format

Usage

## S3 method for class 'sigr_statistic'
format(x, ...)

Arguments

Value

formatted string

Examples

format(wrapSignificance(1/300))

Detect rendering format (using knitr).

Description

Detect rendering format (using knitr).

Usage

getRenderingFormat()

Value

rendering format

Examples

getRenderingFormat()

Estimate model utility

Description

Compute the utility of a model score on a classification data set. For each threshold of interest we compute the utility of the classification rule of taking all items with model score greater than or equal to the threshold. The user specifies the outcome (a binary classification target), a model score (numeric), and the utility values (positive, negative, or zero) of each case: true positives, false positives, true negatives, and false negatives. What is returned is a table of model thresholds and the total value of using this model score plus the given threshold as a classification rule. NA is used to mark a threshold where no rows are selected.

Usage

model_utility(
  d,
  model_name,
  outcome_name,
  ...,
  outcome_target = TRUE,
  true_positive_value_column_name = "true_positive_value",
  false_positive_value_column_name = "false_positive_value",
  true_negative_value_column_name = "true_negative_value",
  false_negative_value_column_name = "false_negative_value"
)

Arguments

Details

A worked example can be found here: https://github.com/WinVector/sigr/blob/main/extras/UtilityExample.md.

Value

data.frame of all threshold values.

Examples

d <- data.frame(
  predicted_probability = c(0, 0.5, 0.5, 0.5),
  made_purchase = c(FALSE, TRUE, FALSE, FALSE),
  false_positive_value = -5,    # acting on any predicted positive costs $5
  true_positive_value = 95,     # revenue on a true positive is $100 minus action cost
  true_negative_value = 0.001,  # true negatives have no value in our application
                                # but just give ourselves a small reward for being right
  false_negative_value = -0.01  # adding a small notional tax for false negatives,
                                # don't want our competitor getting these accounts.
  )

values <- model_utility(d, 'predicted_probability', 'made_purchase')
best_strategy <- values[values$total_value >= max(values$total_value), ][1, ]
t(best_strategy)



# a bigger example

d <- data.frame(
  predicted_probability = stats::runif(100),
  made_purchase = sample(c(FALSE, TRUE), replace = TRUE, size = 100),
  false_positive_value = -5,    # acting on any predicted positive costs $5
  true_positive_value = 95,     # revenue on a true positive is $100 minus action cost
  true_negative_value = 0.001,  # true negatives have no value in our application
                                # but just give ourselves a small reward for being right
  false_negative_value = -0.01  # adding a small notional tax for false negatives,
  # don't want our competitor getting these accounts.
)

values <- model_utility(d, 'predicted_probability', 'made_purchase')

# plot the estimated total utility as a function of threshold
plot(values$threshold, values$total_value)

best_strategy <- values[values$total_value >= max(values$total_value), ][1, ]
t(best_strategy)


# without utilities example

d <- data.frame(
  predicted_probability = c(0, 0.5, 0.5, 0.5),
  made_purchase = c(FALSE, TRUE, FALSE, FALSE))
model_utility(d, 'predicted_probability', 'made_purchase')

Perform AUC permutation test.

Description

Estimate significance of AUC by permutation test.

Usage

permTestAUC(
  d,
  modelName,
  yName,
  yTarget = TRUE,
  ...,
  na.rm = FALSE,
  returnScores = FALSE,
  nrep = 100,
  parallelCluster = NULL
)

Arguments

Value

AUC statistic

Examples

set.seed(25325)
d <- data.frame(x1=c(1,2,3,4,5,6,7,7),
                y=c(FALSE,TRUE,FALSE,FALSE,
                    TRUE,TRUE,FALSE,TRUE))
permTestAUC(d,'x1','y',TRUE)

Empirical permutation test of significance of scoreFn(modelValues,yValues) >= scoreFn(modelValues,perm(yValues)).

Description

Treat permutation re-samples as similar to bootstrap replications.

Usage

permutationScoreModel(
  modelValues,
  yValues,
  scoreFn,
  ...,
  na.rm = FALSE,
  returnScores = FALSE,
  nRep = 100,
  parallelCluster = NULL
)

Arguments

Value

summaries

Examples

set.seed(25325)
y <- 1:5
m <- c(1,1,2,2,2)
cor.test(m,y,alternative='greater')
f <- function(modelValues,yValues) cor(modelValues,yValues)
permutationScoreModel(m,y,f)

Print

Description

Print

Usage

## S3 method for class 'sigr_statistic'
print(x, ...)

Arguments

Value

formatted string

Examples

print(wrapSignificance(1/300))

Objects exported from other packages

Description

These objects are imported from other packages. Follow the links below to see their documentation.

wrapr

%:=%, %.>%, %.>%, apply_left, apply_right, build_frame, build_frame, draw_frame, draw_frame, let, let, map_to_char, mk_tmp_name_source, qae, qc, qc, qchar_frame, qchar_frame, qe

Format summary roughly in "APA Style" ( American Psychological Association ).

Description

Format summary roughly in "APA Style" ( American Psychological Association ).

Usage

render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

See Also

render.sigr_significance, render.sigr_ftest

Format sigr_Bernoulli_diff_test (test of difference of Bernoulli processes).

Description

Format sigr_Bernoulli_diff_test (test of difference of Bernoulli processes).

Usage

## S3 method for class 'sigr_Bernoulli_diff_test'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Examples

Bernoulli_diff_stat(2000, 5000, 100, 200)
Bernoulli_diff_stat(2000, 5000, 100, 200, 0.1)
Bernoulli_diff_stat(2000, 5000, 100, 199)
Bernoulli_diff_stat(2000, 5000, 100, 199, 0.1)

Format an AUC-test (quality of a probability score)

Description

Format an AUC-test (quality of a probability score)

Usage

## S3 method for class 'sigr_aucpairtest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Format an AUC-test (quality of a probability score)

Description

Format an AUC-test (quality of a probability score)

Usage

## S3 method for class 'sigr_aucpermtest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Format an AUC-test (quality of a probability score)

Description

Format an AUC-test (quality of a probability score)

Usage

## S3 method for class 'sigr_aucresamptest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Format binom.test (test of rate of a Binomial/Bernoulli experiment).

Description

Format binom.test (test of rate of a Binomial/Bernoulli experiment).

Usage

## S3 method for class 'sigr_binomtest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Examples

bt <- binom.test(7, 10, 0.5)
wrapBinomTest(bt)

Format a chi-square test (quality of categorical prediction)

Description

Format a chi-square test (quality of categorical prediction)

Usage

## S3 method for class 'sigr_chisqtest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Format Cohen-D (effect size between groups)

Description

Format Cohen-D (effect size between groups)

Usage

## S3 method for class 'sigr_cohend'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 1,
  pSmallCutoff = 0
)

Arguments

Value

formatted string

Format cor.test (test of liner correlation).

Description

Format cor.test (test of liner correlation).

Usage

## S3 method for class 'sigr_cortest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
                y=c(1,1,2,2,3,3,4,4))
ct <- cor.test(d$x,d$y)
wrapCorTest(ct)

Format an empirical test (quality of categorical prediction)

Description

Format an empirical test (quality of categorical prediction)

Usage

## S3 method for class 'sigr_emptest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Format fisher.test (test of categorical independence).

Description

Format fisher.test (test of categorical independence).

Usage

## S3 method for class 'sigr_fishertest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string and fields

Examples

d <- data.frame(x=c('b','a','a','a','b','b','b'),
                y=c('1','1','1','2','2','2','2'))
ft <- fisher.test(table(d))
wrapFisherTest(ft)

Format an F-test

Description

Format an F-test

Usage

## S3 method for class 'sigr_ftest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Format an empirical test (quality of categorical prediction)

Description

Format an empirical test (quality of categorical prediction)

Usage

## S3 method for class 'sigr_permtest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Format a pwr-test

Description

Format a pwr-test

Usage

## S3 method for class 'sigr_pwr_htest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 1,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Format a significance

Description

Format a significance

Usage

## S3 method for class 'sigr_significance'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Examples

cat(render(wrapSignificance(1/300),format='html'))

Format a Student-T tolerance-style interval around an estimate of a mean.

Description

Report sample size (n), sample mean, bias-corrected standard deviation estimate (assuming normality, using a chi-square distribution correction from https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation#Bias_correction), and a Student t-test tolerance-style confidence interval.

Usage

## S3 method for class 'sigr_tinterval'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Examples

set.seed(2018)
d <- rnorm(100) + 3.2
TInterval(d)

Format a T-test (difference in means by group)

Description

Format a T-test (difference in means by group)

Usage

## S3 method for class 'sigr_ttest'
render(
  statistic,
  ...,
  format,
  statDigits = 4,
  sigDigits = 4,
  pLargeCutoff = 0.05,
  pSmallCutoff = 1e-05
)

Arguments

Value

formatted string

Studentized bootstrap variance estimate for scoreFn(yValues,modelValues).

Description

Studentized bootstrap variance estimate for scoreFn(yValues,modelValues).

Usage

resampleScoreModel(
  modelValues,
  yValues,
  scoreFn,
  ...,
  na.rm = FALSE,
  returnScores = FALSE,
  nRep = 100,
  parallelCluster = NULL
)

Arguments

Value

summaries

Examples

set.seed(25325)
y <- 1:5
m1 <- c(1,1,2,2,2)
cor.test(m1,y,alternative='greater')
f <- function(modelValues,yValues) {
 if((sd(modelValues)<=0)||(sd(yValues)<=0)) {
   return(0)
 }
 cor(modelValues,yValues)
}
s <- sigr::resampleScoreModel(m1,y,f)
print(s)
z <- (s$observedScore-0)/s$sd # should check size of z relative to bias!
pValue <- pt(z,df=length(y)-2,lower.tail=FALSE)
pValue

Studentized bootstrap test of strength of scoreFn(yValues,model1Values) > scoreFn(yValues,model1Values).

Description

Studentized bootstrap test of strength of scoreFn(yValues,model1Values) > scoreFn(yValues,model1Values) sampled with replacement.

Usage

resampleScoreModelPair(
  model1Values,
  model2Values,
  yValues,
  scoreFn,
  ...,
  na.rm = FALSE,
  returnScores = FALSE,
  nRep = 100,
  parallelCluster = NULL,
  sameSample = FALSE
)

Arguments

Details

True confidence intervals are harder to get right (see "An Introduction to the Bootstrap", Bradely Efron, and Robert J. Tibshirani, Chapman & Hall/CRC, 1993.), but we will settle for simple p-value estimates.

Value

summaries

Examples

set.seed(25325)
y <- 1:5
m1 <- c(1,1,2,2,2)
m2 <- c(1,1,1,1,2)
cor(m1,y)
cor(m2,y)
f <- function(modelValues,yValues) {
  if((sd(modelValues)<=0)||(sd(yValues)<=0)) {
    return(0)
  }
  cor(modelValues,yValues)
}
resampleScoreModelPair(m1,m2,y,f)

Wrap AUC resampling test results.

Description

Estimate significance of AUC by resampling test.

Usage

resampleTestAUC(
  d,
  modelName,
  yName,
  yTarget = TRUE,
  ...,
  na.rm = FALSE,
  returnScores = FALSE,
  nrep = 100,
  parallelCluster = NULL
)

Arguments

Value

AUC statistic

Examples

set.seed(25325)
d <- data.frame(x1=c(1,2,3,4,5,6,7,7),
                y=c(FALSE,TRUE,FALSE,FALSE,
                    TRUE,TRUE,FALSE,TRUE))
resampleTestAUC(d,'x1','y',TRUE)

Compute the shape1_pos, shape2_pos, shape1_neg, shape2_neg graph.

Description

Compute specificity and sensitivity given specificity and model fit parameters.

Usage

sensitivity_and_specificity_s12p12n(
  Score,
  ...,
  shape1_pos,
  shape2_pos,
  shape1_neg,
  shape2_neg
)

Arguments

Value

Score, Specificity and Sensitivity data frame

Examples

library(wrapr)

empirical_data <- rbind(
  data.frame(
    Score = rbeta(1000, shape1 = 3, shape2 = 2),
    y = TRUE),
  data.frame(
    Score = rbeta(1000, shape1 = 5, shape2 = 4),
    y = FALSE)
)

unpack[shape1_pos = shape1, shape2_pos = shape2] <-
  fit_beta_shapes(empirical_data$Score[empirical_data$y])

shape1_pos
shape2_pos

unpack[shape1_neg = shape1, shape2_neg = shape2] <-
  fit_beta_shapes(empirical_data$Score[!empirical_data$y])

shape1_neg
shape2_neg

ideal_roc <- sensitivity_and_specificity_s12p12n(
  seq(0, 1, 0.1),
  shape1_pos = shape1_pos,
  shape1_neg = shape1_neg,
  shape2_pos = shape2_pos,
  shape2_neg = shape2_neg)


empirical_roc <- build_ROC_curve(
  modelPredictions = empirical_data$Score,
  yValues = empirical_data$y
)

# # should look very similar
# library(ggplot2)
# ggplot(mapping = aes(x = 1 - Specificity, y = Sensitivity)) +
#   geom_line(data = empirical_roc, color='DarkBlue') +
#   geom_line(data = ideal_roc, color = 'Orange')

Compute the q-graph.

Description

Based on: https://blog.revolutionanalytics.com/2016/08/roc-curves-in-two-lines-of-code.html

Usage

sensitivity_from_specificity_q(Specificity, q)

Arguments

Value

Sensitivity

Examples

sensitivity_from_specificity_q(seq(0, 1, 0.1), 0.61)

Test AUC pair results.

Description

Estimate significance of difference in two AUCs by resampling.

Usage

testAUCpair(
  d,
  model1Name,
  model2Name,
  yName,
  yTarget = TRUE,
  ...,
  na.rm = FALSE,
  returnScores = FALSE,
  nrep = 100,
  parallelCluster = NULL
)

Arguments

Value

AUC pair test

Examples

set.seed(25325)
d <- data.frame(x1=c(1,2,3,4,5,6,7,7),
                x2=1,
                y=c(FALSE,TRUE,FALSE,FALSE,
                    TRUE,TRUE,FALSE,TRUE))
testAUCpair(d,'x1','x2','y',TRUE)

Wrap binom.test (test of Binomial/Bernoulli rate).

Description

Wrap binom.test (test of Binomial/Bernoulli rate).

Usage

wrapBinomTest(x, ...)

Arguments

See Also

wrapBinomTest.htest, wrapBinomTestS, wrapBinomTest.logical, wrapBinomTest.numeric, wrapBinomTest.data.frame

Wrap binom.test (test of Binomial/Bernoulli rate).

Description

Wrap binom.test (test of Binomial/Bernoulli rate).

Usage

## S3 method for class 'data.frame'
wrapBinomTest(
  x,
  ColumnName,
  SuccessValue = TRUE,
  ...,
  p = NA,
  alternative = c("two.sided", "less", "greater"),
  conf.level = 0.95,
  na.rm = FALSE
)

Arguments

Value

wrapped stat

See Also

wrapBinomTest, wrapBinomTest.htest, wrapBinomTestS, wrapBinomTest.logical, wrapBinomTest.numeric, wrapBinomTest.data.frame

Examples

d <- data.frame(x = c(rep(0, 3), rep(1, 7)))
wrapBinomTest(d, "x", 1, p = 0.5)
d <- data.frame(x = c(rep(0, 15), rep(1, 35)))
wrapBinomTest(d, "x", 1, p = 0.5)

Wrap binom.test (test of Binomial/Bernoulli rate).

Description

Wrap binom.test (test of Binomial/Bernoulli rate).

Usage

## S3 method for class 'htest'
wrapBinomTest(x, ...)

Arguments

Value

wrapped stat

See Also

wrapBinomTest, wrapBinomTest.htest, wrapBinomTestS, wrapBinomTest.logical, wrapBinomTest.numeric, wrapBinomTest.data.frame

Examples

bt <- binom.test(7, 10, 0.5)
wrapBinomTest(bt)

Wrap binom.test (test of Binomial/Bernoulli rate).

Description

Wrap binom.test (test of Binomial/Bernoulli rate).

Usage

## S3 method for class 'logical'
wrapBinomTest(
  x,
  ...,
  p = NA,
  alternative = c("two.sided", "less", "greater"),
  conf.level = 0.95,
  na.rm = FALSE
)

Arguments

Value

wrapped stat

See Also

wrapBinomTest, wrapBinomTest.htest, wrapBinomTestS, wrapBinomTest.logical, wrapBinomTest.numeric, wrapBinomTest.data.frame

Examples

x = c(rep(FALSE, 3), rep(TRUE, 7))
wrapBinomTest(x)
x = c(rep(FALSE, 15), rep(TRUE, 35))
wrapBinomTest(x)

Wrap binom.test (test of Binomial/Bernoulli rate).

Description

Wrap binom.test (test of Binomial/Bernoulli rate).

Usage

## S3 method for class 'numeric'
wrapBinomTest(
  x,
  SuccessValue = TRUE,
  ...,
  p = NA,
  alternative = c("two.sided", "less", "greater"),
  conf.level = 0.95,
  na.rm = FALSE
)

Arguments

Value

wrapped stat

See Also

wrapBinomTest, wrapBinomTest.htest, wrapBinomTestS, wrapBinomTest.logical, wrapBinomTest.numeric, wrapBinomTest.data.frame

Examples

x = c(rep(0, 3), rep(1, 7))
wrapBinomTest(x, 1)
x = c(rep(0, 15), rep(1, 35))
wrapBinomTest(x, 1)

Wrap binom.test (test of Binomial/Bernoulli rate) from summary.

Description

Wrap binom.test (test of Binomial/Bernoulli rate) from summary.

Usage

wrapBinomTestS(
  x,
  n,
  ...,
  p = NA,
  alternative = c("two.sided", "less", "greater"),
  conf.level = 0.95
)

Arguments

Value

wrapped stat

See Also

wrapBinomTest, wrapBinomTest.htest, wrapBinomTestS, wrapBinomTest.logical, wrapBinomTest.numeric, wrapBinomTest.data.frame

Examples

wrapBinomTestS(3, 7, p = 0.5)
wrapBinomTestS(300, 700, p = 0.5)

Wrap quality of a categorical prediction roughly in "APA Style" ( American Psychological Association ).

Description

Wrap quality of a categorical prediction roughly in "APA Style" ( American Psychological Association ).

Usage

wrapChiSqTest(x, ...)

Arguments

See Also

wrapChiSqTestImpl, wrapChiSqTest.glm, and wrapChiSqTest.data.frame

Format ChiSqTest from anova of logistic model.

Description

Format ChiSqTest from anova of logistic model.

Usage

## S3 method for class 'anova'
wrapChiSqTest(x, ...)

Arguments

Value

list of formatted string and fields

Examples

d <- data.frame(x1= c(1,2,3,4,5,6,7,7),
                x2= c(1,0,3,0,5,0,7,0),
                y= c(TRUE,FALSE,FALSE,FALSE,TRUE,TRUE,TRUE,FALSE))
model <- glm(y~x1+x2, data=d, family=binomial)
summary(model)
render(wrapChiSqTest(model),
       pLargeCutoff=1, format='ascii')
anov <- anova(model)
print(anov)
lapply(sigr::wrapChiSqTest(anov),
       function(ti) {
         sigr::render(ti,
                      pLargeCutoff= 1,
                      pSmallCutoff= 0,
                      statDigits=4,
                      sigDigits=4,
                      format='ascii')
       })

Format ChiSqTest from data.

Description

Format ChiSqTest from data.

Usage

## S3 method for class 'data.frame'
wrapChiSqTest(
  x,
  predictionColumnName,
  yColumnName,
  ...,
  yTarget = TRUE,
  nParameters = 1,
  meany = mean(x[[yColumnName]] == yTarget),
  na.rm = FALSE
)

Arguments

Value

wrapped test

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
      y=c(TRUE,FALSE,FALSE,FALSE,TRUE,TRUE,TRUE,FALSE))
model <- glm(y~x, data=d, family=binomial)
summary(model)
d$pred <- predict(model,type='response',newdata=d)
render(wrapChiSqTest(d,'pred','y'),pLargeCutoff=1)

Format ChiSqTest from model.

Description

Format ChiSqTest from model.

Usage

## S3 method for class 'glm'
wrapChiSqTest(x, ...)

Arguments

Value

wrapped test

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
      y=c(TRUE,FALSE,FALSE,FALSE,TRUE,TRUE,TRUE,FALSE))
model <- glm(y~x,data=d,family=binomial)
summary(model)
render(wrapChiSqTest(model),pLargeCutoff=1,format='ascii')

Format ChiSqTest from model summary.

Description

Format ChiSqTest from model summary.

Usage

## S3 method for class 'summary.glm'
wrapChiSqTest(x, ...)

Arguments

Value

wrapped test

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
      y=c(TRUE,FALSE,FALSE,FALSE,TRUE,TRUE,TRUE,FALSE))
model <- glm(y~x,data=d,family=binomial)
sum <- summary(model)
render(wrapChiSqTest(sum),pLargeCutoff=1,format='ascii')

Format quality of a logistic regression roughly in "APA Style" ( American Psychological Association ).

Description

Format quality of a logistic regression roughly in "APA Style" ( American Psychological Association ).

Usage

wrapChiSqTestImpl(df.null, df.residual, null.deviance, deviance)

Arguments

Value

wrapped statistic

Examples

wrapChiSqTestImpl(df.null=7,df.residual=6,
    null.deviance=11.09035,deviance=10.83726)

Wrap Cohen's D (effect size between groups).

Description

Wrap Cohen's D (effect size between groups).

Usage

wrapCohenD(x, ...)

Arguments

See Also

wrapCohenD.data.frame

Wrap Cohen's D (effect size between groups).

Description

Wrap Cohen's D (effect size between groups).

Usage

## S3 method for class 'data.frame'
wrapCohenD(x, Column1Name, Column2Name, ..., na.rm = FALSE)

Arguments

Value

formatted string and fields

Examples

d <- data.frame(x = c(1,1,2,2,3,3,4,4),
                y = c(1,2,3,4,5,6,7,7))
render(wrapCohenD(d,'x','y'))

Wrap Cohen's D (effect size between groups).

Description

Wrap Cohen's D (effect size between groups).

Usage

## S3 method for class 'numeric'
wrapCohenD(x, treatment, ..., na.rm = FALSE)

Arguments

Value

formatted string and fields

Examples

d <- data.frame(x = c(1,1,2,2,3,3,4,4),
                y = c(1,2,3,4,5,6,7,7))
render(wrapCohenD(d$x, d$y))

Wrap cor.test (test of liner correlation).

Description

Wrap cor.test (test of liner correlation).

Usage

wrapCorTest(x, ...)

Arguments

See Also

wrapCorTest.htest, and wrapCorTest.data.frame

Wrap cor.test (test of liner correlation).

Description

Wrap cor.test (test of liner correlation).

Usage

## S3 method for class 'data.frame'
wrapCorTest(
  x,
  Column1Name,
  Column2Name,
  ...,
  alternative = c("two.sided", "less", "greater"),
  method = c("pearson", "kendall", "spearman"),
  exact = NULL,
  conf.level = 0.95,
  continuity = FALSE,
  na.rm = FALSE
)

Arguments

Value

wrapped stat

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
                y=c(1,1,2,2,3,3,4,4))
wrapCorTest(d,'x','y')

Wrap cor.test (test of liner correlation).

Description

Wrap cor.test (test of liner correlation).

Usage

## S3 method for class 'htest'
wrapCorTest(x, ...)

Arguments

Value

wrapped stat

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
                y=c(1,1,2,2,3,3,4,4))
ct <- cor.test(d$x,d$y)
wrapCorTest(ct)

Wrap F-test (significance identity relation).

Description

Wrap F-test (significance identity relation).

Usage

wrapFTest(x, ...)

Arguments

See Also

wrapFTestImpl, wrapFTest.lm, and wrapFTest.data.frame

Wrap quality statistic of a linear relation from anova.

Description

Wrap quality statistic of a linear relation from anova.

Usage

## S3 method for class 'anova'
wrapFTest(x, ...)

Arguments

Value

list of formatted string and fields

Examples

d <- data.frame(x1 = c(1,2,3,4,5,6,7,7),
                x2 = c(1,0,3,0,5,6,0,7),
                y =  c(1,1,2,2,3,3,4,4))
model <- lm(y~x1+x2, data=d)
summary(model)
sigr::wrapFTest(model)
anov <- stats::anova(model)
print(anov)
lapply(sigr::wrapFTest(anov),
       function(ti) {
         sigr::render(ti,
                      pLargeCutoff= 1,
                      pSmallCutoff= 0,
                      statDigits=4,
                      sigDigits=4,
                      format='ascii')
       })

Wrap quality statistic of identity relation from data.

Description

Wrap quality statistic of identity relation from data.

Usage

## S3 method for class 'data.frame'
wrapFTest(
  x,
  predictionColumnName,
  yColumnName,
  nParameters = 1,
  meany = mean(x[[yColumnName]]),
  ...,
  na.rm = FALSE,
  format = NULL
)

Arguments

Value

formatted string and fields

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
                y=c(1,1,2,2,3,3,4,4))
model <- lm(y~x,data=d)
summary(model)
d$pred <- predict(model,newdata=d)
sigr::wrapFTest(d,'pred','y')

Wrap F-test (ratio of variances).

Description

Wrap F-test (ratio of variances).

Usage

## S3 method for class 'htest'
wrapFTest(x, ..., format = NULL)

Arguments

Value

formatted string

Examples

v <- var.test(c(1,2,3,4,5,6,7,7), c(1, 1, 2))
sigr::wrapFTest(v)

Wrap quality statistic of identity r regression.

Description

Wrap quality statistic of identity r regression.

Usage

## S3 method for class 'lm'
wrapFTest(x, ..., format = NULL)

Arguments

Value

formatted string

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
                y=c(1,1,2,2,3,3,4,4))
model <- lm(y~x,data=d)
summary(model)
sigr::wrapFTest(model)

Wrap quality statistic of linear regression summary.

Description

Wrap quality statistic of linear regression summary.

Usage

## S3 method for class 'summary.lm'
wrapFTest(x, ..., format = NULL)

Arguments

Value

formatted string

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
                y=c(1,1,2,2,3,3,4,4))
model <- lm(y~x,data=d)
sum <- summary(model)
sigr::wrapFTest(sum)

Wrap F-test (significance of identity relation).

Description

Wrap F-test (significance of identity relation).

Usage

wrapFTestImpl(numdf, dendf, FValue, ..., format = NULL)

Arguments

Value

wrapped statistic

Examples

wrapFTestImpl(numdf=2,dendf=55,FValue=5.56)

Wrap quality statistic of a linear relation from ezANOVA (package ez).

Description

Please see https://github.com/WinVector/sigr/issues/1#issuecomment-322311947 for an example.

Usage

wrapFTestezANOVA(x, ...)

Arguments

Value

list of formatted string and fields

Wrap fisher.test (test of categorical independence).

Description

Wrap fisher.test (test of categorical independence).

Usage

wrapFisherTest(x, ...)

Arguments

See Also

wrapFisherTest.htest, and wrapFisherTest.data.frame

Wrap fisher.test (test of categorical independence).

Description

Wrap fisher.test (test of categorical independence).

Usage

## S3 method for class 'data.frame'
wrapFisherTest(
  x,
  Column1Name,
  Column2Name,
  ...,
  na.rm = FALSE,
  workspace = 2e+05,
  hybrid = FALSE,
  control = list(),
  or = 1,
  alternative = "two.sided",
  conf.int = TRUE,
  conf.level = 0.95,
  simulate.p.value = FALSE,
  B = 2000
)

Arguments

Value

wrapped test.

Examples

d <- data.frame(x=c('b','a','a','a','b','b','b'),
                y=c('1','1','1','2','2','2','2'))
wrapFisherTest(d, 'x', 'y')

Wrap fisher.test (test of categorical independence).

Description

Wrap fisher.test (test of categorical independence).

Usage

## S3 method for class 'htest'
wrapFisherTest(x, ...)

Arguments

Value

wrapped test.

Examples

d <- data.frame(x=c('b','a','a','a','b','b','b'),
                y=c('1','1','1','2','2','2','2'))
ft <- fisher.test(table(d))
wrapFisherTest(ft)

Wrap fisher.test (test of categorical independence).

Description

Wrap fisher.test (test of categorical independence).

Usage

## S3 method for class 'table'
wrapFisherTest(
  x,
  ...,
  workspace = 2e+05,
  hybrid = FALSE,
  control = list(),
  or = 1,
  alternative = "two.sided",
  conf.int = TRUE,
  conf.level = 0.95,
  simulate.p.value = FALSE,
  B = 2000
)

Arguments

Value

wrapped test.

Examples

d <- data.frame(x=c('b','a','a','a','b','b','b'),
                y=c('1','1','1','2','2','2','2'))
t <- table(d)
wrapFisherTest(t)

Wrap pwr test (difference in means by group).

Description

Wrap pwr test (difference in means by group).

Usage

wrapPWR(x, ...)

Arguments

See Also

pwr.2p.test

Wrap pwr test.

Description

Wrap pwr test.

Usage

## S3 method for class 'power.htest'
wrapPWR(x, ...)

Arguments

Value

formatted string and fields

Examples

if(require("pwr", quietly = TRUE)) {
  # Example from pwr package
  #  Exercise 6.1 p. 198 from Cohen (1988)
  test <- pwr::pwr.2p.test(h=0.3,n=80,sig.level=0.05,alternative="greater")
  wrapPWR(test)
}

Wrap a significance

Description

Wrap a significance

Usage

wrapSignificance(significance, symbol = "p")

Arguments

Value

wrapped significance

Examples

wrapSignificance(1/300)

Wrap t.test (difference in means by group).

Description

Wrap t.test (difference in means by group).

Usage

wrapTTest(x, ...)

Arguments

See Also

wrapTTest.htest, and wrapTTest.data.frame

Wrap t.test (difference in means by group).

Description

Wrap t.test (difference in means by group).

Usage

## S3 method for class 'data.frame'
wrapTTest(
  x,
  Column1Name,
  Column2Name,
  ...,
  y = NULL,
  alternative = c("two.sided", "less", "greater"),
  mu = 0,
  paired = FALSE,
  var.equal = FALSE,
  conf.level = 0.95,
  na.rm = FALSE
)

Arguments

Value

formatted string and fields

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
                y=c(1,1,2,2,3,3,4,4))
render(wrapTTest(d,'x','y'),pLargeCutoff=1)
# confirm p not order depedent
render(wrapTTest(d,'y','x'),pLargeCutoff=1)

Wrap t.test (difference in means by group).

Description

Wrap t.test (difference in means by group).

Usage

## S3 method for class 'htest'
wrapTTest(x, ...)

Arguments

Value

formatted string and fields

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
                y=c(1,1,2,2,3,3,4,4))
tt <- t.test(d$x,d$y)
render(wrapTTest(tt),pLargeCutoff=1)
# confirm not rescaling, as a correlation test would
render(wrapTTest(t.test(d$x,2*d$y)),pLargeCutoff=1)

Wrap t.test (difference in means by group).

Description

Wrap t.test (difference in means by group).

Usage

## S3 method for class 'numeric'
wrapTTest(
  x,
  pop2,
  ...,
  y = NULL,
  alternative = c("two.sided", "less", "greater"),
  mu = 0,
  paired = FALSE,
  var.equal = FALSE,
  conf.level = 0.95,
  na.rm = FALSE
)

Arguments

Value

formatted string and fields

Examples

d <- data.frame(x=c(1,2,3,4,5,6,7,7),
                y=c(1,1,2,2,3,3,4,4))
render(wrapTTest(d$x, d$y), pLargeCutoff=1)
# confirm p not order depedent
render(wrapTTest(d$y, d$x),pLargeCutoff=1)