Title:	Extract Statistics from Articles and Recompute P-Values
Version:	1.5.0
Date:	2024-02-16
Description:	A "spellchecker" for statistics. It checks whether your p-values match their accompanying test statistic and degrees of freedom. statcheck searches for null-hypothesis significance test (NHST) in APA style (e.g., t(28) = 2.2, p < .05). It recalculates the p-value using the reported test statistic and degrees of freedom. If the reported and computed p-values don't match, statcheck will flag the result as an error. If the reported p-value is statistically significant and the recomputed one is not, or vice versa, the result will be flagged as a decision error. You can use statcheck directly on a string of text, but you can also scan a PDF or HTML file, or even a folder of PDF and/or HTML files. Statcheck needs an external program to convert PDF to text: Xpdf. Instructions on where and how to download this program, how to install statcheck, and more details on what statcheck can and cannot do can be found in the online manual: https://rpubs.com/michelenuijten/statcheckmanual. You can find a point-and-click web interface to scan PDF or HTML or DOCX articles on http://statcheck.io.
License:	GPL-3
Depends:	R (≥ 2.14.2)
Imports:	ggplot2, plyr, rlang, rmarkdown, stringi, tcltk
Suggests:	testthat
ByteCompile:	yes
Encoding:	UTF-8
LazyLoad:	yes
RoxygenNote:	7.3.1
URL:	https://github.com/MicheleNuijten/statcheck
BugReports:	https://github.com/MicheleNuijten/statcheck/issues
NeedsCompilation:	no
Packaged:	2024-02-16 19:18:37 UTC; miche
Author:	Michele B. Nuijten [aut, cre], Sacha Epskamp [aut], Willem Sleegers [ctb], Edoardo Costantini [ctb], Paul van der Laken [ctb], Sean Rife [ctb], John Sakaluk [ctb], Chris Hartgerink [ctb], Steve Haroz [ctb]
Maintainer:	Michele B. Nuijten <m.b.nuijten@uvt.nl>
Repository:	CRAN
Date/Publication:	2024-02-16 19:30:02 UTC

statcheck: Extract statistics from articles and recompute p-values

Description

The package statcheck can extract Null Hypothesis Significance Test (NHST) results from articles (or plain text) and recomputes p-values to check whether a reported NHST result is internally consistent or not.

Details

statcheck can be used for multiple purposes, including:

• Self-checks: you can use statcheck to make sure your manuscript doesn't contain copy-paste errors or other inconsistencies before you submit it to a journal.

• Peer review: editors and reviewers can use statcheck to check submitted manuscripts for statistical inconsistencies. They can ask authors for a correction or clarification before publishing a manuscript.

• Research: statcheck can be used to automatically extract statistical test results from articles that can then be analyzed. You can for instance investigate whether you can predict statistical inconsistencies (see e.g., Nuijten et al., 2017 <doi:10.1525/collabra.102>), or use it to analyze p-value distributions (see e.g., Hartgerink et al., 2016 <doi:10.7717/peerj.1935>).

Using statcheck on a string of text

The most basic usage of statcheck is to directly extract NHST results and check for inconsistencies in a string of text. See statcheck for details and an example of how to do this.

Using statcheck on an article

Another option is to run statcheck on an article (PDF or HTML). This is a useful option if you want to check for inconsistencies in a single article (e.g., as a final check before you submit it). Depending on whether you want to check an article in HTML or PDF, you can use checkHTML or checkPDF, respectively. Note: it is recommended to check articles in HTML, as converting PDF files to plain text sometimes results in some conversion errors.

Using statcheck on a folder of articles

Finally, it is possible to run statcheck on an entire folder of articles. This is often useful for meta-research. To do so, you can use checkPDFdir to check all PDF articles in a folder, checkHTMLdir to check all PDF articles in a folder, and checkdir to check both PDF and HTML articles in a folder.

Accuracy of the algorithm in detecting inconsistencies

It is important to note that statcheck is not perfect. Its performance in detecting NHST results depends on the type-setting and reporting style of an article and can vary widely. However, statcheck performs well in classifying the retrieved statistics in different consistency categories. We found that statcheck’s sensitivity (true positive rate) and specificity (true negative rate) were high: between 85.3 respectively, depending on the assumptions and settings. The overall accuracy of statcheck ranged from 96.2 can be found in Nuijten et al., 2017.

Manual

Details on what statcheck can and cannot do, and how to install the package and the necessary program Xpdf can be found in the online manual.

Web app

statcheck is also available as a free, online web app at http://statcheck.io.

Author(s)

Maintainer: Michele B. Nuijten m.b.nuijten@uvt.nl (ORCID)

Authors:

• Sacha Epskamp mail@sachaepskamp.com (ORCID)

Other contributors:

• Willem Sleegers (ORCID) [contributor]

• Edoardo Costantini [contributor]

• Paul van der Laken (ORCID) [contributor]

• Sean Rife (ORCID) [contributor]

• John Sakaluk (ORCID) [contributor]

• Chris Hartgerink (ORCID) [contributor]

• Steve Haroz (ORCID) [contributor]

References

Hartgerink, C. H. J., Van Aert, R. C. M., Nuijten, M. B., Wicherts, J. M., Van Assen, M. A. L. M. (2016). Distributions of p-values smaller than .05 in psychology: What is going on? PeerJ, 4, e1935. doi: 10.7717/peerj.1935

Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., Van Assen, M. A. L. M., & Wicherts, J. M. (2017). Journal data sharing policies and statistical reporting inconsistencies in psychology. Collabra: Psychology, 3(1), 1-22. doi: 10.1525/collabra.102.

Nuijten, M. B., Van Assen, M. A. L. M., Hartgerink, C. H. J., Epskamp, S., & Wicherts, J. M. (2017). The validity of the tool "statcheck" in discovering statistical reporting inconsistencies. Preprint retrieved from https://osf.io/preprints/psyarxiv/tcxaj/.

See Also

Useful links:

• https://github.com/MicheleNuijten/statcheck

• Report bugs at https://github.com/MicheleNuijten/statcheck/issues

Extract statistics from PDF/HTML articles and recalculate p-values

Description

These functions search for NHST results in PDF and/or HTML articles and send the extracted statistics to statcheck.

Usage

checkHTML(files, ...)

checkPDF(files, ...)

Arguments

Value

A statcheck data frame with the extracted statistics. See statcheck for details.

Extract statistics from folders with PDF/HTML articles and recalculate p-values

Description

These functions search for NHST results in all PDF and/or HTML articles in a certain folder and send the extracted statistics to statcheck.

Usage

checkHTMLdir(dir, subdir = TRUE, extension = TRUE, ...)

checkPDFdir(dir, subdir = TRUE, ...)

checkdir(dir, subdir = TRUE, ...)

Arguments

Value

A statcheck data frame with the extracted statistics. See statcheck for details.

Identify specific points in a statcheck plot.

Description

With this function you can simply point and click on the datapoints in the plot to see the corresponding statcheck details, such as the paper from which the data came and the exact statistical results.

Usage

## S3 method for class 'statcheck'
identify(x, alpha = 0.05, ...)

Arguments

Examples

## Not run: 

# First we need a statcheck object
# Here, we create one by running statcheck on some raw text

txt <- "This test is consistent t(28) = 0.2, p = .84, but this one is 
inconsistent: F(2, 28) = 4.2, p = .01. This final test is even a
gross/decision inconsistency: z = 1.23, p = .03"

result <- statcheck(txt)

# Now, we can run identify.statcheck(), or shorter, simply identify():
identify(result)

# Further instructions:
# click on one or multiple points of interest
# press Esc
# a dataframe with information on the selected points will appear


## End(Not run)

Plot method for statcheck

Description

Function for plotting of statcheck objects. Reported p values are plotted against recalculated p values, which allows the user to easily spot if articles contain miscalculations of statistical results.

Usage

## S3 method for class 'statcheck'
plot(x, alpha = 0.05, APAstyle = TRUE, group = NULL, ...)

Arguments

Details

If APAstyle = FALSE, inconsistencies between the reported and the recalculated p value are indicated with an orange dot. Recalculations of the p value that render a previously non significant result (p >= .5) as significant (p < .05), and vice versa, are considered decision errors, and are indicated with a red dot. Exactly reported p values (i.e. p = ..., as opposed to p < ... or p > ...) are indicated with a diamond.

Acknowledgements

Many thanks to John Sakaluk who adapted the plot code to create graphs in APA style.

See Also

statcheck

Examples

# First we need a statcheck object
# Here, we create one by running statcheck on some raw text

txt <- "This test is consistent t(28) = 0.2, p = .84, but this one is 
inconsistent: F(2, 28) = 4.2, p = .01. This final test is even a
gross/decision inconsistency: z = 1.23, p = .03"

result <- statcheck(txt)

# We can then plot the statcheck object 'result' by simply calling plot() on 
# "result". R will know what kind of plot to make, because "result" is of 
# class "statcheck"
plot(result)

Extract statistics and recompute p-values

Description

statcheck extracts Null Hypothesis Significance (NHST) results from strings and returns the extracted values, reported p-values and recomputed p-values.

Usage

statcheck(
  texts,
  stat = c("t", "F", "cor", "chisq", "Z", "Q"),
  OneTailedTests = FALSE,
  alpha = 0.05,
  pEqualAlphaSig = TRUE,
  pZeroError = TRUE,
  OneTailedTxt = FALSE,
  AllPValues = FALSE,
  messages = TRUE
)

Arguments

Details

statcheck roughly works in three steps.

1. Scan text for statistical results

statcheck uses regular expressions to recognizes statistical results from t-tests, F-tests, \chi2-tests, Z-tests, Q-tests, and correlations. statcheck can only recognize these results if the results are reported exactly according to the APA guidelines:

• t(df) = value, p = value

• F(df1, df2) = value, p = value

• r(df) = value, p = value

• \chi2 (df, N = value) = value, p = value (N is optional)

• Z = value, p = value

• Q(df) = value, p = value (statcheck can distinguish between Q, Qw / Q-within, and Qb / Q-between)

statcheck takes into account that test statistics and p values may be exactly (=) or inexactly (< or >) reported. Different spacing has also been taken into account.

2. Recompute p-value

statcheck uses the reported test statistic and degrees of freedom to recompute the p-value. By default, the recomputed p-value is two-sided

3. Compare reported and recomputed p-value

This comparison takes into account how the results were reported, e.g., p < .05 is treated differently than p = .05. Incongruent p values are marked as an error. If the reported result is significant and the recomputed result is not, or vice versa, the result is marked as a decision_error.

Correct rounding is taken into account. For instance, a reported t-value of 2.35 could correspond to an actual value of 2.345 to 2.354 with a range of p-values that can slightly deviate from the recomputed p-value. statcheck will not count cases like this as errors.

Note that when statcheck flags an error or decision_error, it implicitly assumes that the p-value is the inconsistent value, but it could just as well be the case that the test statistic or degrees of freedom contain a reporting error. statcheck merely detects wether a set of numbers is consistent with each other.

Value

A data frame containing for each extracted statistic:

source

Name of the file of which the statistic is extracted

test_type

Character indicating the statistic that is extracted

df1

First degree of freedom (if applicable)

df2

Second degree of freedom

test_comp

Reported comparison of the test statistic, when importing from pdf this will often not be converted properly

test_value

Reported value of the statistic

p_comp

Reported comparison, when importing from pdf this might not be converted properly

reported_p

The reported p-value, or NA if the reported value was n.s.

computed_p

The recomputed p-value

raw

Raw string of the statistical reference that is extracted

error

The computed p value is not congruent with the reported p-value

decision_error

The reported result is significant whereas the recomputed result is not, or vice versa.

one_tailed_in_txt

Logical. Does the text contain the string "sided", "tailed", and/or "directional"?

apa_factor

What proportion of all detected p-values was part of a fully APA reported result?

See Also

For more details, see the online manual.

Examples

txt <- "blablabla the effect was very significant (t(100)=1, p < 0.001)"
statcheck(txt)

Generate HTML report for statcheck output

Description

This function uses R Markdown to generate a nicely formatted HTML report of statcheck output.

Usage

statcheckReport(statcheckOutput, outputFileName, outputDir)

Arguments

Details

This function temporarily saves the inserted statcheck output as an .RData file in the "output" folder in the statcheck package directory. This file is then called by the .Rmd template that is saved in the folder "rmd", also in the statcheck package directory. After the HTML report is generated, the .RData file is removed again.

Value

An HTML report, saved in the directory specified in the argument "outputDir".

Examples

## Not run: 

# first generate statcheck output, for instance by using the statcheck() 
function

txt <- "blablabla the effect was very significant (t(100)=1, p < 0.001)"
stat <- statcheck(txt)

# next, use this output to generate a nice HTML report of the results
statcheckReport(stat, outputFileName="statcheckHTMLReport", 
                outputDir="C:/mydocuments/results")

# you can now find your HTML report in the folder 
# "C:/mydocuments/results" under the name "statcheckHTMLReport.html".


## End(Not run)

Summary method for statcheck

Description

Gives the summaries for a statcheck object.

Usage

## S3 method for class 'statcheck'
summary(object, ...)

Arguments

Value

A data frame containing for each source of statistics:

source

Name of the file/origin of which the statistics are extracted

nr_p_values

The number of extracted reported p values per article

nr_errors

The number of errors per article

nr_decision_errors

The number of decision errors per article

Examples

txt <- "blablabla the effect was very significant (t(100)=1, p < 0.001)"
stat <- statcheck(txt)
summary(stat)

Trimming method for statcheck output

Description

Returns a subset of columns of a statcheck object.

Usage

trim(object, ...)

Arguments

Value

A data frame containing for each source of statistics:

source

Name of the file/origin of which the statistics are extracted

raw

Raw string of the statistical reference that is extracted

computed_p

The recomputed p-value

error

The computed p value is not congruent with the reported p-value

decision_error

The reported result is significant whereas the recomputed result is not, or vice versa.

Examples

txt <- "blablabla the effect was very significant (t(100)=1, p < 0.001)"
stat <- statcheck(txt)
trim(stat)