Reliability

Toolkit for Analyzing Reliability of a Diagnostic Test or a Measurement

How to cite the contents of this page and the Repeatability Coefficient (RC) Calculator?
Park JE, Han K, Sung YS, Chung MS, Koo HJ, Yoon HM, Choi YJ, Lee SS, Kim KW, Shin Y, An S, Cho HM, Park SH. Selection and Reporting of Statistical Methods to Assess Reliability of a Diagnostic Test: Conformity to Recommended Methods in a Peer-Reviewed Journal. Korean J Radiol. 2017 Nov-Dec;18(6):888-897.
FULLTEXT
  • 1. What is reliability?

    It refers to how the results of a test or a measurement are consistent when obtained repeatedly.

    We use the term “reliability” in this page as an umbrella term to cover various concepts such as reproducibility, repeatability, and agreement except when we distinguish “reliability parameter” and “agreement parameter” which is explained below.

  • 2. Repeatability versus reproducibility?

    These terms are often used confusingly or incorrectly.

    Repeatability is when measurements are repeated under identical or near-identical conditions, using the same measurement procedure, same operators, same measuring system, same operating conditions, and same physical location over a short period.

    Reproducibility is rerunning a measurement in slightly different settings, for example, different locations, operators, scanners, and so on so forth.

  • 3. What statistical tests or parameters should be used?

    Dichotomous or nominal data Ordinal data Continuous data

    Kappa

    Proportion of agreement

    Weighted kappa

    Intraclass correlation coefficient (ICC)

    Reliability parameters:

    Intraclass correlation coefficient (ICC)

    Concordance correlation coefficient (CCC)

    Agreement parameters:

    Within-subject standard deviation (wSD)

    Repeatability coefficient (RC) and Reproducibility coefficient (RDC)

    Coefficient of variation (CV)

    Bland-Altman limits of agreement (LOA)

    Table note: ICC has three different models including one-way random, two-way random, and two-way mixed models, and can use either consistency or absolute agreement assumptions.
    As ICC value for the same set of data may change according to the model and the assumption used, it is desirable to describe the model and the assumption.

  • 4. Reliability parameters versus agreement parameters?

    Reliability parameters

    How well the subjects in the study set can be distinguished from each other, for example, ICC = between-subject variability / (between-subject variability + within-subject variability)?

    These give relative information. A high reliability is obtained when there is a small variability between repeated measurements as well as when there is a large variability between the subjects in the study set.

    Agreement parameters

    Exactly how close are the repeated measurements?

    These show the measurement variability/error in absolute terms.

    These are crucial parameters for a test or a measurement to be used to monitor changes of a particular disease/health state over time

    For more information, please refer to “J Clin Epidemiol 2006;59:1033-1039.”

  • 5. Repeatability coefficient (RC) calculator

    REMEMBER: RC is an agreement parameter that is essential for a quantitative biomarker to monitor changes of a particular disease/health state in a longitudinal follow up as it is the smallest change that is detectable.
    Use this easy and fast RC calculator for your analysis! RC can be calculated not only for two sets of repeated measurements but also for more than two sets of repeated measurements.

    Upload your data as an Excel file. [Download Sample File]

    Select the number of decimal places for the output.

How to cite the contents of this page and the Repeatability Coefficient (RC) Calculator?
Park JE, Han K, Sung YS, Chung MS, Koo HJ, Yoon HM, Choi YJ, Lee SS, Kim KW, Shin Y, An S, Cho HM, Park SH. Selection and Reporting of Statistical Methods to Assess Reliability of a Diagnostic Test: Conformity to Recommended Methods in a Peer-Reviewed Journal. Korean J Radiol. 2017 Nov-Dec;18(6):888-897.
FULLTEXT