KIN6005 Statistical Analysis Studio

Reliability & Agreement

Evaluate how consistently repeated measurements distinguish participants and how closely two devices or raters agree.

Which analysis should I use?

Intraclass correlation coefficient (ICC): Estimate relative reliability across two or more measurements of the same quantity. Choose agreement or consistency and single or average measurements.
Bland–Altman analysis: Inspect bias and limits of agreement between two measurements in the same units.
Measurement error and detectable change: Examine SEM and minimum detectable change. These describe measurement precision, not whether a change is meaningful.

Choose the method from your study design and research question. The question-mark buttons explain each setting and result.

How to use the selected analysis

Use one row per participant and separate numeric columns for devices, raters or trials. Select 2–6 columns for ICC; the last two cannot be deselected.
Choose agreement versus consistency and single versus average measurement reliability. Bland–Altman uses a separate A–B pair; its sample count may differ from ICC.
Original-scale meaningful-change thresholds are disabled on the log-ratio scale. Limits of agreement describe individuals; confidence intervals describe uncertainty in those limits.

Data source

Participants

Between-person SD

Measurement-error SD

Device B fixed bias

Device B proportional bias (%)

Upload data or download a template

Upload instructions and data privacy

Use wide format: one row per participant and a separate numeric column for each device, rater, or trial. Measurements must describe the same quantity in compatible units.

The template contains synthetic example rows. Replace them with your observations, then upload the CSV. Keep the headers or select your own column names below.

This page accepts up to 1,000 rows and 2 MB. Data stay in your browser; upload them separately on each tab.

Dataset: Simulated measurement study (40 rows).

How to format your CSV

Save your spreadsheet as CSV UTF-8 (comma delimited), not an Excel workbook. Put unique variable names in the first row, with no title or description above them. Spaces in variable names are allowed.

Put one observation on each row. Keep the same number of columns throughout the file.
Put numbers alone in measurement cells: use 42.5, not 42.5 cm. Put units in the header, such as Jump height (cm). Use a decimal point and omit thousands separators.
Use consistent group labels. Both text (Control, Training) and numeric codes (0, 1) work. Blank measurements are missing, not zero.
Select your outcome, grouping variable, and predictors after uploading. Suggested selections are not a substitute for checking your variables.
For paired tests, measurement agreement, and Friedman tests, use one row per participant with each occasion or device in its own column.
For repeated and mixed models, use one row per participant per occasion, with participant ID, time, and outcome columns. Use numeric time for mixed models.
For categorical analyses, use individual observations, not a summarized frequency table. Each row is one person or event.

Explore accepts up to 10,000 rows and 5 MB. Other CSV labs accept up to 1,000 rows and 2 MB. Use the example download on each page to see its data layout.

Variable types and graph colors (4 variables)

Review the suggested types. Numeric codes such as 0 and 1 can represent categories. A type describes the variable; it does not recode cells or choose a statistical test.

Use variable colors in graphs

Measurements color their plotted values. Category colors identify groups when that column is selected for grouping. Derived plots use the outcome’s color. Settings apply to this dataset on this page.

Presets repeat after seven colors. Use custom hex colors when needed, and check that groups remain distinguishable.

Paste or edit CSV

CSV data

ICC measurement columns

Device ADevice BDevice C

ICC model

Measurement unit

Agreement measurement A

Agreement measurement B

Agreement scale

Confidence level

Meaningful change threshold

APA reporting guide

Use the results as parts of a report. Explain the study design, variables, units, missing data, and assumptions. Include effect estimates and confidence intervals where available. Do not invent an interval that this tool does not estimate.

ICC. Name the one- or two-way model, agreement or consistency, single or average unit, number of measurements, participant count, ICC, and interval method and level. This tool uses a participant bootstrap.

Agreement. Report B − A bias and its CI, then limits of agreement and their CIs separately. State whether the scale is raw or log. Approximately 95% limits of agreement are not 95% confidence intervals for the mean.

Measurement error. Distinguish measurement SEM from the SE of a mean. Define the typical error, CV, and MDC assumptions. A detectable change is not necessarily an important change.

Report exact p values to three decimals, or p < .001. Omit leading zeros for probabilities and correlations. Use italics for Latin statistical symbols; keep Greek symbols upright. Precision should suit the measurement scale.

APA numbers and statistics guide · APA reporting standards

Reliability of single and averaged measurements

Source: Simulated measurement study. This analysis uses 40 of 40 rows; Excluded rows: 0 (unselected groups or missing/invalid selected values).

Inspect input rows

First 12 rows (before exclusions)

First 12 rows (before exclusions)
Device A	Device B	Device C
45.341	48.014	48.163
51.472	60.135	49.594
50.825	52.129	46.970
40.593	45.248	40.315
40.593	46.733	42.334
32.551	40.046	32.841
51.272	58.044	55.503
47.761	50.703	44.175
50.062	58.432	54.250
40.247	48.773	45.605
52.316	54.852	52.048
44.730	44.726	43.352

ICC(2,1)

.79

95% bootstrap ICC interval

.69 to .85

Measurement SEM

3.24

Model-based MDC95

8.99

ICC(2, 1) = .79, 95% participant-bootstrap CI [.69, .85]. The estimate uses 40 complete participants and 3 measurements. SEM = 3.24.

Variance components behind the selected ICC

Variance components behind the selected ICC
Between-person MS	Measurement MS	Residual MS	Within-person MS
124.54	250.19	4.38	10.52

The ICC is not clipped to zero. A negative method-variance component is set to zero for the SEM calculation only; the original ANOVA ICC estimate is retained.

Try this: Increase between-person SD while holding measurement-error SD fixed. Then add a fixed bias to Device B.

Explain what happens

More participant heterogeneity can raise ICC without improving measurement precision. A fixed device offset lowers absolute agreement more than consistency. Correlation can remain high even when devices are not interchangeable.

Assumptions and method

Independent participants and complete crossed measurements for two-way ICC. Generalizing beyond sampled raters requires a random-rater design; consistency for these fixed raters has a different scope. Single and average units must match the intended use. Bootstrap intervals resample participants, not measurement cells.

Read the method reference

Agreement, bias, and individual differences

Agreement uses 40 complete Device A–Device B pairs. ICC may use a different number of participants because it requires every selected measurement.

Horizontal: Average of A and B. Vertical: B − A.

Bland–Altman: B minus A versus pair average

Export APA PNG

The export uses your selected graph colors and Arial, with 10-point graph labels and an 11-point caption at 6.5 inches wide (300 dpi). Check variable units, title case, and the note before downloading. Custom captions persist until you reset them.

Figure number

Custom figure title

Custom note below the figure

Export in black and white (grayscale)

Export background

Transparent exports keep dark text and are intended for light backgrounds. Grayscale removes color from the entire figure, including the legend. Check group labels and line styles.

Include figure number, title, and note

Turn this off if you will add the caption in your manuscript. Match your manuscript’s figure numbering and caption font. A figure does not replace a complete statistical report.

Reference lines: bias 4.06, lower limit -1.14, upper limit 9.27, meaningful-change bounds ±5. The log plot uses log units; the table below reports back-transformed ratios.

Bias and approximate 95% limits of agreement

Bias and approximate 95% limits of agreement
Quantity	Estimate	95% CI lower	95% CI upper
Mean B − A	4.06	3.21	4.91
Lower agreement limit	-1.14	-2.61	0.32
Upper agreement limit	9.27	7.80	10.73

Pearson correlation

.92

Typical error

1.88

Approximate CV (%)

3.88

Difference-versus-mean slope

0.05

Pairwise MDC95 is 5.21 units, versus the chosen meaningful-change threshold of 5.00. The estimated detectable change is larger than the chosen important change. Systematic bias must be considered separately.

Try this: Increase Device B's fixed bias. Watch Pearson r, the mean difference, and the agreement limits.

Explain what happens

A fixed bias can leave correlation nearly unchanged while shifting all differences. Device interchangeability depends on whether individual discrepancies are acceptable, not whether r is large or a bias test is nonsignificant.

Assumptions and method

Independent participant pairs; approximately normal differences with constant spread for original-scale limits. Log limits assume roughly constant multiplicative error and normal log differences. LoA confidence intervals use a t-based large-sample approximation. Repeated trials per person require an extended agreement model. Detectable change is not evidence of clinical importance.