RehabTools
Home/Reliability & agreement
← All how-to guides
Data analysis & research workflow

Reliability & agreement

Compare paired measurements with ICC, Bland–Altman plots, measurement error and statistical tests. The calculator runs locally in your browser.

Includes a calculatorOriginal address: /stats.html
PAIRED MEASUREMENT ANALYSIS

Reliability & agreement calculator

Paste two matched columns: one row per participant or independent unit. Inspect absolute agreement, consistency, measurement error and paired differences.

ICC model reference: Koo & Li ↗Calculated locally with NumPy and SciPy.
Methods and assumptions

ICC uses a two-way ANOVA with two measurements: ICC(2,1) and ICC(2,2) assess absolute agreement; ICC(C,1) and ICC(C,2) assess consistency. The model must match your sampling and intended generalisation. Negative ICC estimates are retained.

Differences are B − A. Standard Bland–Altman limits are mean difference ± 1.96 × sample SD of differences; they are estimated limits, not confidence intervals. The regression plot shows fitted bias with residual limits, assuming constant residual variance. SEM = pooled sample SD × √(1 − ICC(2,1)); MDC95 = 1.96 × √2 × SEM. Percentages use the positive grand mean, and are suppressed when that mean is zero or negative.

Paired t and Wilcoxon tests assess differences. Pearson r assesses correlation. Shapiro–Wilk examines each measurement and paired differences; median-centred Levene examines variance equality, not sphericity. Paired Cohen’s dz and pooled-SD d use their explicitly labelled denominators. No clinical thresholds are assigned; this is a newly implemented research calculator.

Download calculation source ↓
Original guide and demonstrations

These instructions describe the earlier application. The calculator above provides the current workflow.

 Online calculator and plot generator for reliability and validity

The calculator below generates absolute (abs) and relative (rel) intraclass correlation coefficient (ICC) for model types 2,1 and 2,k, percentage based minimum detectable change (MDC) and standard error of the measurement (SEM) scores for two columns of data. It also generates a scatterplot with linear fit and confidence intervals, standard and linear regression based Bland and Altman plots, quartile-quartile plots, paired samples t-tests - both parametric and nonparametric (Wilcoxon), Cohen's d effect size, Levene's test of sphericity, Shapiro-Wilk tests and histograms with normal distribution shading for the datasets. 

None of your data or interactions are saved, and noone can see what you are doing other than people looking at your screen.

****Please cite this website (www.rehabtools.org/stats.html) if you use this analysis program

Exporting to Excel

Download the .xlsx file below as a template if you want to copy and save the results of this page. Simply open the template, then for each analysis copy all of the data from the "Results - Excel Ready" box and paste it into the first column. You can also screenshot the graphs, either by a simple print screen command or preferably using something like the snippet tool in Windows.

Download rehabtools_-_validity_and_reliability_headings.xlsx

rehabtools_-_validity_and_reliability_headings.xlsx
File Size: 9 kb
File Type: xlsx

Download File

Paste entire spreadsheet version

Paste entire spreadsheet versionOpen video ↗

Quick Text How to Guide:
Using Notepad++ (Windows only)

  1.  Select and copy all of the data you want to analyse in your spreadsheet program
  2.  Paste it into Notepad++ (this example is for that) or Sublime Text (works on Macs)
  3.  Go to Search, Replace (the shortcut is CTRL+H on Windows)
  4. YOU MUST select "Extended" under Search Mode
  5. Enter the following - Find what:\r\n     Replace with:]     
  6. Choose Replace All
  7. Copy all of the data - which is now in a single row - and paste it into the text box on the website
  8.  Choose the 2 rows of data to analyse based on the numbers in the grey boxes on the left of the table
  9. Press the green "Click to Process" button, review the results, and copy the data at the bottom and paste into your analysis spreadsheet. You can then either print the page as a pdf file or do a screen print for capturing the figures.

** Do not include any spaces in the \r\n or ] inputs

Using Sublime Editor (Windows or Mac)
Follow the exact same guidelines as above, but paste into Sublime Text Editor and search for \n

Paste entire spreadsheet version

The Reliability / Validity Calculator