High observed agreement does not necessarily imply a high kappa. When one outcome category dominates the sample, chance-corrected agreement measures can behave in surprisingly different ways.

The MRMC / Kappa Agreement Explorer generates a simulated multi-reader, multi-case study from parameters selected by the user, including prevalence, sensitivity, specificity, number of cases, readers and repeated readings. It then compares observed agreement, Fleiss' kappa and Gwet's AC1 and shows how these measures change as prevalence changes.

The tool also explains the generated dataset and interprets the results dynamically using the values from the current simulation.

Outputs include simulated MRMC data, observed inter- and intra-reader agreement, Fleiss' kappa, Gwet's AC1, diagnostic performance measures, a prevalence sensitivity analysis, and a downloadable PDF summary of the simulation.

Try changing prevalence while keeping sensitivity and specificity constant.
This often reveals why two studies with very similar reader performance can produce very different kappa values.