Analytically, this technique is a wonderful idea. But in practice, the technique can be difficult to execute judiciously. First, there is always the question of sample size. For attribute data, relatively large samples are required to be able to calculate percentages with relatively low confidence intervals. If an expert looks at 50 different error scenarios – twice – and the match rate is 96 percent (48 votes vs. 50), the 95 percent confidence interval ranges from 86.29% to 99.51 percent. It is a fairly large margin of error, especially in terms of the challenge of choosing the scenarios, checking them in depth, making sure the value of the master is assigned, and then convincing the examiner to do the job – twice. If the number of scenarios is increased to 100, the 95 per cent confidence interval for a 96 per cent match rate will be reduced to a range of 90.1 to 98.9 per cent (Figure 2). Repeatability and reproducibility are components of accuracy in an analysis of the attribute measurement system, and it is advisable to first determine if there is a precision problem. This means that before designing an attribute contract analysis and selecting the appropriate scenarios, an analyst should urgently consider monitoring the database to determine if past events have been properly coded. If the test is planned and designed effectively, it can reveal enough information about the causes of the accuracy problems to justify a decision not to use attribute analysis at all.
In cases where the trial does not provide sufficient information, the analysis of the attribute agreement allows for a more detailed review to inform the introduction of training changes and error correction in the measurement system. An attribute analysis was developed to simultaneously assess the effects of repeatability and reproducibility on accuracy. It allows the analyst to review the responses of several reviewers if they look at multiple scenarios multiple times. It establishes statistics that assess the ability of evaluators to agree with themselves (repeatability), with each other (reproducibility) and with a master or correct value (overall accuracy) known for each characteristic – over and over again.