Shekelle PG, Kahan JP, Leape LL, Bernstein SJ, Park RE, Kamberg CJ; International Society of Technology Assessment in Health Care. Meeting.
Annu Meet Int Soc Technol Assess Health Care Int Soc Technol Assess Health Care Meet. 1995; 11: Abstract No. 59.
RAND, Santa Monica, CA, USA.
OBJECTIVE: The RAND/UCLA Appropriateness Method is a structured group process that has been used in the USA and Europe for assessing the appropriateness of health care technologies. The objective of this study was to formally test the reliability of the Appropriateness Method. METHODS: The Appropriateness Method relies on a 9 member multidisciplinary panel of clinical experts to rate for appropriateness a comprehensive array of indications (or clinical scenarios) for the procedure in question using a structured group process which includes two rounds of appropriateness ratings, between which occurs a face-to-face meeting of panelists. The sensitivity of the ratings to the selection of particular panelists is considered by some to be the major threat to reliability. We are testing this by conducting 3 replications of the Appropriateness Method for each of the two procedures, with stratified random sampling from a "universe" of experts determining which expert serves on which panel. The two procedures are coronary revascularization (CABG/PTCA) and hysterectomy. Nominations for expert panelists were solicited in the usual manner from the relevant specialty societies, thereby creating the "universe" of experts. Stratified random sampling from this universe resulted in 3 panels of similar composition for each procedure. Final round appropriateness ratings from each panel will be compared using the 3-way kappa statistic and pairwise 2-way kappa statistics for each panel. We will compare the reliability of ratings for both the entire set of indications (2115 and 1718 for coronary revascularization and hysterectomy, respectively), and for judgments of the appropriateness of the care of real patients, from detailed patient data previously collected on a random sample of 1350 and 600 patients who underwent coronary revascularization or hysterectomy, respectively. RESULTS: We present here results of the first round ratings of coronary revascularization; final round results for both panels will not be available until December 1994. Sixty-three percent of theoretical coronary revascularization indications were given the same appropriateness rating (appropriate, uncertain, inappropriate) by all three panels. Of the indications that were rated differently by at least two panels, only 0.8% were rated extremely different (i.e. rated appropriate by one panel and inappropriate by another). The three way kappa statistic for first round coronary revascularization (all 2115 indications) was 0.62, with agreement beyond chance better for indications rated appropriate or inappropriate (0.73 and 0.65, respectively) than for indications rated uncertain. Pairwise 2-way kappa statistics between panels were 0.62, 0.60, and 0.63. CONCLUSIONS: First round ratings for coronary revascularization appropriateness ratings show good agreement beyond chance. Final round ratings will allow estimation of the reliability of the Appropriateness Method.
Publication Types:
Keywords:
- Angioplasty, Transluminal, Percutaneous Coronary
- Coronary Artery Bypass
- Europe
- Evaluation Studies
- Female
- Humans
- Patient Selection
- Research Design
- Sensitivity and Specificity
- Specialties, Medical
- methods
- statistics & numerical data
- surgery
- hsrmtgs
Other ID:
UI: 102215813
From Meeting Abstracts