An Investigation of the Standardised Patient Interview Rating Scale (SPIRS) for the Assessment of Speech Pathology Students in a Simulation Clinic

  • Anne E Hill The University of Queensland
  • Bronwyn J Davidson The University of Melbourne
  • Deborah G Theodoros The University of Queensland
Keywords: assessment, clinical education, speech pathology, standardised patients

Abstract

Standardised patients (SPs) are increasingly utilised in health sciences education to assist students in the development of clinical competencies, including interviewing skills. This study investigated the development and validation of a rating scale for formative assessment of speech pathology students in an interview with an SP. Participants in this study were 76 undergraduate speech pathology students and 10 clinical educators who participated in a simulated clinic module. As part of the module, pairs of students interviewed an SP portraying a parent of a child with speech delay. The Standardised Patient Interview Rating Scale (SPIRS) was developed to assess students’ foundation clinical competencies of communication, interviewing and professional practice skills. Students’ interviews were videotaped, rated individually on the SPIRS by the clinical educator, and later re-rated by an expert rater. Data were analysed to determine the content validity, internal consistency, and inter-rater reliability of the tool. In addition, descriptive statistics were used to report student performance levels. Results indicated that the SPIRS had good content validity and internal consistency but that there may be some redundancy in individual items. An acceptable level of inter-rater reliability was achieved. Students generally scored highly, with non-verbal communication being the easiest and professional practice the most difficult skill to demonstrate. The SPIRS was found to be an appropriate tool for formative assessment of students in this simulated clinic module. Recommendations for improving its reliability were made. Further research is required to investigate use of the SPIRS as an assessment tool in other contexts utilising standardised patients.

Author Biography

Anne E Hill, The University of Queensland
Anne Hill is a Lecturer in Speech Pathology at The University of Queensland, Australia. She researches in the area of student learning, in particular models of clinical education for students.

References

Arthur, D. (1999) ‘Assessing Nursing Students' Basic Communication and Interviewing Skills: the Development and Testing of a Rating Scale’. Journal of Advanced Nursing 29, 658-665

Barry, M., Bradshaw, C. and Noonan, M. (2013) ‘Improving the Content and Face Validity of OSCE Assessment Marking Criteria on an Undergraduate Midwifery Programme: A Quality Initiative’. Nursing Education in Practice 13, 477-480

Becker, K., Rose, L., Berg, J., Park, H. and Shatzer, J. (2006) ‘The Teaching Effectiveness of Standardized Patients’. Journal of Nursing Education 45, 103-111

Boulet, J., Van Zanten, M., De Champlain, A., Hawkins, R. and Peitzman, S. (2008) ‘Checklist Content on a Standardized Patient Assessment: an Ex Post Facto Review’. Advances in Health Sciences Education 13, 59-69

Boursicot, K. and Roberts, T. (2006) ‘Setting Standards in a Professional Higher Education Course: defining the Concept of the Minimally Competent Student in Performance-Based Assessment at the Level of Graduation from Medical School’. Higher Education Quarterly 60, 74-90

Brigden, D. and Dangerfield, P. (2008) ‘The Role of Simulation in Medical Education’. The Clinical Teacher 5, 167-170

Briggs, S. and Cheek, J. (1986) ‘The Role of Factor Analysis in the Development and Evaluation of Personality Scales’. Journal of Personality 54 (1), 106-148

Clauser, B., Harik, P. and Margolis, M. (2006) ‘A Multivariate Generalizability Analysis of Data from a Performance Assessment of Physicians' Clinical Skills’. Journal of Educational Measurement 43, 173-191

Clauser, B., Margolis, M., Holtman, M., Katsufrakis, P. and Hawkins, R. (2012) ‘Validity Considerations in the Assessment of Professionalism’. Advances in Health Sciences Education 17, 165-181

Cohen, D., Colliver, J., Marcy, M., Fried, E. and Swartz, M. (1996) ‘Psychometric Properties of a Standardized-Patient Checklist and Rating-Scale Form used to Assess Interpersonal and Communication Skills’. Academic Medicine 71, s87-s89

Cook, D. and Beckman, T. (2009) ‘Does Scale Length matter? A Comparison of Nine- Versus Five-Point Rating Scales for the Mini-CEX’. Advances in Health Sciences Education 14, 655-664

Cook, D., Beckman, T., Mandrekar, J. and Pankratz, V. (2010) ‘Internal Structure of Mini-CEX scores for Internal Medicine Residents: Factor Analysis and Generalizability’. Advances in Health Sciences Education 15, 633-645

Cox III, E. (1980) ‘The Optimal Number of Response Alternatives for a Scale: a Review’. Journal of Marketing Research 17, 407-422

Crossley, J., Davies, H., Humphris, G. and Jolly, B. (2002) ‘Generalisability: a Key to unlock Professional Assessment’. Medical Education 36, 972-978

Crossley, J. and Jolly, B. (2012) ‘Making Sense of Work-Based Assessment: Ask the Right Questions, in the Right Way, about the Right Things, of the Right People’. Medical Education 46, 28-37

Cunnington, J., Neville, A. and Norman, G. (1997) ‘The Risks of Thoroughness: Reliability and Validity of Global Ratings and Checklists in an OSCE’. Advances in Health Science Education 1, 227-233

De Champlain, A., Margolis, M., King, A. and Klass, D. (1997) ‘Standardized Patients' Accuracy in recording Examinees' Behaviors using Checklists’. Academic Medicine 72, s85-s87

DeVellis, R. (2003) Scale Development: Theory and Applications. Thousand Oaks, California: Sage

Downing, S. (2004) ‘Reliability: on the Reproducibility of Assessment Data’. Medical Education 38, 1006-1012

Ginsburg, S., Regehr, G. and Mylopoulos, M. (2009) ‘From Behaviours to Attributions: Further Concerns regarding the Evaluation of Professionalism’. Medical Education 43, 414-425

Hattie, J. and Timperley, H. (2007) ‘The Power of Feedback’. Review of Educational Research 77, 81-112

Henry, B. (2007) ‘Use of the Standardized Patient Model to Develop Nutrition Counseling Skills’. Journal of Nutrition Education and Behavior 39, 50-51

Hill, A., Davidson, B. and Theodoros, D. (2010) ‘A Review of Standardised Patients in Clinical Education: Implications for Speech-Language Pathology Programs’. International Journal of Speech-Language Pathology 12, 259-270

Hodges, B., Regehr, G., Hanson, M. and McNaughton, N. (1998) ‘Validation of an Objective Structured Examination in Psychiatry’. Academic Medicine 73, 910-912

Holmboe, E. and Hawkins, R. (1998) ‘Methods for Evaluating the Clinical Competence of Residents in Internal Medicine’. Annals of Internal Medicine 129, 42-48

Huber, P., Baroffio, A., Chamot, E., Herrmann, R., Nendaz, M. and Vu, N. (2005) ‘Effects of Item and Rater Characteristics on Checklist Recording: What should we look for?’ Medical Education 39, 852-858

Iramaneerat, C., Myford, C., Yudkowsky, R. and Lowenstein, T. (2009) ‘Evaluating the Effectiveness of Rating Instruments for a Communication Skills Assessment of Medical Residents’. Advances in Health Sciences Education 14, 575-594

Ladyshewsky, R., Baker, R., Jones, M. and Nelson, L. (2000) ‘Reliability and validity of an Extended Simulated Patient Case: a Tool for Evaluation and Research in Physiotherapy’. Physiotherapy Theory and Practice 16, 15-25

Landis, J. and Koch, G. (1977) ‘The Measurement of Observer Agreement for Categorical Data’. Biometrics 33, 159-174

Le Maistre, C. and Paré, A. (2004) ‘Learning in Two Communities: the Challenge for Universities and Workplaces’. Journal of Workplace Learning 16 (1/2), 44-52

Levine, A. and Swartz, M. (2008) ‘Standardized Patients: the "Other" Simulation’. Journal of Critical Care 23, 179-184

Loftus, S. and Mackey, S. (2008) ‘Interviewing Patients and Clients’. In Communicating in the Health Sciences. 2nd edn. ed. by Higgs,J., Ajjawi, R., McAllister,L. Trede, F. and Loftus, S.). South Melbourne, Victoria, Oxford University Press., 111-116

Margolis, M., Clauser, B., Cuddy, M., Ciccone, A., Mee, J., Harik, P. and Hawkins, R. (2006) ‘Use of the Mini-Clinical Evaluation Exercise to rate Examinee Performance on a Multiple-Station Clinical Skills Examination: a Validity Study’. Academic Medicine 81, s56-s60

Mazor, K., Zanetti, M., Alper, E., Hatem, D., Barrett, S., Meterko, V., Gammon, W. and Pugnaire, M. (2007) ‘Assessing Professionalism in the Context of an Objective Structured Clinical Examination: an In-Depth Study of the Rating Process’. Medical Education 41, 331-340

McAllister, L. and Lincoln, M. (2004) Clinical Education in Speech-Language Pathology. London: Whurr

McAllister, S., Lincoln, M., Ferguson, A. and McAllister, L. (2010) ‘Issues in developing Valid Assessments of Speech Pathology Students' Performance in the Workplace’. International Journal of Language and Communication Disorders 45, 1-14

McGraw, R. and O’Connor, H. (1999) ‘Standardized Patients in the Early Acquisition of Clinical Skills’. Medical Education 33, 572-578

Norcini, J., Blank, L., Duffy, F. and Fortna, G. (2003) ‘The Mini-CEX: A Method for assessing Clinical Skills’. Annals of Internal Medicine 138, 476-483

Norcini, J. and Burch, V. (2007) ‘Workplace-ABased Assessment as an Educational Tool: AMEE Guide No. 31’. Medical Teacher 29, 855-871

Panzarella, K. and Manyon, A. (2007) ‘A Model for Integrated Assessment of Clinical Competence’. Journal of Allied Health 36, 157-164

Petrusa, E., Blackwell, T., Rogers, L., Saydjari, C., Parcel, S. and Guckian, J. (1987) ‘An Objective Measure of Clinical Performance’. The American Journal of Medicine 83, 34-42

Scheffer, S., Muehlinghaus, I., Froehmel, A. and Ortwein, H. (2008) ‘Assessing Students' Communication Skills: Validation of a Global Rating’. Advances in Health Sciences Education 13, 583-592

Sheepway, L., Lincoln, M. and Togher, L. (2011) ‘An International Study of Clinical Education Practices in Speech-Language Pathology’. International Journal of Speech-Language Pathology 13, 174-185

Silverman, J., Archer, J., Gillard, S., Howells, R. and Benson, J. (2011) ‘Initial Evaluation of EPSCALE, a Rating Scale that assesses the Process of Explanation and Planning in the Medical Interview’. Patient Education and Counseling 82, 89-93

Stillman, P., Brown, D., Redfield, D. and Sabers, D. (1977) ‘Construct Validation of the Arizona Clinical Interview Rating Scale’. Educational and Psychological Measurement 37, 1031-1038

Stillman, P., Sabers, D. and Redfield, B. (1976) ‘The Use of Paraprofessionals to teach Interviewing Skills’. Pediatrics 57, 769-774

Streiner, D. and Norman, G. (2008) Health Measurement Scales: a Practical Guide to their Development and Use. 4th edn. Oxford: Oxford University Press

Tamblyn, R., Klass, D., Schnabl, G. and Kopelow, M. (1991) ‘Sources of Unreliability and Bias in Standardized-Patient Rating’. Teaching and Learning in Medicine 3, 74-85

Theodoros, D., Davidson, B., Hill, A. and MacBean, N. (2010) Integration of Simulated Learning Environments into Speech Pathology Clinical Education Curricula: A National Approach [on-line] available from https://www.hwa.gov.au/sites/uploads/sles-in-speech-pathology-curricula-201108.pdf [14 July 2014]

Van Der Vleuten, C., Norman, G. and De Graaff, E. (1991) ‘Pitfalls in the Pursuit of Objectivity: Issues of Reliability’. Medical Education 25, 110-118

Vargas, A., Boulet, J., Errichetti, A., Van Zanten, M., Lopez, M. and Reta, A. (2007) ‘Developing Performance-Based Medical School Assessment Programs in Resource-Limited Environments’. Medical Teacher 29, 192-198

Vu, N., Marcy, M., Colliver, J., Verhulst, S., Travis, T. and Barrows, H. (1992) ‘Standardized (Simulated) Patients' Accuracy in recording Clinical Performance Check-List Items’. Medical Education 26, 99-104

Welkowitz, J., Cohen, B. and Ewen, R. (2007) Introductory Statistics for the Behavioral Sciences. New York: John Wiley and Sons Inc.

Wood, T. J. (2014) ‘Exploring the Role of First Impressions in Rater-Based Assessments’. Advances in Health Sciences Education 19, 409-427

Zraick, R. (2012) ‘Review of the Use of Standardized Patients in Speech-Language Pathology Clinical Education’. International Journal of Therapy and Rehabilitation 19, 112-118

Zraick, R., Allen, R. and Johnson, S. (2003) ‘The Use of Standardized Patients to teach and test Interpersonal and Communication Skills with Students in Speech-Language Pathology’. Advances in Health Sciences Education 8, 237-248