An Investigation of the Standardised Patient Interview Rating Scale (SPIRS) for the Assessment of Speech Pathology Students in a Simulation Clinic
DOI:
https://doi.org/10.18552/ijpblhsc.v3i1.213Keywords:
assessment, clinical education, speech pathology, standardised patientsAbstract
Standardised patients (SPs) are increasingly utilised in health sciences education to assist students in the development of clinical competencies, including interviewing skills. This study investigated the development and validation of a rating scale for formative assessment of speech pathology students in an interview with an SP. Participants in this study were 76 undergraduate speech pathology students and 10 clinical educators who participated in a simulated clinic module. As part of the module, pairs of students interviewed an SP portraying a parent of a child with speech delay. The Standardised Patient Interview Rating Scale (SPIRS) was developed to assess students’ foundation clinical competencies of communication, interviewing and professional practice skills. Students’ interviews were videotaped, rated individually on the SPIRS by the clinical educator, and later re-rated by an expert rater. Data were analysed to determine the content validity, internal consistency, and inter-rater reliability of the tool. In addition, descriptive statistics were used to report student performance levels. Results indicated that the SPIRS had good content validity and internal consistency but that there may be some redundancy in individual items. An acceptable level of inter-rater reliability was achieved. Students generally scored highly, with non-verbal communication being the easiest and professional practice the most difficult skill to demonstrate. The SPIRS was found to be an appropriate tool for formative assessment of students in this simulated clinic module. Recommendations for improving its reliability were made. Further research is required to investigate use of the SPIRS as an assessment tool in other contexts utilising standardised patients.References
Arthur, D. (1999) ‘Assessing Nursing Students' Basic Communication and Interviewing Skills: the Development and Testing of a Rating Scale’. Journal of Advanced Nursing 29, 658-665
Barry, M., Bradshaw, C. and Noonan, M. (2013) ‘Improving the Content and Face Validity of OSCE Assessment Marking Criteria on an Undergraduate Midwifery Programme: A Quality Initiative’. Nursing Education in Practice 13, 477-480
Becker, K., Rose, L., Berg, J., Park, H. and Shatzer, J. (2006) ‘The Teaching Effectiveness of Standardized Patients’. Journal of Nursing Education 45, 103-111
Boulet, J., Van Zanten, M., De Champlain, A., Hawkins, R. and Peitzman, S. (2008) ‘Checklist Content on a Standardized Patient Assessment: an Ex Post Facto Review’. Advances in Health Sciences Education 13, 59-69
Boursicot, K. and Roberts, T. (2006) ‘Setting Standards in a Professional Higher Education Course: defining the Concept of the Minimally Competent Student in Performance-Based Assessment at the Level of Graduation from Medical School’. Higher Education Quarterly 60, 74-90
Brigden, D. and Dangerfield, P. (2008) ‘The Role of Simulation in Medical Education’. The Clinical Teacher 5, 167-170
Briggs, S. and Cheek, J. (1986) ‘The Role of Factor Analysis in the Development and Evaluation of Personality Scales’. Journal of Personality 54 (1), 106-148
Clauser, B., Harik, P. and Margolis, M. (2006) ‘A Multivariate Generalizability Analysis of Data from a Performance Assessment of Physicians' Clinical Skills’. Journal of Educational Measurement 43, 173-191
Clauser, B., Margolis, M., Holtman, M., Katsufrakis, P. and Hawkins, R. (2012) ‘Validity Considerations in the Assessment of Professionalism’. Advances in Health Sciences Education 17, 165-181
Cohen, D., Colliver, J., Marcy, M., Fried, E. and Swartz, M. (1996) ‘Psychometric Properties of a Standardized-Patient Checklist and Rating-Scale Form used to Assess Interpersonal and Communication Skills’. Academic Medicine 71, s87-s89
Cook, D. and Beckman, T. (2009) ‘Does Scale Length matter? A Comparison of Nine- Versus Five-Point Rating Scales for the Mini-CEX’. Advances in Health Sciences Education 14, 655-664
Cook, D., Beckman, T., Mandrekar, J. and Pankratz, V. (2010) ‘Internal Structure of Mini-CEX scores for Internal Medicine Residents: Factor Analysis and Generalizability’. Advances in Health Sciences Education 15, 633-645
Cox III, E. (1980) ‘The Optimal Number of Response Alternatives for a Scale: a Review’. Journal of Marketing Research 17, 407-422
Crossley, J., Davies, H., Humphris, G. and Jolly, B. (2002) ‘Generalisability: a Key to unlock Professional Assessment’. Medical Education 36, 972-978
Crossley, J. and Jolly, B. (2012) ‘Making Sense of Work-Based Assessment: Ask the Right Questions, in the Right Way, about the Right Things, of the Right People’. Medical Education 46, 28-37
Cunnington, J., Neville, A. and Norman, G. (1997) ‘The Risks of Thoroughness: Reliability and Validity of Global Ratings and Checklists in an OSCE’. Advances in Health Science Education 1, 227-233
De Champlain, A., Margolis, M., King, A. and Klass, D. (1997) ‘Standardized Patients' Accuracy in recording Examinees' Behaviors using Checklists’. Academic Medicine 72, s85-s87
DeVellis, R. (2003) Scale Development: Theory and Applications. Thousand Oaks, California: Sage
Downing, S. (2004) ‘Reliability: on the Reproducibility of Assessment Data’. Medical Education 38, 1006-1012
Ginsburg, S., Regehr, G. and Mylopoulos, M. (2009) ‘From Behaviours to Attributions: Further Concerns regarding the Evaluation of Professionalism’. Medical Education 43, 414-425
Hattie, J. and Timperley, H. (2007) ‘The Power of Feedback’. Review of Educational Research 77, 81-112
Henry, B. (2007) ‘Use of the Standardized Patient Model to Develop Nutrition Counseling Skills’. Journal of Nutrition Education and Behavior 39, 50-51
Hill, A., Davidson, B. and Theodoros, D. (2010) ‘A Review of Standardised Patients in Clinical Education: Implications for Speech-Language Pathology Programs’. International Journal of Speech-Language Pathology 12, 259-270
Hodges, B., Regehr, G., Hanson, M. and McNaughton, N. (1998) ‘Validation of an Objective Structured Examination in Psychiatry’. Academic Medicine 73, 910-912
Holmboe, E. and Hawkins, R. (1998) ‘Methods for Evaluating the Clinical Competence of Residents in Internal Medicine’. Annals of Internal Medicine 129, 42-48
Huber, P., Baroffio, A., Chamot, E., Herrmann, R., Nendaz, M. and Vu, N. (2005) ‘Effects of Item and Rater Characteristics on Checklist Recording: What should we look for?’ Medical Education 39, 852-858
Iramaneerat, C., Myford, C., Yudkowsky, R. and Lowenstein, T. (2009) ‘Evaluating the Effectiveness of Rating Instruments for a Communication Skills Assessment of Medical Residents’. Advances in Health Sciences Education 14, 575-594
Ladyshewsky, R., Baker, R., Jones, M. and Nelson, L. (2000) ‘Reliability and validity of an Extended Simulated Patient Case: a Tool for Evaluation and Research in Physiotherapy’. Physiotherapy Theory and Practice 16, 15-25
Landis, J. and Koch, G. (1977) ‘The Measurement of Observer Agreement for Categorical Data’. Biometrics 33, 159-174
Le Maistre, C. and Paré, A. (2004) ‘Learning in Two Communities: the Challenge for Universities and Workplaces’. Journal of Workplace Learning 16 (1/2), 44-52
Levine, A. and Swartz, M. (2008) ‘Standardized Patients: the "Other" Simulation’. Journal of Critical Care 23, 179-184
Loftus, S. and Mackey, S. (2008) ‘Interviewing Patients and Clients’. In Communicating in the Health Sciences. 2nd edn. ed. by Higgs,J., Ajjawi, R., McAllister,L. Trede, F. and Loftus, S.). South Melbourne, Victoria, Oxford University Press., 111-116
Margolis, M., Clauser, B., Cuddy, M., Ciccone, A., Mee, J., Harik, P. and Hawkins, R. (2006) ‘Use of the Mini-Clinical Evaluation Exercise to rate Examinee Performance on a Multiple-Station Clinical Skills Examination: a Validity Study’. Academic Medicine 81, s56-s60
Mazor, K., Zanetti, M., Alper, E., Hatem, D., Barrett, S., Meterko, V., Gammon, W. and Pugnaire, M. (2007) ‘Assessing Professionalism in the Context of an Objective Structured Clinical Examination: an In-Depth Study of the Rating Process’. Medical Education 41, 331-340
McAllister, L. and Lincoln, M. (2004) Clinical Education in Speech-Language Pathology. London: Whurr
McAllister, S., Lincoln, M., Ferguson, A. and McAllister, L. (2010) ‘Issues in developing Valid Assessments of Speech Pathology Students' Performance in the Workplace’. International Journal of Language and Communication Disorders 45, 1-14
McGraw, R. and O’Connor, H. (1999) ‘Standardized Patients in the Early Acquisition of Clinical Skills’. Medical Education 33, 572-578
Norcini, J., Blank, L., Duffy, F. and Fortna, G. (2003) ‘The Mini-CEX: A Method for assessing Clinical Skills’. Annals of Internal Medicine 138, 476-483
Norcini, J. and Burch, V. (2007) ‘Workplace-ABased Assessment as an Educational Tool: AMEE Guide No. 31’. Medical Teacher 29, 855-871
Panzarella, K. and Manyon, A. (2007) ‘A Model for Integrated Assessment of Clinical Competence’. Journal of Allied Health 36, 157-164
Petrusa, E., Blackwell, T., Rogers, L., Saydjari, C., Parcel, S. and Guckian, J. (1987) ‘An Objective Measure of Clinical Performance’. The American Journal of Medicine 83, 34-42
Scheffer, S., Muehlinghaus, I., Froehmel, A. and Ortwein, H. (2008) ‘Assessing Students' Communication Skills: Validation of a Global Rating’. Advances in Health Sciences Education 13, 583-592
Sheepway, L., Lincoln, M. and Togher, L. (2011) ‘An International Study of Clinical Education Practices in Speech-Language Pathology’. International Journal of Speech-Language Pathology 13, 174-185
Silverman, J., Archer, J., Gillard, S., Howells, R. and Benson, J. (2011) ‘Initial Evaluation of EPSCALE, a Rating Scale that assesses the Process of Explanation and Planning in the Medical Interview’. Patient Education and Counseling 82, 89-93
Stillman, P., Brown, D., Redfield, D. and Sabers, D. (1977) ‘Construct Validation of the Arizona Clinical Interview Rating Scale’. Educational and Psychological Measurement 37, 1031-1038
Stillman, P., Sabers, D. and Redfield, B. (1976) ‘The Use of Paraprofessionals to teach Interviewing Skills’. Pediatrics 57, 769-774
Streiner, D. and Norman, G. (2008) Health Measurement Scales: a Practical Guide to their Development and Use. 4th edn. Oxford: Oxford University Press
Tamblyn, R., Klass, D., Schnabl, G. and Kopelow, M. (1991) ‘Sources of Unreliability and Bias in Standardized-Patient Rating’. Teaching and Learning in Medicine 3, 74-85
Theodoros, D., Davidson, B., Hill, A. and MacBean, N. (2010) Integration of Simulated Learning Environments into Speech Pathology Clinical Education Curricula: A National Approach [on-line] available from https://www.hwa.gov.au/sites/uploads/sles-in-speech-pathology-curricula-201108.pdf [14 July 2014]
Van Der Vleuten, C., Norman, G. and De Graaff, E. (1991) ‘Pitfalls in the Pursuit of Objectivity: Issues of Reliability’. Medical Education 25, 110-118
Vargas, A., Boulet, J., Errichetti, A., Van Zanten, M., Lopez, M. and Reta, A. (2007) ‘Developing Performance-Based Medical School Assessment Programs in Resource-Limited Environments’. Medical Teacher 29, 192-198
Vu, N., Marcy, M., Colliver, J., Verhulst, S., Travis, T. and Barrows, H. (1992) ‘Standardized (Simulated) Patients' Accuracy in recording Clinical Performance Check-List Items’. Medical Education 26, 99-104
Welkowitz, J., Cohen, B. and Ewen, R. (2007) Introductory Statistics for the Behavioral Sciences. New York: John Wiley and Sons Inc.
Wood, T. J. (2014) ‘Exploring the Role of First Impressions in Rater-Based Assessments’. Advances in Health Sciences Education 19, 409-427
Zraick, R. (2012) ‘Review of the Use of Standardized Patients in Speech-Language Pathology Clinical Education’. International Journal of Therapy and Rehabilitation 19, 112-118
Zraick, R., Allen, R. and Johnson, S. (2003) ‘The Use of Standardized Patients to teach and test Interpersonal and Communication Skills with Students in Speech-Language Pathology’. Advances in Health Sciences Education 8, 237-248
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons License "Attribution-NonCommercial No Derivs 4.0 International" (CC BY-NC-ND 4.0) which permits others to use the publication as long as the authors are appropriately cited.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
- The Author grants to Coventry University an irrevocable, royalty-free, worldwide, non-exclusive licence to publish this article in this journal in addition to the licence granted at paragraph 1 of this copyright notice.
