Simulated patients and objective structured clinical examinations: review of their use in medical education

Jeremy Wallace is currently a specialist registrar in general adult and old age psychiatry and assistant course organiser of the Guy’s, King’s and St Thomas’ (GKT) Royal College of Psychiatrists’ membership (MRCPsych) course. His interests include transcultural psychiatry and evaluating medical education. Ranga Rao is a consultant psychiatrist with the South London and Maudsley NHS Trust (Ladywell University Hospital, Lewisham, London SE13 6LH, UK) and senior lecturer at the GKT Medical School, University of London. He is the organiser of the GKT MRCPsych course based at Guy’s Hospital and his research interests include substance misuse, neuroimaging and evaluating medical education. Richard Haslam is a specialist registrar in general adult psychiatry based at the York Clinic, Guy’s Hospital, and assistant organiser of the GKT MRCPsych course. His interests include evaluating medical education and communication technology in medicine.

Simulated or standardised patients have been used in medical education and other medical settings for some 30 years (Box 1). Their use encompasses undergraduate and postgraduate learning, the monitoring of doctors’ performance and standardisation of clinical examinations. Simulation has been used for instruction in industry and the military for much longer (Jason et al, 1971) but the first known effective use of simulated patients was by Barrows & Abrahamson (1964), who used them to appraise students’ performance in clinical neurology examinations.

Box 1 Uses of simulated patients in educational settings Teaching communication skills Teaching clinical skills Monitoring the performance of doctors Clinical examinations

 

The objective structured clinical examination (OSCE) was first described by Harden & Gleeson as, ‘a timed examination in which medical students interact with a series of simulated patients in stations that may involve history-taking, physical examination, counselling or patient management’ (Harden & Gleeson, 1979). Because OSCEs have been shown to be feasible and have good reliability and validity (Hodges et al, 1998) their use has become widespread as the standard for performance-based assessment, particularly in undergraduate examinations.

The use of OSCEs in undergraduate examinations (‘summative’ use) occurs in every London medical college and was pioneered by the Royal London and St Bartholomew’s Hospitals. Many colleges across the UK have now adapted their examinations to include these components. In addition, there is considerable uptake in the use of simulated patients for medical student training (‘formative’ use).

Several of the medical Royal Colleges have introduced an OSCE component into their postgraduate membership examinations. For example, the Royal College of Anaesthetists includes an OSCE in Part I of the fellowship examination, and the Royal College of Obstetricians and Gynaecologists has an OSCE in Part II of their examinations. The Royal College of Surgeons, London, are introducing OSCEs and a pilot is being planned for this year. The Royal College of Physicians has a Practical Assessment of Clinical Examination Skills (PACES) in their clinical examinations, part of which comprises a communication and ethics station, in which simulated patients are used. Although the preferred method of examination is with videotape of real consultations, the Royal College of General Practitioners has a ‘simulated surgery’ that about 5% of candidates use.

The Royal College of Psychiatrists has recently proposed changes to the existing membership examinations with a view to increasing their reliability and validity (http://www.rcpsych.ac.uk/traindev/exams/exam_recent.htm). The main changes are to the Part I examination, and from spring 2003 the existing individual patient assessment will be replaced by an OSCE, in which simulated patients are likely to be used. The Part II examination will essentially retain the same format.

In view of the proposed changes we feel that a review of the literature surrounding the use of simulated patients and OSCEs would be both useful and timely. In this paper we focus on the following issues. First, the definition of simulated patients; second, the feasibility of standardising simulators; third, a historical overview of simulated patients in medical settings; and fourth, the reliability and validity of OSCEs. We also comment on the possible burden to the people who are playing the part of patients.

Who are simulated patients?

TOP Who are simulated patients? Uses of simulated patients... Standardisation and training of... Objective structured clinical... Is there a burden... Conclusion Multiple choice questions References

In some cases simulated patients are professionally trained actors playing the part of patients (Norman et al, 1982). However, Sanson-Fisher & Poole (1980), when comparing medical students’ performance with that of real and simulated patients, used volunteer simulators who were not ‘members of the acting profession’. In a description of new medical student teaching at Michigan State University, Jason et al(1971) write that their simulated patients were ‘primarily drama students from our campus’ and additionally ‘several housewives, some of whom had no previous acting experience’. Rubin & Philp (1998), in a study of the health perceptions of simulators, state that simulated patients were recruited from ‘the allied health programmes at local colleges, community volunteer programmes, the community senior citizen programme and from clinics at a university-based department of family medicine’. Others, for example Hodges et al(1996), do not clearly state where they recruited their simulated patients. Therefore, simulated patients are not a homogeneous group and their only common characteristic is that of simulating real patients. This raises the question of whether such a diverse group can be trained to behave in a ‘standardised’ way.

Uses of simulated patients in clinical and educational settings

TOP Who are simulated patients? Uses of simulated patients... Standardisation and training of... Objective structured clinical... Is there a burden... Conclusion Multiple choice questions References

In a well-controlled study, Sanson-Fisher & Poole (1980) demonstrated the validity of using simulated patients in the assessment of medical students’ interpersonal skills. The simulators were psychiatry out-patients and the dependent measure was a rating of empathy based on a retrospective review of audio recordings. They demonstrated that student performance, on a rating of empathy, was not significantly different with genuine as opposed to simulated patients. In addition, they reported that students were unable to discriminate between persons simulating a patient role and those who presented a real history.

Teaching clinical skills
Norman et al(1982), in a study within a postgraduate setting, showed that simulated patients could be used in areas that went beyond communication skills alone. Using a sample of 10 residents within hospital and family medicine they compared residents’ performance on four real patients with chronic stable conditions and on four simulators coached to present the same problem. They found that there were no significant differences in the number of questions asked in the history, physical examination findings, diagnosis considered or investigations proposed by the residents. Interestingly, the residents elicited more historical information from the simulated patients, but this was found to be due to a single case in which the real patient, a woman with multiple sclerosis, had memory loss. Residents also correctly identified 67% of the patients as real or simulated (against a chance 50%). Arguably, a problem with the studies by both Norman et al(1982) Sanson-Fisher & Poole (1980) is whether they can be generalised beyond the settings and patient problems involved.

Monitoring doctors’ performance
Another use of simulated patients is in the monitoring of doctors’ performance. For example, Rethans & Boven (1987) in The Netherlands introduced simulated patients into 48 general practitioners’ (GPs’) surgeries and collected data about their performance. The simulators described symptoms of a urinary tract infection and the researchers were interested in whether the GPs acted according to a consensus standard for managing a patient presenting with this condition. The GPs did not detect the simulators and GP performance was on the whole shown to be a more accurate reflection of actual practice than is elicted by written simulations.

Rethans et al(1991) repeated this study for other clinical problems such as headaches, diarrhoea, diabetes and shoulder pain and found similar results, which led them to conclude that ‘standardised patients may be the method of choice in the assessment of quality of actual care of doctors’. They state that existing methods for performance assessment, such as written tests and clinical examinations, have doubtful reliability and validity. Although audio and video recordings of consultations can be reliable and valid, they feel that the limited control over patient characteristics makes it difficult to compare performance between doctors.

A study by McLure et al(1985) introduced trained patients with uncomplicated rheumatic disease into the consulting rooms of 26 family physicians in Arizona. The physicians’ ability to collect diagnostic information and formulate a plan was investigated. They found that despite the fact that most of the physicians neglected areas such as psychosocial impact and mental health issues, the majority made an adequate assessment and virtually all developed an adequate care plan. They felt that the simulated patient method provided an unobtrusive way of auditing physicians’ encounters and that the technique can be used in peer review and the planning of training programmes. It is interesting to note, however, that additional postgraduate education does not seem to change the practice of doctors (Sibley et al, 1982).

Simulated patients and clinical examinations
The traditional oral examination has been criticised for its inherent unreliability and poor validity (Hodges et al, 1997). Data gathered by the National Board of Medical Examinations in the USA (1960–1963), involving over 10 000 medical students, showed that the correlation of independent evaluations by two examiners was less than 0.25 (Hubbard et al, 1963). It has also been demonstrated that the luck of the draw in selection of examiner and patient played a significant role in the outcome of postgraduate oral examinations in psychiatry (Leichner et al, 1984). Indeed, Leichner et al went on to suggest that the use of OSCEs may be a feasible means of improving the validity and reliability of oral examinations (Leichner et al, 1986).

An advantage of simulated patients over real patients is that of allowing different students to be presented with a similar challenge, thereby reducing an important source of variability (Norman et al, 1985). Other advantages include their reliable availability and adaptability, which enables the reproduction of a wide range of clinical phenomena tailored to the student’s level of skill. In addition, they can simulate scenarios that may be distressing for a real patient, such as bereavement or terminal illness (Sanson-Fisher & Poole, 1980).

A disadvantage of simulated patients is that they may become ideal ‘textbook cases’, to which real patients, with all their idiosyncrasies, do not often conform. Another disadvantage, discussed by Hodges et al(1997), is the expense of simulated patients, whose training and time spent performing accounted for the largest proportion of the direct cost of setting up OSCEs. Hodges et al do, however, go on to point out that this cost is more than made up for by the reduced faculty hours per annum that it takes to set up OSCEs compared with traditional oral examinations (Box 2).

Box 2 Advantages and disadvantages of objective structured clinical examinations Advantages Simulations of real-life situations Close to reality Controlled and safe Feedback from the actors (simulators) Ready availability when required Stations can be tailored to level of skill to be assessed Scenarios that are distressing to real patients can be simulated The patient variable in examination is uniform across trainees Disadvantages The idealised ‘textbook’ scenarios may not mimic real-life situations May not allow assessment of complex skills Cost Training issues in setting up the stations

 

Standardisation and training of simulated patients

TOP Who are simulated patients? Uses of simulated patients... Standardisation and training of... Objective structured clinical... Is there a burden... Conclusion Multiple choice questions References

We feel that standardisation has two components: the validity or accuracy of performance and the reliability or consistency of performance when faced with different examinees. The first of these has been determined in a number of ways. Both content and face validity have been addressed by getting an expert to review the simulations and determine their accuracy. For example, Hodges et al(1997) mention that each simulated patient was observed performing the role by the OSCE station author to verify the realism of the performance. Indirect indicators of validity might include the fact that simulators are rarely distinguished from real patients (Sanson-Fisher & Poole, 1980; Baerheim & Malterud, 1995). This non-detectability suggests that their behaviour is similar to that of real patients. This was further explored in a double-blind study in which simulated patients were substituted for real patients in the individual patient assessment of mock clinical examinations in psychiatry (further details available from the author upon request). Neither the examiners nor the students could detect the presence of simulated patients among the real patients. Hodges et al(1997) demonstrated the construct validity of OSCE psychiatry clerkship examinations by comparing the performance of residents and medical students. Concurrent validity was demonstrated by asking tutors responsible for the students to rank them with respect to their interviewing skills. The tutors’ rankings predicted the rankings generated by the performance on OSCE stations. Assessment of content validity was made by asking residents how real the simulations were: 80% described the scenarios as real or very real.

Badger et al(1995) investigated reliability of simulated patients in their consistency of performance over time and between trainees. Their study analysed the performance of 13 simulated patients during 228 doctor–patient encounters in a year-long study related to the diagnosis of depression. Results revealed high intra- and interperformance reliabilities, even when intervals between performances were as long as 3 months. The doctors detected depression in 30% of simulators, roughly the same detection rate as in real patients. In a review of simulated patients, Vu & Barrows (1994) state that ‘with good training, the simulated patients can be accurate and consistent in the essential features of their simulations’. Vu et al(1987) have also reported that simulated patients are able to enact their roles reliably up to 12 times a day.

A key requisite for achieving both accuracy and consistency of simulated patient performance is good training. Much of the literature refers to the standard methods described by Barrows (1971). For example, in a study to test the validity of psychiatric undergraduate OSCEs, Hodges et al(1997) describe the work of an experienced simulated patient trainer: ‘training for a role begins with the presentation of written material and where possible, video footage of real patients. Each simulated patient is then observed performing the role by the station’s author to verify the realism of the portrayal and to ensure consistency across the simulated patients in their presentation of affect and in their response to questions.’

Objective structured clinical examinations and psychiatry

TOP Who are simulated patients? Uses of simulated patients... Standardisation and training of... Objective structured clinical... Is there a burden... Conclusion Multiple choice questions References

Famuyiwa et al(1991), at the university of Lagos in Nigeria, investigated whether the OSCE examination in psychiatry was effective and valid. By comparing the OSCE scores of 123 students with criterion-based reference scores on multiple choice questions he found that the multiple choice marks correlated significantly with the OSCE marks.

Much of the work in this area, however, has been carried out by Hodges and colleagues at the University of Toronto. For example, in 1995, following the success of a number of pilot OSCEs, they included an eight-station OSCE into the curriculum as a criterion for medical students to pass the psychiatry clerkship. Subsequent analysis of this OSCE suggested that a psychiatric OSCE is a feasible method for assessing complex psychiatric skills. These skills were evaluated in three ways. First, behaviours identified as important to a successful interview were scored as ‘done’ or ‘not done’ on a binary checklist. For example, a station on depression might typically include the questions ‘Asks about sleep?’ and ‘Asks about suicidal ideation?’ Box 3 shows amended extracts from Guy’s, King’s and St Thomas’ psychiatric OSCE rating form. Second, a global rating scale for performance was scored using four 5-point scales that addressed each student’s organisation, rapport, interview-building skills and control of his or her emotions during the interview. Third, each examiner recorded a global impression of each student’s performance. They examined 192 medical students on two parallel forms (A and B) of the examination and found it to have a reasonably high interstation reliability (Form A: Cronbach’s = 0.64; Form B, = 0.66). However, they add that ’Whilst it is generally desirable to have a reliability of 0.80 or greater for high stakes examinations ... [t]he reliability we have found is adequate for decisions regarding a clerkship rotation’ (Hodges et al, 1997).

Box 3 Extracts from binary checklist (Yes /No) for the core features of depression in a psychiatric undergraduate examination (Unpublished: reproduced with permission of Teifon Davis) [Candidate] elicits the following aspects of patient’s main complaint Mood low/tired Feelings of loss of self-esteem Early morning wakening Loss of appetite Loss of concentration Loss of libido Work circumstance/stress Past psychiatric history Family history of mental disorder Patient’s response to mental disorder: concerns about medication/self-weakness Home circumstances: concerns about losing job

 

We therefore feel that the following issues need consideration with regard to the introduction of OSCEs into Part I of the Royal College of Psychiatrists’ membership examinations:

1. In view of this being a ‘high-stakes’ postgraduate examination, how are reliability and validity being established?
   
2. What is the added value of OSCEs over the current method of examination?
   
3. What skills can be adequately tested by binary checklists?
   
4. How can key psychiatric skills (e.g. empathy and building rapport with patients be assessed?
   

Is there a burden to simulated patients?

TOP Who are simulated patients? Uses of simulated patients... Standardisation and training of... Objective structured clinical... Is there a burden... Conclusion Multiple choice questions References

Conclusion

TOP Who are simulated patients? Uses of simulated patients... Standardisation and training of... Objective structured clinical... Is there a burden... Conclusion Multiple choice questions References

With careful planning, it does seem that OSCEs are a feasible means of testing postgraduates in psychiatry, and attempts to make the examination more valid and reliable are to be welcomed. There is the possibility of a burden to those who simulate patients, but this can be minimised by careful selection and debriefing following the examination.

When considering the use of OSCEs to test postgraduates we feel it is important to remember that their nature is to break down clinical skills into small ‘testable’ tasks. This runs the risk of training doctors who are very good at performing these piecemeal tasks without being able to assimilate them into a coherent assessment. An analogy might be a pianist who can play beautiful scales and arpeggios but cannot play a complete sonata. We would hope that senior house officers preparing for OSCEs do not forget how to take a history, make a diagnosis and formulate a management plan. Even though this will be tested in Part II of the membership examinations, we feel these are essential skills for all doctors, from house officer to consultant.

Multiple choice questions

TOP Who are simulated patients? Uses of simulated patients... Standardisation and training of... Objective structured clinical... Is there a burden... Conclusion Multiple choice questions References

1. With regard to the selection of simulated patients:
   1. they are always professional actors
      
   2. they may be real patients
      
   3. careful selection is unnecessary
      
   4. housewives have been used
      
   5. it is inadvisable to use students of acting.
      
   
   
2. The following are interchangeable with the term ‘simulated patient’:
   1. surrogate patient
      
   2. pseudopatient
      
   3. pretend patient
      
   4. standardised patient
      
   5. real patient.
      
   
   
3. Uses of simulated patients include:
   1. teaching communication skills to medical students
      
   2. reducing variability in examinations by presenting different students with the same challenge
      
   3. teaching clinical skills to postgraduates
      
   4. psychiatry examinations
      
   5. monitoring the performance of doctors.
      
   
   
4. Simulated patients have the advantage of:
   1. never being detected
      
   2. being more intelligent than real patients
      
   3. being able to play scenarios which real patients may find distressing
      
   4. being cheaper than real patients
      
   5. having no personal emotions that might influence the doctor–patient relationship.
      
   
   
5. Objective structured clinical examinations (OSCEs):
   1. have no proven reliability or validity in psychiatry examinations
      
   2. test only communication skills
      
   3. lend themselves easily to the testing of skills such as empathy or building rapport with the patient
      
   4. can be used to test ethics
      
   5. are difficult to set up.
      
   
   

View this table: [in this window] [in a new window]

MCQ answers

References

TOP Who are simulated patients? Uses of simulated patients... Standardisation and training of... Objective structured clinical... Is there a burden... Conclusion Multiple choice questions References

1. Badger, L. W., DeGruy, F., Hartman, J., et al (1995) Stability of standardised patients’ performance in a study of clinical decision making. Family Medicine, 27, 126–133.[Medline]
2. Baerheim, A. & Malterud, K. (1995) Simulated patients for the practical examination of medical students: intentions, procedures and experiences. Medical Education, 29, 410–413.[Medline]
3. Barrows, H. S. (1971) Simulated Patients: The Development and Use of a New Technique in Medical Education. Springfield, IL: Charles C. Thomas.
4. Barrows, H. S. & Abrahamson, S. (1964) The programmed patient: a technique for appraising student performance in clinical neurology. Journal of Medical Education, 39, 802–805.[Medline]
5. Cox, K. (1990) No Oscar for OSCE. Medical Education, 24, 540–545.[Medline]
6. Famuyiwa, O. O., Zachariah, M. P. & Ilechukwu, S. T. C. (1991) The objective structured clinical examination in undergraduate psychiatry. Medical Education, 25, 45–50.[Medline]
7. Harden, R. M. & Gleeson, F. A. (1979) Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education, 13, 41–54.[Medline]
8. Hodges, B., Turnbull, J., Cohen, R., et al (1996) Evaluating communication skills in the objective structured clinical examination format: reliability and generalizability. Medical Education, 30, 38–43.[Medline]
9. Hodges, B., Regehr, G., Hanson, M., et al (1997) An objective structured clinical examination for evaluating psychiatric clinical clerks. Academic Medicine, 72, 715–721.[Medline]
10. Hodges, B., Regehr, G., Hanson, M., et al (1998) Validation of an objective structured clinical examination in psychiatry. Academic Medicine, 73, 910–912.[Medline]
11. Hubbard, J. P., Levitt, E. J., Schumacher, C. F., et al (1963) An objective evaluation of clinical competence. New England Journal of Medicine, 272, 1321–1328.
12. Jason, H., Kagan, N., Werner, A., et al (1971) New approaches to teaching basic interview skills to medical students. American Journal of Psychiatry, 127, 140–142.
13. Leichner, P., Sisler, G. C. & Harper, D. (1984) A study of the reliability of the clinical oral examination in psychiatry. Canadian Journal of psychiatry, 29, 394–397.[Medline]
14. Leichner, P., Sisler, G. C. & Harper, D. (1986) The clinical oral examination in psychiatry: the patient variable. Annals of the Royal College of Physicians and Surgeons, Canada, 19, 283–284.
15. McClure, C. L., Gall, E., Meredith, K. E., et al (1985) Assessing clinical judgement with standardized patients. Journal of Family Practice, 20, 457–464.
16. Norman, G. R., Tugwell, P. & Feightner, J. W. (1982) A comparison of resident performance on real and simulated patients. Journal of Medical Education, 27, 708–715.
17. Norman, G. R., Barrows, H. S., Gliva, G., et al (1985) Simulated patients. In Assessing Clinical Competence (eds V. R. Neufield & G. R. Norman). New York: Springer.
18. Rethans, J. J. E. & Boven, C. P. A. (1987) Simulated patients in general practice: a different look at the consultation. British Medical Journal, 294, 809–812.
19. Rethans, J. J. E. , Sturmans, F., Drop, R., et al (1991) Assessment of the performance of general practitioners by the use of standardized (simulated) patients. British Journal of General Practice, 41, 97–99.[Medline]
20. Rubin, N. J. & Philp, E. B. (1998) Health care perceptions of the standardized patient. Medical Education, 32, 538–542.[CrossRef][Medline]
21. Sanson-Fisher, R. W. & Poole, A. D. (1980) Simulated patients and the assessment of medical students’ interpersonal skills. Medical Education, 14, 249–253.[Medline]
22. Sibley, J. C., Sackett, D. L., Neufeld, V., et al (1982) A randomized trial of continuing medical education. New England Journal of Medicine, 306, 511–515.[Abstract]
23. Vu, N. V. & Barrows, H. S. (1994) Use of standardized patients in clinical assessments: recent developments and measurement findings. Educational Researcher, 23, 23–30.
24. Vu, N. V. , Steward, D. E. & Marcy, M. (1987) An assessment of the consistency and accuracy of standardized patients’ simulations. Journal of Medical Education, 62, 1000–1002.[Medline]

Related articles in APT:

Commentary

Femi Oyebode
APT 2002 8: 348-350. [Full Text]  

This article has been cited by other articles:

				A. M. Brenner Uses and Limitations of Simulated Patients in Psychiatric Education Acad Psychiatry, March 1, 2009; 33(2): 112 - 119. [Abstract] [Full Text] [PDF]

				C. A. Vassilas, T. Kuruvilla, V. Gupta, and M. El-Nadeef Preparing trainees for the MRCPsych examinations Advan. Psychiatr. Treat., July 1, 2007; 13(4): 239 - 250. [Abstract] [Full Text] [PDF]

				P. Cartney Using Video Interviewing in the Assessment of Social Work Communication Skills Br. J. Soc. Work, July 1, 2006; 36(5): 827 - 844. [Abstract] [Full Text] [PDF]

				I. Pryde, A. Sachar, S. Young, A. Hukin, T. Davies, and R. Rao Organising a mock OSCE for the MRCPsych Part I examination Psychiatr. Bull., February 1, 2005; 29(2): 67 - 70. [Abstract] [Full Text] [PDF]

				A. Naeem, J. Rutherford, and C. Kenn The MRCPsych OSCE workshop: a new game to play? Psychiatr. Bull., February 1, 2004; 28(2): 62 - 65. [Full Text] [PDF]

				F. Oyebode Commentary Advan. Psychiatr. Treat., September 1, 2002; 8(5): 348 - 350. [Full Text] [PDF]

This Article Full Text (PDF) Submit an eLetter Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Related articles in APT Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Wallace, J. Articles by Haslam, R. Search for Related Content PubMed Articles by Wallace, J. Articles by Haslam, R.