One strategy to facilitate valid, evidenced-based methods of evaluation is for faculty to develop a shared mental model (SMM). McComb and Simpson (2014) describe a SMM as “individually held knowledge structures that help team members function collaboratively in their environments and are comprised of the attributes of content, similarity, accuracy and dynamics” (p. 1485). A shared mental model would enable faculty to have a more consistent and standard approach for student assessment (Boulet, Jeffries, Hatala, Korndorffer, Feinstein, & Roche, 2011; Kardong-Edgren, et al., 2017), which should lead to more fair and equitable evaluation of student performance. Kardong-Edgren et al. (2017) examine the challenges of inter- and intrarater reliability and developing a SMM and stress the importance of utilizing faculty with “similar values and professional judgment who are willing and capable of basing their judgments on the set criteria” (p.66). They also state “Our findings demonstrate how important this preparatory work is when embarking on legally defensible high-stakes testing,” (Kardong-Edgren et al.,2017, p. 67).
This presentation describes the strategies employed to build a shared mental model for faculty evaluators in simulation performance assessment. A nationwide, experimental study was conducted to test the effectiveness of a training intervention in enhancing intra and inter-rater reliability among nursing faculty evaluating student performance in simulation. The study was an extension of the NLN Project to Explore the Use of Simulation for High Stakes (Rizzolo, Kardong-Edgren, Oermann, & Jeffries, 2015) which evaluated the process and feasibility of using manikin-based high fidelity simulation for high stakes assessment in pre-licensure RN programs. A total of 102 nursing faculty recruited from nursing programs across the nation were randomized into control and intervention groups. Participants used the Creighton Competency Evaluation Instrument (CCEI) to evaluate student performance in the video recorded simulations. Following implementation of a pilot study to refine the full study procedures, researchers formulated shared mental model agreements to more clearly interpret the CCEI criteria and incorporated this information in the training intervention.
This presentation will also share findings from a qualitative analysis conducted to identify themes relative to the elements identified by study participants to support their decision of student competency in the video-recorded high stakes simulation performance. Participants were asked “Do you consider this student competent to practice nursing?” with ‘yes’ or ‘no’ response options. Two key elements in the student performance that supported this conclusion were then listed. The definition of clinical competency for purposes of the study was the ability to “observe and gather information, recognize deviations from expected patterns, prioritize data, make sense of data, maintain a professional response demeanor, provide clear communication, execute effective interventions, perform nursing skills correctly, evaluate nursing interventions, and self-reflect for performance improvement within a culture of safety” (Hayden, Jeffries, Kardong-Edgren, & Spector, 2011). Performance behaviors that reflect student competency can further enhance the shared mental model in simulation evaluation.
The National League for Nursing (NLN) calls for “fair and equitable testing in relation to high-stakes evaluation” (NLN, 2012, p. 1). As nurse educators strive to provide nursing students with an education that follows national standards, best practice guidelines, and prepares nursing students to practice in their roles as professional nurses, nurse educators must make decisions about student performance. When a SMM is formulated in the context of subjective evaluation, faculty have a clearer understanding of definitions and criteria, and can apply that SMM towards student evaluations in a fair and equitable manner that allows for more consistent evaluations (Kardong-Edgren et al., 2017). The benefit to students is that faculty are more consistent in subjective evaluations. Benefits to faculty are that a shared mental model enhances reliability in evaluation and may provide defensible evaluations in high-stakes situations if students grieve the evaluation or decide to pursue legal action. This study produced important conclusions about building a shared mental model which informs best practices in high stakes assessment.