Patient safety is a global priority for nurses and healthcare providers alike, yet medical errors are not decreasing. The Agency for Healthcare Research and Quality (AHRQ, 2014f) indicates that one out of seven Medicare patients in the United States will be the victim of a medical error. James (2013) concluded that medical error accounts for 200,000 to 440,000 deaths per year, with many of these errors related to ineffective teamwork and communication (AHRQ, 2014f). In an effort to improve teamwork and communication in healthcare, the United States Department of Defense and AHRQ developed the Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPs®) Instructional Framework to facilitate teamwork education and evaluation. TeamSTEPP’s® evaluation instruments include the Teamwork Attitudes Questionnaire, Teamwork Perceptions Questionnaire, and the Teamwork Performance Observation Tool (TPOT). The TPOT (AHRQ, 2014a) was designed to objectively measure teamwork performance but has not been sufficiently studied. The AHRQ encourages nationwide implementation of these essential teamwork concepts in order to decrease morbidity and mortality associated with medical error.
Significance
Teamwork principles can be implemented in every high-fidelity simulation according to Clapper and Kong (2012). The NLN’s Vision for Simulation in Nursing Education emphasizes the imperative for nurse educators to purposefully integrate simulation throughout curricula to promote clinical reasoning and good judgment (NLN Vision statement, 2015). Learning outcomes from these experiences need to be evaluated using reliable and valid instruments especially when these experiences replace clinical time (NLN Vision Statement, 2015). The ultimate goal of nursing education, despite teaching pedagogy, is for students to translate learning into clinical practice (Adamson, Kardong-Edgren, & Willhaus, 2013).
Zhang (2015) applied the Event Based Training (EBT) methodology to teamwork education and the development of Targeted Behavioral Markers (TBMs) which are scenario specific and reflect teamwork behaviors from the Team Performance Observation Tool (TPOT). With EBT, simulation scenarios are embedded with events which trigger an expected response creating an objective evaluation and facilitating feedback for improvement (Dwyer, Oser, Salas, and Fowlkes, 1999).
Purpose:
The purpose of this research study was to analyze psychometric properties of the Teamwork Performance Observation Tool (TPOT) enhanced with Targeted Behavioral Markers (TBM) using a methodology by which validation can be efficiently accomplished and replicated.
Methods:
This presentation will discuss a methodological study which sought to demonstrate the viability of the Team Performance Observation Tool enhanced with Targeted Behavioral Markers (TBMs) as a sensitive and valid measure of teamwork performance and a superior approach to teamwork assessment without TBMs. This study provides psychometric data for TeamSTEPPs® Teamwork evaluation instruments and a discussion of interrater reliability concerns, including implications for future studies using observational instruments.
Research Design
The study employed a quasi-experimental Multitrait-Multimethod (MTMM) design to investigate construct validity of the TPOT with TBMs. It compared methods of evaluation and traits being evaluated to determine if the evaluation method affects the outcome of testing. In this study, instruments with established reliability (TPOT, T-TPQ, T-TAQ) and varying methods of evaluation were utilized to evaluate teamwork along with newly developed TBMs which identify scenario specific teamwork behaviors. The Teamwork Nomological Network was developed for this study to illustrate theoretical relationships between the concept of teamwork, individual traits and outcomes, and associated measurement instruments. The Multitrait-Multimethod Correlation Matrix is a numerical expression of the Teamwork Nomological Network.
Aim and Research Questions
The aim of this research was to assess the convergent and divergent validity of the Team Performance Observation Tool (TPOT) enhanced with Targeted Behavioral Markers (TBM) as a valid measure of teamwork skill and performance acquisition in simulation training.
RQ 1:What is the relationship between TPOT, TPOT with TBMs, and the T-TPQ, T-TAQ, and the NLN Simulation Checklist (assesses convergent validity)?
RQ2: What is the relationship between the TPOT, TPOT with TBMs, and the Clinical Skills Self Efficacy Scale (assesses divergent validity)?
Sample, Setting, Recruitment
The study used a convenience sample of senior level baccalaureate nursing students enrolled in a clinical capstone nursing course (n=57) at a medium sized University and within a ten bed high fidelity Simulation Center. After Institutional Review Board (IRB) approval was obtained, the PI explained the purpose and design of the study, answer questions, invited clinical capstone students to participate, and obtained their voluntary written consent. Students were eligible to participate in the research study if all prior course work was complete. No student enrolled in the capstone course was excluded from the study since all students were expected to participate in the final clinical experience and were expected to successfully complete the nursing program. Participation in the study involved completing surveys and analysis of observational data. Choosing to participate or not participate in the study did not affect the student’s grade for the course.
Rater Training
Three baccalaureate nursing students, who are not in the study sample, were recruited to participate in testing the simulation scenario and create a video for conducting interrater reliability among study raters. These volunteers received the TeamSTEPPs® teamwork education workshop but did not complete study surveys or demographic forms. The volunteer simulation was realistic and revealed technical issues in the scenario implementation, pre-scenario instructions and checklists which needed to be made prior to the study. Five faculty and two TeamSTEPPs® Master Trainers participated in a four hour faculty development workshop conducted by the PI (TeamSTEPPs® Master Trainer) in order to review the simulation, study instruments, and obtain baseline interrater reliability. Baseline interrater reliability of 80% was established for the observational instruments.
Data Collection
Five nurse faculty members each evaluated and scored three or four teams consisting of three nursing students using the observational instruments. Each team participated in the same simulation developed by the National League for Nursing (NLN) and Laerdal®. The PI viewed all simulations and rated each group on the observational instruments. The two TeamSTEPPs® Master Trainers viewed and rated 50% of the simulations and participated in the faculty development workshop. Ratings from the five faculty members, PI, and two TeamSTEPPs® Master Trainers were compared and interrater reliability was examined. Participating students completed the Clinical Skills Self Efficacy Scale, the T-TPQ (teamwork perceptions) and T-TAQ (teamwork attitudes) using anonymous Blackboard® surveys after the simulation.
Data Management
Data consisted of individual Likert style individual student surveys (Demographics, Clinical Skills Self Efficacy Scale, T-TPQ, T-TAQ) as well as group team performance evaluations (TPOT, TPOT with TBMs, NLN Simulation Checklist) and recorded simulations. Each student was assigned an individual identification number, team identification number, and faculty identification number. A code book linking subjects with their individual, team, and faculty evaluator identification numbers was maintained on the PI’s laptop computer which was maintained in a locked office and password protected along with videos of the simulations. Team raw scores and mean scores were calculated for the individual surveys in order to analyze relationships between survey data and teamwork observational data.
Data Analysis
The data was analyzed using the Statistical Package for Social Science (SPSS) version 22. Preliminary analysis included demographics, reliability statistics for each instrument, and Intraclass Correlations to analyze interrater reliability between raters on the observational instruments. Pearson correlations were calculated and placed in the Multitrait-Multimethod Correlation Matrix for interpretation. Pearson Correlations were calculated between the TPOT without TBMs, TPOT with TBMs, T-TPQ, T-TAQ, and NLN Simulation Checklist and entered in to the Multitrait-Multimethod Correlation Matrix. Moderate correlations (.4 - .6) indicate convergence while high correlations (.6 - .9) may indicate that the concepts are very similar. Pearson Correlations for the TPOT, TPOT with TBMs, and the Clinical Skills Self Efficacy Scale were also calculated. Low correlations (<.4) between self efficacy, team performance measures, and outcomes indicate divergence.
Results:
Correlations between the TPOT and TBMs (r 2=.87, p < .01) and between the TAQ and TPQ surveys were significant and high (r 2=.47, p < .01). However, correlations between observed teamwork performance (TPOT and TBM), teamwork attitudes (TAQ), and perceptions (TPQ) overall scores were weak in this study (r < 0.4). No study was identified for comparison. Unexpected findings may be related to rater training, faculty teamwork knowledge and simulation experience, ambiguity of behavioral markers, contamination of TPOT results by exposure to TBMs, and varying student performance. Conversely, self-efficacy with clinical skills demonstrated low correlations with teamwork attitudes and perceptions as predicted, but unexpected high correlations with observed teamwork scores. High correlations between clinical skill self-efficacy and observed teamwork performance scores (p < .05) may indicate that clinical skill self-efficacy influences teamwork performance. Additional study of these relationships using fully crossed designs for interrater agreement are needed in order to clarify the MTMM Correlation Matrix and provide a framework to assess future teamwork behavioral markers.
Conclusion:
Teamwork education and evaluation in nursing education is essential to promote patient safety. Assessment of teamwork perceptions and attitudes can be used to indicate the effectiveness of teamwork education (Brock et al., 2013); however, attitudes and perceptions do not always translate to clinical practice. Established rigorous methods for determining reliability and validity need to be applied to testing objective teamwork performance instruments. Findings from this study provide much needed psychometric data for the TPOT and unexpected findings regarding the relationship between the TPOT, TAQ, and TPQ.
See more of: Research Sessions: Oral Paper & Posters