Predicting NCLEX Failures: Standardized Content Mastery Assessments as a Catalyst for Improving Pass Rates

The National Council of State Boards of Nursing reports the passing percentage of first-attempt, US educated, baccalaureate degree candidates taking NCLEX-RN in 2014 at 84.93%, continuing the downward trend from recent years with reports of 85.18% in 2013 and 91.66% in 2012 (www.ncsbn.org/exam-statistics-and-publications.htm). Many pre-licensure programs of nursing adopt commercially available NCLEX readiness assessment packages to assist faculty and students to achieve success on the first-attempt at licensure. The literature supports high levels of accuracy in predicting success on NCLEX-RN using comprehensive end-of-program exit exams in a variety of sample populations (Emory, 2013, Homard, 2013; Yoem, 2013; Reinhardt, Keller, Ochart Summers, Schultz, 2012; Alameida, Prive, Davis, Landry, Renwanz-Boyle, Dunham, 2011; Harding, 2010). The limitation of end-of-program exit exams is the lack of opportunity for early intervention for students identified as at risk. Instead, the focus becomes crisis management (Reinhardt, Keller, Ochart Summers & Schultz, 2013). An added limitation of standardized assessments is in predicting the success or failure of low performing students. The body of knowledge surrounding NCLEX-RN and implementation of standardized assessments has done little to improve the ability to predict failure thus, the problem persists. Scientists in nursing education have begun to focus on standardized content-specific mastery assessments (CMA) to guide efforts in early identification and intervention for students at risk of NCLEX-RN failure. In separate studies by Emory (2013) and Yeom (2013), both scientists found the pharmacology CMA as significant in identifying potential NCLEX-RN success or failure with higher accuracy found in the ability to predict success. Other studies surrounding the subject have found a variety of CMA highly predictive of NCLEX-RN success. Missing from the literature are studies focused on predicting NCLEX-RN failure with high percentages of probability. A gap in the literature exists in the predictability of NCLEX-RN failures with high levels of probability. CMA, administered throughout the academic program, can assist in recognizing students’ weaknesses and set a foundation for remedial plans for success. The primary purpose of this pilot study was to determine if the reduction modeling can identify CMA accounting for a large amount of the variability in the standardized assessment scores for use in further analyses for predicting NCLEX-RN failure. The secondary purpose was to explore the utility of using a Principal Components Analysis (PCA) as a reduction procedure to eliminate redundancy in CMA to identify those most important in a sample of first-attempt NCLEX-RN failures. The study design was non-experimental and retrospective with a cross sectional sample of CMA scores. One baccalaureate degree granting program of nursing from the southern region of the United States implemented a standardized testing package from Assessment Technologies Incorporated (ATI), a commercial company. The program admitted students twice annually. The retrospective data collection period included students completing the program of study spring 2009 through spring 2014. The available first-attempt ATI assessment scores were collected for students who failed NCLEX-RN (N=75). Students with scores for less than three CMA were eliminated resulting in a reduced sample (n=68). The CMA consisted of seven content areas, including; fundamentals; pharmacology; maternal newborn; care of the child; mental health; adult medical-surgical; and, leadership. The community CMA was not utilized by the program.

To align with the purposes of the study, PCA was utilized to recognize the redundancy in the CMA scores. All scores were transformed to Z-scores prior to analysis to eliminate the variations in the different versions of the assessments during the data collection period. The analysis was completed using STATISTICA Extract, Transform and Load (ETL) for specialized data processing capabilities including: filtering, aggregation and analyses for trend detection (www.statsoft.com/Products/STATISTICA/Extract-Transform-and-Load). The de-identified CMA scores were imported from the existing excel spreadsheet into the STATISTICA program. The exploratory multivariate extraction technique was implemented with orthogonal rotation.

Principal Components Analysis applied a nonlinear iterative partial least squares (NIPALS) estimation to the transformed Z-scores for the CMA scores. The scientists found support for a three-component model accounting for approximately 75% of the variability in observed scores. The first component is highly correlated with the five CMA areas, especially Care of the Child, Pharmacology, and Maternal Newborn. This could suggest some redundancy in these CMA areas. The remaining CMA areas – Leadership and Adult Medical/Surgical – load on the second component and contribute greatly to the overall variation in transformed CMA scores as well. The choice of the number of principal components was guided by the analysis of the scree plot of all components, the proportion of variance accounted for by the components, and the interpretation of substantive meaning that the components contributed. The results showed the three-component model was the best fit for the observed values ranking Leadership, Fundamentals and Mental Health content as most important. The first component, with an eigenvalue of 2.97, accounts for 42.50 % of the total variance in the transformed CMA scores. Combined together, the three retained principal components – Leadership, Fundamentals and Mental Health – accounted for 75.19% of the total variance in the transformed CMA scores. The findings suggest use of a three-component model may be useful to nursing educators to recognize NCLEX-RN failure. The CMA scores found to be important in the analysis can inform future studies for predicting NCLEX-RN failure and can assist faculty in narrowing the focus of remedial efforts. The results can potentially identify the students at risk based on a combination of scores or pattern of performance increasing predictive value of failure in the student population. Understanding the phenomenon of NCLEX-RN failure as it relates to CMA has the potential to transform nursing education to focus on the most relevant content areas and thus increase the probability of recognizing students at risk. Furthermore, nurse educators can utilize the findings as a means of quality improvement in course development by recognizing the importance of the content to student success. Using this novel approach as a catalyst for change in interpretation and implementation of standardized assessments by nurse educators can impact the ability to predict NCLEX-RN failure.