Objective: Missing values in questionnaires occur frequently in studies involving human subjects. Although imputation methods are developed to replace missing values by plausible/estimated values, the effects of the imputed data on analysis outcomes are lacking. The objective of this study is to assess the performance of four imputation methods on a wide range of outcomes under various data conditions and under the mechanism of missing completely at random.
Design: an experimental 4*3*3 design was used. There were four imputation methods, three levels of percentage of subjects with missing values, and three levels of percentage of missing items in the scale.
Population, Sample, Setting, Years: This study was performed using secondary data from a Health survey of the elderly collected during 1998 and 1999. A random sample of 500 was selected from the total of 2058 cases with complete data to carry out this analysis.
Intervention and outcome variables: The intervention was four imputation methods including item mean substitution (IMS), person mean substitution (PMS), expectation-maximization algorithm (EM), and stochastic regression imputation (SRI). The other two independent variables were percentage of subjects with missing values (10%, 25% and 40%) and percentage of missing items (10%, 30%, and 50%). The outcome variables were accuracy and bias of parameters estimated. The parameters included item mean, item SD, correlations among items, scale mean, scale SD, correlations among scales, and internal consistency (coefficient alpha).
Methods: This study consisted of multiple steps. First, a random sample of 500 was selected from the dataset. Second, randomly selected values from the sample were deleted to create missing data under different percentages of missing values. Third, the imputation methods were applied and statistics from the imputed datasets were computed. Fourth, differences of statistics between imputed data and those obtained from the complete data, or accuracy, and the direction of the differences, or bias, were computed. Finally, the accuracy and bias for each parameter were compared and contrasted among the four methods.
Findings: There were significant differences in accuracy for imputation methods regardless of data conditions.IMS was consistently the worst in terms of accuracy for all but two parameters. PMS was the second worst in parameter estimates. In contrast, EM and SRI produced more accurate estimates for most parameters considered. EM ranked the best for the estimation of item mean, correlations among items, and correlations among scales. SRI ranked the best for item SD and coefficient alpha. The two methods were tied as the best for estimating scale SD. In terms of bias, IMS consistently underestimated both item and scale SDs. IMS also tended to underestimate correlations among items, correlations among scales, and coefficient alpha. PMS tended to underestimate item SDs but overestimate correlations among items. As a result, it tended to inflate coefficient alpha. The scale SDs tended to be overestimate by PMS. EM tended to underestimate item and scale SDs but the size of the bias was smaller than using IMS. EM tended to overestimate correlations among items and among scales. Thus it also inflated coefficient alpha but the inflation was smaller than using PMS. Comparing with the other three methods, SRI tended to produce less bias for estimation of item and scale SDs, correlations among items and coefficient alpha. The results for estimations of correlations among scales were similar for SRI, PMS and EM. Percentage of subjects with missing values and percentage of missing items significantly impacted the accuracy and bias of the imputation methods. The differences among methods increased as the percentage of subjects with missing values increased and as the percentage of missing items increased.
Conclusions: Model-based imputation approaches, SRI and EM, produced more accurate parameter estimates than IMS and PMS for missing items in Likert type measures. However, random errors should be considered when using EM method because it tended to produce bias for several parameters.
Implications: To ensure conclusion validity from research studies, SRI should be used to handle missing item values in Likert-type measures.
Back to Analysis and Design of Measurement Scales
Back to The Advancing Nursing Practice Excellence: State of the Science