Graduate-Level Online Grading Rubrics: Validity and Reliability

Monday, 18 November 2013: 10:20 AM

Roberta A Raymond, PhD, RN
MSN Online Program, Chamberlain College of Nursing, Downer's Grove, IL
Kathleen M. Hunter, PhD, RN-BC, CNE
Chamberlain College of Nursing, Downer's Grove, IL
Rebecca Sisk, PhD, RN, CNE
MSN Online Program, Chamberlain College of Nursing, Downers' Grove, IL

Learning Objective 1: Examine assignment grading rubrics for validity and reliability;

Learning Objective 2: Develop accurate guidelines for grading threaded discussions in an online program

Grading rubrics are frequently used in online educational programs as a part of the assessment of student learning.  With the diversity of faculty teaching in these programs, there is a concern with unbiased and impartial feedback for students.  Faculty members at an online Master of Science in nursing program have been working on a program of research to assure the inter-rater reliability and content validity of the rubrics used to grade threaded discussions (TDs).  

This presentation will describe research conducted on the TD grading rubric thus far. The following questions were addressed in the initial phase of the research: “Do faculty members apply the TD grading rubric similarly, and does the grading rubric reflect the student knowledge and skills intended by faculty?” Faculty members randomly selected weekly posts of 20 students in an evidence-based practice nursing course, resulting in 196 sets of posts related to TD questions. Two faculty members simultaneously applied the rubric to each student’s set of posts. Cohen’s kappa was used to estimate inter-rater reliability.

The first study, completed in 2011, resulted in a Cohen’s kappa of 0.227 with overall agreement of 56% between the two raters indicating inconsistency in grading.   Based on these results, the rubric was revised with specific rubric guidelines rewritten for students and faculty. 

The study was repeated in 2012 using two different raters and a different course.  There were minimal changes in the outcomes of the study.  The Cohen’s kappa was 0.049 with an overall agreement of 52%.  Faculty is currently investigating the specific issues in the second study causing the low Cohen’s Kappa.

The research group will then revise the guidelines and directions for the grading rubric to develop accurate guidelines for grading TDs and assignments and to pursue a program to continually improve grading and feedback for students.