This research was published in College & Research Libraries. View the full article.
Course | Sample Size | Percentage of Enrollment | Scoring Method |
ENGL 1010: First-year composition | 270 | 14% | Norming exercise; divided papers between 3 final raters who reached good level of agreement |
ENGL 2010: Second-year composition | 468 | 27% | Norming exercise; divided papers between 3 final raters who reached good level of agreement |
PSY 3500: Introduction to Research Methods | 101 | 49% | Three scorers reached consensus on all papers |
HIST 4990: Capstone Seminar | 45 | 83% | Two scorers reached consensus on all papers |
Total | 884 |
The ENGL 1010 and 2010 sample was quite similar to the total enrolled population in the categories of gender, GPA, class standing, and admissions index (a combination of high school GPA and ACT score).
Inter-rater reliability was calculated with Krippendorff's alpha coefficient, using the online calculator ReCal (OIR).
ENGL 1010 Inter-rater reliability (n=30 papers) |
ENGL 2010 Inter-rater reliability (n=30 papers or * n=9 papers) |
|||
Category | Level of Agreement | Category | Level of Agreement | |
One | 0.61 | One* | 0.604 | |
Two | 0.627 | Two | 0.844 | |
Three | 0.619 | Three | 0.615 | |
Four | 0.635 | Four | 0.609 | |
Five | 0.743 | Five* | 0.752 | |
* required second round of norming with 9 papers to reach good level of agreement |
Below are the scores for the papers we collected for all four classes. The papers were scored with 0 being the lowest possible score and 4 being the highest. In most cases, the "peaks" moved to the right (the higher scores on the X axis), suggesting that students did improve their information literacy skills across the curriculum.
While no class performed extremely well in defining the information need, English 1010 scores were particularly low, while upper division classes rated higher. Most students struggled to narrow a topic and define key concepts.
Students showed evidence of being able to find information (often in library databases) and demonstrated improvement across the curriculum, especially in PSY 3500. In ENGL 2010, students often relied on a single database search, as seen in the URLs of their bibliographies. History Capstone students sometimes relied too heavily on one or two primary and/or secondary sources, suggesting that they were not conducting comprehensive searches.
Students struggled with evaluating information in context. Many assignments did not directly ask students to evaluate the information they cited in their papers (especially in ENGL 1010) and used "checklists" of required types of sources. So many students found scholarly sources, but were not able to articulate why they provided useful evidence for their argument or thesis. Students across the curriculum had a tendency to take each source at face value.
Students at each level struggled to use their information, which included the ability to synthesize and organize their information. The most common problem, especially in ENGL 1010, ENGL 2010, and PSY 3500, was a "claim-then-quote" style. Each paragraph would begin with a topic sentence and then quote a source at length. Then the next paragraph would do the same.
Most students scored in the 3-4 range, citing their sources proficiently both within the text and in bibliographies. We did not account for correct citation styles, but most students deomonstrated the ability to give credit to the sources they used.