Skip to Main Content

An Information Literacy Snapshot: Results of a Large-Scale Rubric-Based Assessment Project: Our Study

W. Holliday, K. Lundstrom, E. Davis, B. Fagerheim, A. Hedrich, P. Martin

Published Paper

This research was published in College & Research LibrariesView the full article.

Our Process

We rated papers from four courses to represent stages of development of information literacy skills across the curriculum at USU. We used the AAC&U VALUE Rubric for Information Literacy to get started, and refined the rubric with more accessible language and specific examples. Several librarians scored the papers, and we spent much time reaching a level of agreement on the scores. 

Our Sample and Method

Course Sample Size Percentage of Enrollment Scoring Method
ENGL 1010: First-year composition 270 14% Norming exercise; divided papers between 3 final raters who reached good level of agreement
ENGL 2010: Second-year composition 468 27% Norming exercise; divided papers between 3 final raters who reached good level of agreement
PSY 3500: Introduction to Research Methods 101 49% Three scorers reached consensus on all papers
HIST 4990: Capstone Seminar 45 83% Two scorers reached consensus on all papers
Total 884

The ENGL 1010 and 2010 sample was quite similar to the total enrolled population in the categories of gender, GPA, class standing, and admissions index (a combination of high school GPA and ACT score).

Reaching Consensus

Inter-rater reliability was calculated with Krippendorff's alpha coefficient, using the online calculator ReCal (OIR).

ENGL 1010 Inter-rater reliability
(n=30 papers)
ENGL 2010 Inter-rater reliability
(n=30 papers or * n=9 papers)
Category Level of Agreement  Category Level of Agreement 
One 0.61 One* 0.604
Two 0.627 Two 0.844
Three 0.619 Three 0.615
Four 0.635 Four 0.609
Five 0.743 Five* 0.752
* required second round of norming with 9 papers to reach good level of agreement

Our Data

Below are the scores for the papers we collected for all four classes. The papers were scored with 0 being the lowest possible score and 4 being the highest. In most cases, the "peaks" moved to the right (the higher scores on the X axis), suggesting that students did improve their information literacy skills across the curriculum.

While no class performed extremely well in defining the information need, English 1010 scores were particularly low, while upper division classes rated higher. Most students struggled to narrow a topic and define key concepts.

Students showed evidence of being able to find information (often in library databases) and demonstrated improvement across the curriculum, especially in PSY 3500. In ENGL 2010, students often relied on a single database search, as seen in the URLs of their bibliographies. History Capstone students sometimes relied too heavily on one or two primary and/or secondary sources, suggesting that they were not conducting comprehensive searches. 

Students struggled with evaluating information in context. Many assignments did not directly ask students to evaluate the information they cited in their papers (especially in ENGL 1010) and used "checklists" of required types of sources. So many students found scholarly sources, but were not able to articulate why they provided useful evidence for their argument or thesis. Students across the curriculum had a tendency to take each source at face value.

Students at each level struggled to use their information, which included the ability to synthesize and organize their information. The most common problem, especially in ENGL 1010, ENGL 2010, and PSY 3500, was a "claim-then-quote" style. Each paragraph would begin with a topic sentence and then quote a source at length. Then the next paragraph would do the same.

Most students scored in the 3-4 range, citing their sources proficiently both within the text and in bibliographies.  We did not account for correct citation styles, but most students deomonstrated the ability to give credit to the sources they used.

Our Library Instruction: What We've Changed (So Far)

  • Target areas of improvement: evaluation and information use (all courses)
  • Currently meeting with instructors and departments to redefine our learning goals to match these areas and to create learning activities.
  • Slow down the research process and build reading and thinking time into classroom instruction (ENGL 1010 and 2010).
    See ENGL 1010 and ENGL 2010 library instruction lesson plans. The ENGL 1010 Library Day One reiterates the importance of reading and thinking as part of the research process.
  • Flip the Classroom: teach mechanics as homework and point-of-need instruction and spend more time thinking, reading, and discussing information in class (all classes). See this online library guide on Flipping the Library Classroom.
  • Teach summary and require it in preliminary assignments (e.g. changed assignment in PSY 2010, a prerequisite to PSY 3500, to include summarizing assignment).
  • No more checklists or required number of sources (ENGL 1010 and ENGL 2010). We emphasize context, usage, and disciplinary standards or evidence over type and number of sources and we encourage instructors to do the same.
  • Teach research skills earlier in curriculum (PSY and HIST).