MINES for Libraries ACRL: Outcome Assessment Tools for

MINES for Libraries ACRL: Outcome Assessment Tools for

MINES for Libraries ACRL: Outcome Assessment Tools for the Library of the Future: MINES at OCUL Toni Olshen York University Association of College and Research Libraries ACRL Conference 2005 Minneapolis April 7, 2005 www.arl.org/stats/ Ontario Council of University Libraries OCUL is a consortium of twenty university libraries in the province of Ontario

The member libraries cooperate to enhance information services through: resource sharing, collective purchasing, document delivery and many other similar activities. www.minesforlibraries.org OCUL Members Those in Green are ARL Libraries Brock University Queen's University Carleton University Royal Military College of Canada University of Guelph

Ryerson Polytechnic University Lakehead University University of Toronto Trent University Laurentian University University of Waterloo McMaster University University of Western Ontario Nipissing University

Wilfrid Laurier University Ontario College of Art & Design University of Ontario Institute of Technology University of Windsor York University University of Ottawa www.minesforlibraries.org Member Institution Enrolments (2003)- Total Undergraduate and graduate students

University of Ontario Institute of Technology 936 Royal Military College of Canada 1,941 Ontario College of Art and Design 3,062

Nipissing University 5,478 Lakehead University 7,304 Trent University 7,388 Laurentian University 8,751 Wilfrid Laurier University 12,426 Brock University 15,527

University of Windsor 16,266 University of Guelph 19,096 Queen's University 20,034 McMaster University 22,064 Carleton University 22,535 University of Waterloo 25,029 Ryerson University 27,221 University of Ottawa 30,948 University of Western Ontario

32,784 York University 46,794 University of Toronto 68,290 Total 391,933 90% undergrads 10% graduate 12,500 faculty www.minesforlibraries.org Scholars Portal What is it?

A unique set of shared information resources and services Resources acquired and managed through OCUL with funding support from a 5-year grant from the Ontario Innovation Trust (OII), a provincial funding body Resources are made available to researchers and students in Ontario through their own university libraries www.minesforlibraries.org Scholars Portal Ontario Council of University Libraries (OCUL) Ontario Information Infrastructure (OII) funded by the Ontario Innovation Trust in 2001 for five years

Consortia-purchased electronic resources offered through the Ontario Scholars Portal March 2004, we began the evaluation phase of $7.6 million dollar OII project www.minesforlibraries.org Scholarly Information Resources As of the end of March, contains 7,547,904 full text articles from 6,783 full text journals published by 12 academic publishers Coverage of most disciplines but concentration in sciences Current and historic coverage One of the largest collections of electronic journals available to researchers anywhere www.minesforlibraries.org

Scholars Portal Resources Academic Press, American Psychological Association,

American Chemical Society, Berkeley Electronic Press, Cambridge University Press, Emerald Publishing, Elsevier Science (Elsevier Science, Harcourt Health Sciences), Kluwer (Kluwer Academic Publishers, Kluwer Law International and Kluwer/Plenum), Oxford University Press, Project MUSE, Springer-Verlag, and John Wiley & Sons. www.minesforlibraries.org Scholars Portal Project Goals Centrally mount and deliver information

resources acquired through OCUL consortia purchases to ensure rapid and reliable access Provide for the long term, secure archiving of resources to ensure continued availability www.minesforlibraries.org Scholars Portal Project Goals Ensure that the resources and services provided meet the needs of faculty, students and staff. Ensure that resources and services can be seamlessly integrated to the local library and information systems www.minesforlibraries.org

Measuring Success OCUL provides a sophisticated statistical report mechanism. (see next slide). Download statistics are a rough measure of value but we need more to properly assess impacts. Need to measure also the significance for research of access to e-journals Employing ARL MINES Survey methodology to capture information on how resources are being used (from where, by whom, and for what purposes) www.minesforlibraries.org SP Statistics and Report Generator

www.minesforlibraries.org Why Evaluation? Feedback to OII and University funders Understand who, where, and why the digital resources are used Supplement usage numbers to answer the key question: What is the impact of Portal content on research at Ontario academic libraries? www.minesforlibraries.org Evaluating Success Evaluating Scholars Portal from user and staff points of view Use a mix of quantitative and qualitative tools for a richer assessment: MINES, focus

groups, staff survey Are OII projects improving research services? Does Scholars Portal meet OCUL user and staff expectations? www.minesforlibraries.org MINES (Measuring the Impact of Networked Electronic Services) MINES survey is one of a new breed of assessment tools that did not exist before because services were not digital. www.minesforlibraries.org

MINES (Measuring the Impact of Networked Electronic Services)-Desired Outcomes To capture in-library and remote web usage of the Scholars Portal in a sound representative sample using MINES methodology; To identify the demographic differences between in-house library users as compared to remote users by status of user; www.minesforlibraries.org MINES (Measuring the Impact of Networked Electronic Services)-Desired Outcomes

To identify users purposes for accessing Scholars Portal electronic services (funded research, non-funded research, instruction/education use, student research papers and course work); To assist with the evaluation of the project as well as to capture information for OCUL about indirect research costs; and To develop an infrastructure to make studies of patron usage of networked electronic resources routine, robust and integrated into the decision-making process.

www.minesforlibraries.org MINES Methodology What user groups use SP? What specific resources are used? From where? How do users learn about SP? Are there differences in the use of digital resources based on the user's location? Why use SP? (sponsored research? Instruction? patient care?)

Does use differ by discipline? user group? location? www.minesforlibraries.org MINES Methodology Web-based surveys conducted over the course of a year for each institution Activated during randomly selected 2-hour survey periods each month as users access one of SPs journals Mandatory, short, and anonymous www.minesforlibraries.org ARL/MINES Jan. 04-Dec. 05

ARL developed random schedule of two-hour sessions per month OCUL designed local questions, mounted survey, collects and sends data to ARL ARL compiles survey results for all sites ARL reports findings on a semi-annual basis

ARL presents findings and final report to project participants on an aggregated and individual institution basis www.minesforlibraries.org Development of survey form Finding balance between simplicity, ease and richness of data elements Bilingual University of Ottawa, Laurentian University, Glendon College at York University Ultimately a change in focus to the creation of a unique data set www.minesforlibraries.org

MINES Survey Form Five Questions and a Comment Box www.minesforlibraries.org Survey Form Survey form determined : users status Discipline (affiliation) location or where accessed from purpose of use (sponsored research, instruction, patient care, course work) how the resource was identified (bibliography, colleague, librarian, important journal in field etc.) www.minesforlibraries.org

OCUL Definition of Usage for MINES A successful search connecting the user to an article of interest for viewing, printing or downloading Unique to Scholars Portal because of consortia server setup and archiving of all journals www.minesforlibraries.org MINES Methodology Random sampling plan and the mandatory nature of the questions are both required to create a statistically sound study If the survey is not mandatory, the group of non-respondents is likely to be different from

the group of respondents, and we will not know what that difference is One of the strengths and innovations of this survey technique is that it is based upon actual use, not on predicted, intended, or remembered use www.minesforlibraries.org OCUL Implementation of MINES Once the survey is completed, the respondent's browser is forwarded to the desired networked electronic resource If more than one search is carried out, the survey form is auto-populated with users responses as defaults which

only have to change if response is different www.minesforlibraries.org Informed Consent Because this is a Web-based survey, the respondents consent to participate by electing to fill out the survey questionnaire It is the participating librarys responsibility to provide an explanation of the survey and information pertaining to its confidentiality www.minesforlibraries.org Confidentiality of Data

Institutional data are confidential. Individual institutions and/or their specific data will not be identified. Individual data are anonymous. The respondents privacy is protected because only very indirect information is captured, which would be difficult to trace back to an individual. www.minesforlibraries.org Ethics Review A major step was contacting research ethics officers and/or Ethics Review Boards to get approval, where necessary, to run the survey Purpose of ethics reviews for human subjects is to prevent putting subjects at risk

Officers/Boards on 16 OCUL campuses accepted that no physical or psychological harm would come to library users who are asked to fill out a brief mandatory anonymous survey before they are connected to the title of their choice. www.minesforlibraries.org Ethics Review Reference to interesting opinion piece by J. Paul Grayson. How Ethics Committees are Killing Survey Research on Canadian Students. University Affairs, January 2004. http://www.universityaffairs.ca/issues/ 2004/jan/print/opinion.html

www.minesforlibraries.org Mandatory Survey If individuals chose to avoid filling out the brief anonymous survey, they might be inconvenienced for a maximum of a two-hour period, but they would not be harmed We needed to balance good data for making decisions and the inconvenience caused to the user. www.minesforlibraries.org Ethics Review Issues and Problems

Mandatory nature of the survey required discussion on some campuses Several campuses did not require approval because the survey fell into quality assurance guidelines and was seen as a library management tool (8) Several schools received approval after an application process (8) One Library and Review Board did not support the mandatory nature of the methodology so that school dropped out of the project. www.minesforlibraries.org Pre-testing and False start January March 2004 ARL prepared a schedule for the random two-hour monthly runs.

A test run was planned at York and Wilfrid Laurier in January with the real survey commencing at the end of February. The pilot in January failed at York and highlighted the need for all institutions to be using a link resolver URL when connecting to SP journals from their catalogues or eResources databases. Each site reviewed their configuration and necessary changes were made. www.minesforlibraries.org Pre-testing and False start January March 2004 Survey form and the explanatory material were translated into French for bilingual Ottawa, Laurentian, and Glendon College at York. February run highlighted concerns about the data

collection. The technical infrastructure was capturing only access through library catalogues or eresource databases, but not from the use of the SP directly. There were some technical problems with the February and March runs and the validity of the data was under question. The data-collection programming was revisited. www.minesforlibraries.org Lessons Learned Early runs taught us a great deal about the different ways OCUL libraries access the SP We needed to reflect that in the data gathering

www.minesforlibraries.org Lessons Learned As originally planned, we now capture as much usage as possible that comes from : local eresource databases library catalogues Scholars Portal browse and search functions. www.minesforlibraries.org New Definition of Usage for MINES A successful search is now defined as connecting the user to an article of interest for viewing, downloading or printing Definition is unique to Scholars Portal

because of consortial server setup and archiving of content We cancelled the April 20 run and reset the dates of the survey from May 2004 through April 2005, considering the February and March runs as tests. www.minesforlibraries.org New Definition of Usage for MINES Innovation We continue to build on the unique opportunity we have to gather useful data that is not open to other types of library groups. By the end of March about 22,500 surveys have been completed. One more month to go! By implementing the MINES survey, OCUL is ahead of other projects in that we are not held "hostage" to the limitations and inconsistencies of vendor

statistics We have opportunities to disseminate research on measurement of networked resources through conferences and publications www.minesforlibraries.org MINES Very Preliminary Output: MAY AUGUST 2004 5223 respondents www.minesforlibraries.org Very Preliminary Findings 4 months of data -Subject Affiliation

Applied Sciences Business Education Environmental Studies Fine Arts Humanities Law Medical Health

Sciences Social Sciences Other 804

146 176 160 22 93 21 1341 1031 673 129 www.minesforlibraries.org 17.5% 3.2 3.8 3.5

.5 2.0 .5 29.2 22.4 14.6 2.8 Very Preliminary Findings 4 months of data User Status Faculty 764 16.6%

Graduate/Professional 2068 45.0 Undergraduate 1039 22.6 Library Staff 47

1.0 Staff 427 9.3 Other 251

5.5 www.minesforlibraries.org Very Preliminary Findings 4 months of data - Location Library 578 12.6% Off-Campus 1978

43.6 On-Campus ( but not in the library) 2040 44.4 www.minesforlibraries.org Very Preliminary Findings4 months of data - Purpose of Use Sponsored research Other nonsponsored research Teaching Course work

Patient care Other activities 2189 47.6% 20.0 6.0 14.9 3.1

8.3 919 278 686 143 381 www.minesforlibraries.org Cross Tabulations Purpose of use by affiliation, user status, location, why Location by affiliation, user status, purpose of use, why Why by affiliation, user status, location, purpose of use

Which titles used by which users for which purposes www.minesforlibraries.org Location and Purpose of Use www.minesforlibraries.org Additional Qualitative Data MINES Survey respondent comments Staff Survey: What does the range of institutional experiences reveal? Focus Groups: What anecdotal data can faculty and students add to the development of the Scholars Portal? www.minesforlibraries.org

Thank you for your attention! Questions? Toni Olshen: [email protected] www.minesforlibraries.org

Recently Viewed Presentations

  • Information Literacy - Virginia Union University

    Information Literacy - Virginia Union University

    L. Douglas Wilder Library. What is Information Literacy? Information literacy is the ability to know that you need information and then how to locate, evaluate and effectively use the information in whatever manner you choose, i.e. research papers, presentations, etc....
  • 1.2.4  Command-Line Data Analysis and Reporting 2.1.2.4.4 Command-Line

    1.2.4 Command-Line Data Analysis and Reporting 2.1.2.4.4 Command-Line

    Perl Prompt Tools addband addwell collapsedata column digestvector enzyme extract fields histogram we saw these last time addband - annotate coordinates with cytogenetic bands cytogeneticists frequently use bands to identify regions, not coordinates if you send a cytogeneticist coordinates, he...
  • Word List: tract = pull What is the

    Word List: tract = pull What is the

    the root tract. What is the root that means pull? ... What is the word that describes what your legs are in when they are being pulled back in the hope of popping the bones back into the proper place?...
  • Unit 4: Test Review The Interwar Years, 1900-1939

    Unit 4: Test Review The Interwar Years, 1900-1939

    How do you define a depression? Bank closings. Imports and exports. Business failures. During the Great Depression, which of the following did not increase? ... Hitler pushed for lebensraum which meant "living space."How did Hitler propose to get more living...
  • Jesus Shall Reign - Traditional Music Library

    Jesus Shall Reign - Traditional Music Library

    Jesus shall reign where'er the sun. Doth his successive journeys run; His kingdom stretch from shore to shore, Till moons shall wax and wane no more.
  • Office of Railroad, Pipeline and Hazardous Materials Safety

    Office of Railroad, Pipeline and Hazardous Materials Safety

    Signal Presentation The investigation determined that had these stacked commands been sent prematurely, the signal equipment would have responded as it did during postaccident testing; that is, the signal system would have discovered a potential routing conflict and changed all...
  • Chapter 2: Scaling VLANs Instructor Materials CCNA Routing

    Chapter 2: Scaling VLANs Instructor Materials CCNA Routing

    Trunk negotiation is managed by the Dynamic Trunking Protocol (DTP) DTP is a Cisco proprietary protocol. ... Chapter 2: Scaling VLANsNew Terms and Commands. virtual local area networks (VLANs) trunks . VLAN Trunking Protocol (VTP)
  • Homophones and Homonyms - Weebly

    Homophones and Homonyms - Weebly

    Synonym and Antonym. Synonym: word having the same or nearly the same meaning as another in the language, as happy, joyful, elated. Antonym: a word opposite in meaning to another. Fast is an antonym of . slow.