| ALA American Library Association | Search ALA Contact ALA Login |
|
|||||||||||||
|
|
Task Force on Academic Library Outcomes Assessment ReportJune 27, 1998I. Context and BackgroundSome fifteen to twenty years ago higher education began to focus on measuring the outcomes of its programs as the primary indicator of quality. The forces that impelled this focus included a restructuring of the criteria of the regional accrediting agencies to emphasize assessment, the interest of state legislatures and federal agencies in requiring accountability of the institutions they that fund, and the desire of the institutions themselves to market their product as "high quality" to a (then) shrinking population of college-bound students. Virtually every state higher education coordinating board now requires public institutions to annually provide accountability data in terms of output measures. Recently, a broad coalition of six associations of colleges and universities, representing a majority of the nation's institutions of higher education issued a statement explicitly rejecting accrediting agencies' requiring specific inputs, rather than examining outputs (NASULGC, 1997). The formation of the Task Force was a response to the ACRL Board's perception that the association has no statement on outcomes assessment, and that its standards, largely written as input measures, are out of step with the practices and philosophy of regional and professional accrediting agencies and state higher education agencies. Appointment of this Task Force was approved by the ACRL Board of Directors at its 1996 Annual Meeting in New York (Board Document 11.0, ACRL Board of Directors, June 1996.) The issue of standards and higher education accrediting agencies is not a new one for ACRL. An Initiatives Fund request came from the University Libraries Section in the Spring of 1994 as part of the routine programmatic funding of ACRL section activities. The proposal was "to conduct a study of how university library standards are used by accrediting agencies." (Memo from Althea Jenkins to Susan Campbell et al., July 22, 1994.) This request was considered at the Spring 1996 meeting of the Executive Committee, which recommended the original proposal be expanded to cover all types of libraries. As a result the ACRL Board at its summer 1994 meeting approved the formation of a Standards Study Task Force. This body was chaired by Edward D. Garten, University of Dayton and was charged to develop "a plan for looking at how accreditation agencies view libraries and in particular how they use the ACRL Standards as part of their accreditation process." (Task Force Report, May 30, 1996.) Continuing, the Task Force Report says: "The plan was to identify methodology, costs and time line for such a study and was to be submitted to the Board at Annual Conference 1996." The Standards Study Task Force reached several conclusions (Task Force Report, May 30, 1996):
Based on these conclusions the Task Force felt that its charge was inappropriate and instead made two recommendations:
With this as background the Board noted that the ACRL Standards had remained largely focused on inputs and were out of step with the regional and professional accrediting agencies' philosophies and standards. The Board also noted its strategic plan called for ACRL to be more influential in the environment of higher education associations. 1995/96 ACRL President Patricia Breivik had been deeply involved in working with regional accrediting agencies regarding information literacy, and had personally observed how the lack of an outcomes assessment position left ACRL in a difficult position. She therefore drafted a motion to create and charge the current Task Force on Academic Library Outcomes Assessment. Although the new Task Force was initially also called a Standards Task Force (due to a confusion between two separate Board agenda items unfortunately given the same document number), the charge was much broader than standards and the name was soon corrected to the present Task Force on Academic Library Outcomes Assessment. The Task Force has been given three charges:
The Task Force has held meetings at three ALA conferences and between meetings has had access to the works listed in the bibliography in section VI of this report for study and research. The first of these meetings was designed to help the Task Force better understand the accreditation process and its use of outcomes measures. The Task Force held a forum at the Midwinter '97 meeting in Washington with Larry Braskamp, Executive Director of the Council on Higher Education Accreditation (CHEA) and Barbara Brittingham, who is active in both the National Council for Accreditation of Teacher Education (NCATE) and the New England Association of Schools and Colleges. At the ALA annual conference in San Francisco '97 the Task Force discussed the shape of the report as well as key issues and perspectives to be included. Task Force members were assigned responsibilities for drafting sections of the report. At Midwinter '98 in New Orleans the Task Force held a public forum attended by some 28 people at which many aspects of the issue were discussed. The Task Force held additional meetings to discuss drafts of the report and made plans for hearings at ALA Summer Conference '98 in Washington, D.C. The Task Force is now ready to present its findings and recommendations to the Association. The apparent conflict between input standards on the one hand and the trends towards greater attention to outcomes as a method of assessment has been a profession-wide concern as evidenced by the many articles in the literature. (See bibliography in section VI of this report.) However, none of the literature has developed the kind of philosophical statement about input standards and outcome assessment that creates a new synthetic paradigm or clearly articulates a relationship between ACRL's current Standards, and the outcomes assessment approach used by regional accrediting agencies. Even though ACRL's Performance Measures for Academic Libraries Project (VanHouse, 1990) led to useful methodologies of output measures, this did not have any impact on the revision of the College Standards in 1995. This Task Force report therefore includes suggestions for incorporating outcomes assessment into ACRL standards, as well as for using them in other contexts. II. DefinitionsThe usefulness of this report is contingent on the clarity of the terminology it employs. "Outcomes," "inputs," and "standards" are key terms in this document. They are, in fact, frequently used in any discussion of library assessment. The meanings commonly attached to them are sufficiently varied, however, to warrant preferred definitions for the purposes of this report. Outcomes, as viewed by the Task Force are the ways in which library users are changed as a result of their contact with the library's resources and programs. Satisfaction on the part of a user is an outcome. So is dissatisfaction. The Task Force considers simple satisfaction a facile outcome, however, too often unrelated to more substantial outcomes that hew more closely to the missions of libraries and the institutions they serve. The important outcomes of an academic library program involve the answers to questions like these:
Questions like these are difficult to answer. That is to say, empirically rigorous measurement of academic library outcomes is hard to do. This Task Force firmly posits, however, that it is changes in library users such as the ones addressed in these questions that comprise the outcomes with which academic librarians should be concerned. It may be that these outcomes cannot be demonstrated rigorously, or in a short period of time, or even by very many institutions. The Task Force believes that they can be measured, however, and their relationship to resource inputs and program inputs can be meaningfully determined, through careful and lengthy research. However, the work of Cameron and others has shown that, for the purposes of accrediting agencies and state education authorities' accountability requirements, such rigor is not necessary to produce meaningful results. Nevertheless, it is the considered opinion of the Task Force that information resulting from such research would allow the crafting of the next generation of standards for academic libraries. A distinction is to be made between "outcomes" and "outputs." "Outputs" serve to quantify the work done, i.e., number of books circulated, number of reference questions answered. They are valuable measures for making decisions about staffing levels, setting library hours of operation, and so forth. However, they do not relate these factors to the overall effectiveness of the library in affecting user outcomes. It is important to track the library's outputs, but insufficient for assessing outcomes. Inputs are generally regarded as the raw materials of a library program--the money, space, collection, equipment, and staff out of which a program can arise. As such, the measurement of inputs, or the specification of quantities of them by standards, is viewed by some as a primitive, or at leastinsufficient, way of handling library assessment. A common argument is that it is not so important to determine how much is put into a library in terms of collection and staff, but, rather, what is done with those inputs in terms of, say, faculty/librarian collaboration or bibliographic instruction programs. Some observers are even inclined to term these latter activities the "outputs" or "outcomes" of a library program. This task force does not agree with that interpretation. Instead, the Task Force holds that all raw materials and activities alike, are "inputs." A distinction may usefully be made between a "resource input," such as a database subscription or a staff member, and a "program input," such as a bibliographic instruction session or any interlibrary loan service. It is not so useful, however, to automatically assign a higher level of importance to one category of input or the other. A large collection of books may contain many items that go unread, and are essentially worthless. On the other hand, an ineffective bibliographic instruction project is also of little value, regardless of the number of students it involves. Carrying the argument further, a thoughtfully-conducted expansion of a staff's size may result in a more effective collection and a better bibliographic instruction program, and a budget increase might make it all possible. In sum, the purpose of all inputs--whether of the resource or program variety--is to achieve outcomes. The term standards is used to refer to formal documents, such as the ACRL's Standards for College Libraries, or to refer to the actual or proposed provisions of those documents. It is particularly when new standards are being considered that questions arise regarding the nature of standards. Considerable guidance in the formulation of standards is provided by the definition adopted by ACRL: "Standards are policies which describe shared values and principles of performance for a library serving a Carnegie-classified institution. In order for a document to be classified as a standard it must:
This Task Force accepts the ACRL definition of "standards," and interprets it to mean that standards should directly address the quantity, quality, extent, and level of suitability of programs, services (which include the availability, in a variety of formats, of a collection), and staffing in academic libraries. The Task Force also holds that standards should be reviewed regularly to ensure that the programs, services, and staffing practices they treat are germane to the current state of the profession. The Task Force further believes that standards should be based on evidence of normative practice or (and on this point it goes beyond the ACRL definition) programmatic success as determined by the measurement of outcomes. III. Principles for Applying Outcomes Assessment to Academic LibrariesOutcomes assessment is an integral part of the institutional effectiveness cycle of planning, implementation, assessment, and improvement of the plan. While libraries have traditionally maintained myriad output data (e.g., number of books circulated, number of items cataloged, number of reference questions answered), these data alone do not demonstrate whether the library's mission is being accomplished, and with what degree of quality. The purpose of outcomes assessment of academic libraries is to measure their quality and effectiveness, focusing on an organizational analysis of the library as a whole, and of its constituent activities and services, and the contributions they make to accomplishing the purposes of the university or college of which it is a part. It follows from this purpose that outcomes assessment must begin with an analysis of the parent organization's mission, goals, and objectives, and an identification of the elements of them that the library supports. Assessments should include all such elements: learning/teaching, research, service, administrative processes, and whatever other purposes the institution pursues. All constituencies the library serves should be included: students, faculty, staff, alumni, and possibly other external constituencies. There can be no one set of universally standard desirable outcomes for all academic libraries, as the outcomes must depend on the institution's nature and mission. However, the processes of assessment can be generalized among institutions. Assessments should be client centered rather than institution centered; that is, they should assess changes in the library user resulting from library services or resources. However, outcomes should be related back to inputs wherever possible, in order to identify and establish "best practices." For example, if an institutional purpose is to confer doctoral degrees, the library's effectiveness in contributing to high quality dissertation research could be shown to be related to the size and quality of its collections in the relevant disciplines, or to the efficiency of its interlibrary loan service. Assessments of library outcomes should be integrated with the campus' overall assessment efforts. If there is a campus institutional effectiveness officer, s/he can work with the library to coordinate efforts between units. For example, if the institution conducts exit interviews or surveys, questions about library use and satisfaction can be incorporated into such existing assessments. However, if there is no institution-wide assessment program, the library can nevertheless conduct its own stand-alone program. Outcomes assessments should include a variety of methodologies so that conclusions may be corroborated. For example, Cameron (1978) demonstrated that a survey of administrators' perceptions of organizational effectiveness, compared with verifiable data (e.g., graduation rates) showed that the perceptions tended to be accurate. Therefore, properly conducted surveys of perceptions can accurately serve as proxies for analytical data. In general, outcomes assessments of other parts of the university include both perceptual surveys (e.g., student satisfaction) and objective data (e.g., retention rates). Assessment of outcomes need not address every possible aspect of each library service. Cameron and others recommend selecting a small number, no more than half a dozen, of key outcomes for the area to be assessed, with relevant criteria of quality. The assessment measures themselves need not demonstrate scientific rigor, but should be easily administered and reasonably reliable. Because assessment is strongly linked to planning, both in accreditation requirements and in practical usefulness, assessments should be designed to provide information that can be used to improve services. Finally, outcomes assessment should be a continuous, adaptive process. IV. Samples of Outcomes AssessmentThe literature search conducted for this report revealed that there has already been considerable work done by librarians in this area. Pritchard's (1996) literature review "Determining quality in academic libraries," identifies over seventy citations to writings on the subject, although by no means a majority of them concerned specifically with outcomes assessment. Specifically germane is the work of Bonnie Gratch Lindauer (1997), which will be used (with permission) to provide an example. In the following chart, the standard process of outcomes assessment is illustrated: 1) determination of desired outcomes, in this case following from the objectives of a specific course; 2) establishment of indicators that the outcomes have been met; and 3) methods of collecting the data that reveal whether the outcome was achieved. Selected Information Literacy Outcomes
The forgoing examples are of very small, specific outcomes, as they relate to a specific course in library use. Assessing outcomes for entire library programs will of necessity be much more global and general. Nevertheless, the foregoing exemplifies the general methodology of focusing on the user and the changes in her/his knowledge, abilities, and actions occasioned by interaction with the library's collections, services, and programs. Following [again from Lindauer (1997)] is an example of outcomes assessment for promotion of a program of library literacy. It demonstrates an assessment of work with internal clients, the teaching faculty.
In the foregoing example, the desired changes are in the skills and attitudes of the faculty, not their students, and shows that it is possible for libraries to apply this regimen to all segments of their clientele, not confined to student outcomes. V. Implications for ACRLThe Task Force envisions next steps in several areas that might be undertaken by the Association to promote the development and use of outcomes assessment for academic libraries. They fall under the general rubrics of policy, accreditation, and standards. A) Policy B) Accreditation C) Standards and Guidelines Guidelines are not of the same nature as standards, and it seems that not all of them are amenable to incorporating outcomes, e.g., "Guidelines Regarding Thefts in Libraries." The ACRL Standards and Accreditation Committee (SAC) will work with each committee charged with revising standards and guidelines to determine which ones should be recast to incorporate outcomes assessment. SAC will also be responsible for re-establishing the cycle of standards and guidelines review and revision (list appended). In general, SAC should be delegated the necessary authority to implement the Board's philosophy on the content of standards, not just to review their format and technical aspects. Appointees to this committee should have demonstrated experience with standards writing or application, or with work on accreditation issues or visiting committees. The ACRL Board itself should be more exacting in requiring the standards and guidelines that bear the Association's name to meet its requirements and expectations for outcomes assessment as well as other established requirements. The timeline for implementing these recommendations is best set by SAC, which will be mainly responsible for implementation. VI. Bibliography
APPENDICES
Appendix A Ch., Jill B. Fatzer
Appendix B
This process should be developed in conversation with appropriate ACRL units in terms of their undertaking in this area and their ability to respond in a timely fashion. The outcomes of the task force's work---once approved by the Board---would then be referred to those existing ACRL units or, if need be, to a group established to provide direction for their efforts. The task force should consist of 5 members, one member each from the following sections: CLS, CJCLS, ULS, and ECLSS; one member-at-large and one Board member. The chair of the Standards Committee should be ex-officio. Appendix C WHEREAS the ACRL strategic plan emphasizes the importance of our association influencing the larger higher education environment, and WHEREAS organizations prominent in that environment, such as accrediting agencies, higher education associations, and state boards of higher education have adopted the assessment of educational outcomes as their principal indicator of achievement and quality, and WHEREAS cooperative projects between ACRL and such agencies show promise of producing new avenues for improving our libraries' support and status on their campuses, and WHEREAS, such projects would be furthered by ACRL's policies and practices becoming congruent with national assessment trends, NOW THEREFORE BE IT RESOLVED that ACRL adopt the following policy: Appendix D
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ACRL is a division of the American Library Association |
| © 2008 American Library Association. Copyright Statement Last Revised: May 21, 2007 |