1. Introduction
King’s College London is a research university based on half a dozen campuses spread across Central London. It has around 13,000 students, six main subject divisions (Schools), and an annual budget of $200 million.
As a multi-site university, with the need for multiplication of materials and services, we are heavily dependent on our network. We see the network providing an opportunity for substantial cost savings, both in reduction of duplicated resources, and in reduction of inter-site travelling.
In a series of discussions held during April 1997, the project team at King’s have identified broad areas in which to apply assessment techniques, a project personnel structure, and a series of goals and perceived benefits.
At this stage, details of the programme remain to be finalised.
2. Programme
2.1 Overview
In the initial project bid submitted by King’s, 3 main areas of interest were identified:
- Electronic journals
- King’s participates in a national site-licencing initiative for around 1,200 electronic journals, and we need to know how to target training and support resources.
- Electronic mail
- Following a recent transition to a new email system, on which all new incoming staff and students are automatically registered, we need to know more about the role of e-mail in the College, and how to target training and support, for both new and transiting users.
- Electronic vs printed information
- We are interested in comparing the effectiveness of electronic help and other information sources with the traditional paper-based equivalents. In addition, we need to justify the resources required in order to support both media during a transition.
In addition to these 3 areas, which would be subject to in-depth records, surveying and analysis, it was felt that the project would provide an opportunity to initiate ongoing, broader assessments of the performance/costs associated with the full range of networks and network services at King’s.
2.2 Outline
2.2.1 Personnel
We have identified 3 groups of person resources necessary for the project:
- Steering group, comprising the project team listed in the original bid. The group’s remit is to meet regularly and review progress, obstacles, resource requirements, and to handle project publicity.
- Implementation group, comprising members of the Library and Computing Centre. The group’s remit is to meet regularly, devise implementation details, ensure that the project progresses to schedule.
- User advisory group, comprising representatives from each of the academic disciplines, together with the King’s CNI liaison contact.
2.2.2 Assessment
We propose to explore the measures listed below, with a view to implementation from June 1997. We intend to make full use of automation wherever possible, chiefly because it makes optimum use of limited staff resources. In particular, we believe that WWW-based questionnaires and system usage log analyses will provide much of the data we require.
During the course of this project, we are not intending to achieve very fine granularity in the measures, or to encapsulate every possible instance or eventuality. By embarking upon a coarse-grained assessment in the first instance, we are field-testing both CNI assessment procedures and an ongoing programme of assessment at King’s. For the latter, measures may be refined in the light of experience, with the intention of launching a full production assessment service in 1998.
In general, “quantitative” refers to numeric data (system logs, budgetary data), and “qualitative” refers to survey, questionnaire, interview, or focus groups data. Quantitative measures tell us about Extensiveness (extent of use), Efficiency, and Economy; qualitative measures tell us about Excellence (service quality), Effectiveness (impact), and Usefulness.
“Costs” refer to hardware, software, and staff for all projects; and additionally, printed material, service subscriptions where appropriate.
- Electronic journals
- a) Quantitative
- usage
- costs -> costs per transaction/user
- usage profiles: per day/week/month, per journal, per department
- system availability
- b) Qualitative
- reasons for use
- user satisfaction
- ease of administration
- a) Quantitative
- Electronic mail
- a) Quantitative
- usage: incoming/outgoing, internal vs external
- costs -> costs per transaction/user
- usage profiles: per day/week/month, per department
- system availability
- failure rates
- b) Qualitative
- – reasons for use
- user satisfaction
- ease of administration
- a) Quantitative
- Electronic vs printed information
- a) Quantitative
- usage
- costs -> costs per item/user
- electronic information usage profiles: per day/week/month, per department
- document availability (system vs printed copies)
- b) Qualitative
- reasons for choice of option
- user preferences
- user satisfaction
- ease of administration
- a) Quantitative
- Ongoing quantitative analyses
- a) Services
- WWW
- Cache
- Library catalogues
- Ftp server
- Listserver
- USENET News
- Networked labs
- b) Network infrastructure
- LAN traffic (local/Internet)
- Dialup service
- a) Services
For each service, system availability
- usage profiles: per day/week/month, per department
- costs -> costs per transaction/user
Each area may be further subdivided on the advice of the service administrator, but the above measures are suggested as a basic report requirement.
3. Goals and benefits
The primary motivational requirements underpinning the project are the need to satisfy users and funding bodies that the services provided are value for money (Accountability), and to improve services where necessary (Improvement). McClure and Lopata, in their report on Assessment, suggest a number of measures designed to provide this information. Beyond a range of quantitative, automated system log analyses, it is desirable to explore other issues in a more qualitative fashion: for example,
- reasons for the use of services
- trends in different uses of services
- areas in which services may be reduced, enlarged, removed
- user satisfaction
- areas in which additional promotion may be necessary
- how to manage the transition to electronic information
- and so on
We believe that the exercise will inform the user base, help to improve services, and assist long-term strategic planning, in a field in which working practices are adapting to a new, distributed, network-based model.
4. Other relevant institutional experience
Prior to our involvement with the CNI project, both the Library and Computing Centre have undertaken various assessment exercises. The Library, in conjunction with outside consultants, is currently involved in a major evaluation of the performance of library services; this exercise, using questionnaires/focus groups, is working to identify users’ key priorities for service developments.
The Computing Centre is currently involved in a major restructuring exercise to increase user focus of computing services. It regularly conducts helpdesk user surveys, and produces a range of service traffic statistics.
One of the objectives of our involvement with the CNI programme, in addition to the institutional goals outlined above, is to reinforce and strengthen the ongoing assessment work. We also welcome the opportunity to bring together and standardise a number of disparate exercises, to meet others involved in similar projects through CNI and elsewhere, and to share our experiences.
Questionnaire
- CNI Draft Questionnaire
- Topic: Qualitative Analyses of Electronic Journal use