Searcher
Vol. 10 No. 7 July/August 2002
FEATURE
Beyond the Information Audit:
Checking the Health of an Organization's Information System 
by Chris Dobson President, F1 Services Inc.
Table of Contents Previous Issues Subscribe Now! ITI Home
Information audit, needs assessment, knowledge inventory all are names for a process that examines the secret life of information within an organization. Where does the information come from? Where does it go? How is it used? How transformed? Who keeps it? Who shares it? Perhaps most importantly, the process may identify the information that gets away, leaving workers and decision-makers to rely on hunches and the way things have been done in the past. An information audit can also reveal time and money wasted on information sources that no one uses and show why no one uses them.

The problem with all the names for the process is that they're too narrow. Audit implies an analysis of what's been reported; a needs assessment looks at only the first half of information's organizational life; and inventory assumes all needs are met. What we'll talk about here is an information checkup. Compare it to an annual physical. How is the organization feeling? Are there symptoms that indicate a problem in one of the information systems? What does a complete examination of the organization's information show? Can tests discover maladies that haven't caused problems yet? Like a medical exam, a periodic information checkup can provide benchmarks that can serve as early indicators of illness. "Hmmm, visitors to the library Web site have increased, but article retrieval from one service is down by 20 percent."

Just as medical exams vary according to the age and physical condition of the patient, an information checkup should be customized to the organization. The focus changes depending on the goal of the checkup and the status of information services. If you're setting up a new library, you want to pinpoint needed resources, discover gatekeepers (those people who can answer nearly any question from their personal knowledge or individual library), and find out how the organization will use information once you provide it. For existing information services, you should focus on awareness, relevance, usability, and unmet needs. If you're developing a knowledge management system, you'll need additional data on how information moves within the organization.

Just what can an information checkup do for an established corporate library? You already know what people need. You have the statistics to justify your budget. Your people know that they can't get truly reliable information only from the Web and that they shouldn't waste time trying. That may all be true today, but what about next week? Let's suppose one of these scenarios occurs:

  • You get a new boss who never worked for a company with a library.

  •  
  • You get a new boss from a department that has never used the library.

  •  
  • A vendor sells management on desktop delivery of information so easy it doesn't require an information professional.

  •  
  • The company hires a new batch of MBAs/Ph.D.s used to doing their own research.

  •  
  • The division that encompasses your primary customer base is sold.

  •  
  • Management decrees a 15 percent across-the-board budget cut.

  •  
  • The IT department gets the OK for a knowledge management system that will provide "all the information anyone needs."
Getting Started
Do you really know all that your customers need? Do you deliver information the best possible way? Do you serve the right people? What evidence do you have to prove your claims? It's time for an information checkup.

The first step is to define your goal for the checkup:

  • discovering why some usage statistics have dropped

  •  
  • guidance for collection development

  •  
  • new policy emphasizing electronic resources

  •  
  • concern over why your marketing efforts don't produce more business

  •  
  • input for a strategic plan

  •  
  • support and ammunition for upcoming budget battles

  •  
  • establishing a benchmark to measure against in the future
Although you may want to, you can't get the answer to every question in your first checkup. Checkups take time, not just yours, but that of users who have to answer survey questions. Write down the questions you want answered and then rank them. Make sure you ask "big" questions. You don't want to ask, "Should we get ProQuest?" You want to ask, "What types of information do workers need?" and "What characteristics make electronic resources easy for employees to use?" or "What kinds of research should employees do themselves and what kinds should be handled by library staff?"

Once you have your goals clearly defined, you can approach your manager. Obtaining management support for the information checkup is essential. Chances are you will need to hire a consultant and will need budget approval for the expenditure. More importantly, however, the checkup will require cooperation and at least minimal time commitment from lots of very busy people. An e-mail from the CEO or a vice president will do much to encourage that cooperation.

How do you obtain that level of support? Put your goals in terms management understands. "Guidance for collection development" may become "a study to insure that resources are expended for the areas of greatest need and potential return on investment." Use the corporate lingo. Convince your boss first and then ask him or her for help in framing your proposal. Don't be shy about asking for a consultant. Other departments in the organization probably bring in consultants regularly. Even if a consultant only confirms what you know, the consultant's opinion will carry more weight than yours. Somehow the act of paying for advice and opinions imbues them with tremendous validity.

Next, you need to identify consultants who conduct information checkups. The Special Libraries Association maintains a searchable online directory of consultants CONSULT Online [http://www.sla.org]. Currently the directory lists 43 consultants under "Audits." The Library Administration and Management Association, a division of the American Library Association, publishes an annual directory of consultants as an insert in the fall issue of Library Administration and Management magazine. Postings to electronic discussion groups will often yield responses from consultants themselves, as well as suggestions from those who have hired consultants. Once you have a list of candidates, you're ready to write the Request for Proposal (RFP).

Present background on your current library operations and a description of your organization. Include:

  • the number of employees in the library, at the site, and around the world.

  •  
  • the line of business of your organization.

  •  
  • at least a few details on how the library operates.

  •  
  • who within the organization constitutes your primary clientele.

  •  
  • why you want to conduct an information checkup now.

  •  
  • the specific goals for the checkup.

  •  
  • when you would like to begin the process.

  •  
  • when you would like to have the consultant's report.

  •  
  • when proposals must be received.

  •  
  • how you want to receive them, i.e., e-mail or postal service.
Be as explicit as possible about what you want and how you plan to use the results of the checkup. Once you have drafted your RFP, you may want to talk with someone in your purchasing department. There may be standard clauses that any RFP from your organization must include. The purchasing agent can also review your document for clarity.

You may want to include a ball-park budget. From a consultant's point of view, having a budget makes writing the proposal much simpler. We can tailor our recommendations to fit the budget. If the budget is inadequate for the specifications, we don't have to respond at all. If it seems generous, we have the freedom to expand on what we think the client really needs. If you include a budget, be sure to do some careful research on costs. In general, the larger the organization, the greater the cost. Talk to other librarians and talk to a few consultants. Most will be willing to provide at least a range of costs.

If you decide not to include a preliminary budget in your RFP, you will probably receive proposals with a wide range of prices. Take a careful look at the proposals and be sure that you understand how the consultant would approach the project. The low bidder may not do enough while the high bidder may offer more than you need. Evaluate the proposals based on whether the approach makes sense for your organization. You can always ask for a revised proposal if the budget and/or scope is off.

Before you make your selection, talk with the consultant. Make sure you get good vibes. If you can't communicate effectively with your consultant, you won't get the results you want. While the basic techniques for conducting a checkup don't change, their application to a specific situation does. Be sure that your consultant will customize their approach to fit your goals and your organization.

The Techniques
There are three basic techniques for conducting a checkup: one-on-one interviews, focus groups, and surveys. The goals of the checkup dictate which technique or combination thereof to use.

Interviews work best for evaluating information needs, staff responsiveness, and attitudes toward the library and information services in general. Interviews are the best technique for tracking the flow of information within the company for knowledge management projects. In the interview, the consultant may start with very general questions: "What do you do?" "What information do you use?" "How do you find it?" In the course of the conversation, we probe deeper, suggesting information that might be useful, asking about library services, checking the validity of client perceptions.

The specific questions and the direction of the conversation depend on the goals of the information checkup. The benefit of interviews is that they allow us to look beyond initial reactions. The employee who has all the information he needs at his fingertips may have given up on the library in the days when document delivery took 2 weeks and he never had time to wait. He may not realize that the majority of the articles he needs can be obtained online. Unless you do a terrible job, interviews are the only effective way to investigate problems. Most people will be reluctant to voice criticism in the public setting of a focus group, and a survey may gauge unhappiness but not reveal the causes. A good interviewer will extract examples of service failures so that you can determine if the problem was a one-time incident or evidence of a system failure.

Ask open-ended questions. Take comprehensive notes but do not use a tape recorder. Promise anonymity when requested. Do all you can to create an environment that will yield a candid and relaxed conversation. We are always amazed at what we learn about the inner-workings of a company. If you conduct the interviews, make sure you have time to review your notes immediately after each interview. You'll want to clarify and expand areas where you couldn't write down everything. You'll also want to make sure your notes are legible.

Even though a consultant may conduct the interviews, one important key to their success resides with you selection of those to be interviewed. You always want to include at least a few gatekeepers. These individuals may be your biggest supporters or your main competition. These are the individuals who have been with the organization long enough to know why things are done the way they are. They are often the sources others turn to for information before they call the library. They hoard information and they share it. They can give the consultant a good understanding of how the organization works and doesn't.

Other interview candidates will depend on the goals of the checkup. You may decide on a mix of library users and those you have never seen. You should include the highest level of management available. This is where support from upper management is essential. Interviews last from 30-60 minutes. Without a directive from management scheduling interviews can become extremely difficult. Although the opinions of potential and current users are important in developing your services, the attitudes of vice presidents are critical to whether you'll be able to deliver those services.

We conducted an information checkup as a first step in setting up a library for the R&D department of a medium-sized company. Every scientist we interviewed told us they wanted the librarian to conduct the research. They didn't have time to do the research themselves. Their idea of desktop information delivery was a report sent to them by e-mail. Management, unfortunately, decided they could save the salary of a research librarian by having the scientists do their own research. You will need management support to implement the changes that result from the information checkup. Including executives in the interviews can give them a stake in the outcome.

Focus groups are a good technique when the interaction of participants will generate ideas. Focus groups produce terrific results for strategic planning. If you can get one person to voice an opinion, others will react to it, criticize it, improve on it, and move the conversation in new directions. Focus groups are also excellent for evaluating existing or proposed marketing materials and techniques. They make good sounding boards for ideas for new services.

We discovered the "two-click rule" in a focus group. The librarian was investigating different ways to deliver information electronically. He had been sending a biweekly synopsis of industry news by e-mail and was considering eliminating the e-mail and posting the newsletter on the library Web site. As the discussion evolved, a consensus emerged. If the participants received an e-mail notice that a new issue of the newsletter had been posted and could click on a link in the e-mail to reach it, they would read the newsletter. If they had to go to the library site, click on the newsletter, and then click on the issue, they probably wouldn't bother. This is the kind of critical information that only a focus group can generate.

As with the interviews, your key role is in selecting participants. The ideal size for a focus group is six to 10 individuals. Too small a group doesn't create the energy needed to generate ideas. Too large a group and not everyone can participate. Usually you will need to invite a few more people than you want for the group. In general, participants in a focus group should be on the same organizational level. You don't want supervisors and those they supervise in the same group.

Review the discussion guide with the consultant to make sure that the most important issues are discussed at the beginning of the session. A focus group can usually address three to four primary topics. Most sessions last from 60-90 minutes. Select a comfortable, neutral setting. A consultant will be able to move the discussion from one topic to another without the constraints of the usual meeting dynamics of your organization. You may want to observe and take notes. Although most focus group sessions are recorded, an observer can note body language. Careful notes also help in interpreting the tape by identifying speakers. If you know the participants, your understanding of how they interact will also help in analyzing the session. The analysis of the focus group results will note both ideas on which the group achieved consensus and those supported by fewer participants.

The Survey is the checkup technique that yields quantifiable information appropriate for benchmarking, as well as the graphs so beloved by management. The most important limitation of a survey is the mental effort expended by the survey respondent. Few people will spend 45 minutes on a survey, and the level of attention given to a particular question will be far less than for an interview or focus group. Surveys work well for measuring awareness of existing services, for gathering usage statistics, and for investigating the market share of your competitors. Surveys do not work very well for evaluating the value of your services, identifying usage barriers, or discovering deficiencies in your services or collection.

Successful surveys must be short and have very simple questions. Never survey for information you can find another way. You don't need to ask people if they have checked out books, ordered documents, or requested research you already have sources of statistics for these. Your Webmaster should be able to supply statistics for hits on your site and possibly for duration. Information services vendors can provide a variety of statistics that measure use of services available to end-users. If you already know the answer to a question, or if having the answer won't change the way you do things, don't ask it.

If your consultant develops the survey, be sure that the questions use terminology recognized within your organization. Use "unsatisfactory," "satisfactory," "great" rather than 1, 2, or 3. Convert the answers to numeric values when you receive the responses. It's too easy for rushed respondents to reverse a numeric scale. Avoid asking respondents to rank things. Inevitably respondents get confused. Multiple questions, such as those on importance in the sample, can accomplish the same objective as a ranking question without confusing respondents.

Before you send out a survey, test, test, test. Ask someone who has never used the library to take the survey. Do they understand the questions? Did any terminology confuse them? Do they know how to return the survey? Paper surveys may get more attention than electronic ones, but cost more to send and process. If you use an electronic survey, be sure it works. What happens if a user skips a question? Can they go on to the next screen? Can they go back and change an answer? If they get interrupted, can they get out of the survey and go back later? At the beginning of an electronic survey you should reveal the total number of questions. Otherwise, respondents may get discouraged on the third screen and give up.

Make sure the due date is obvious and reasonable. If employees travel frequently, you will need to allow more time than if they do not. If at all possible, provide an incentive for those who respond by the deadline, such as entry into a drawing for an airline ticket, dinner at a fancy restaurant, or gift certificate. If your sales department provides incentives for top sales people, they may be able to help you out.

You can have your consultant handle the survey results or you may seek in-house assistance on sending the survey out and processing the responses. Then you and your consultant can analyze the results.

Final Diagnosis
Getting the results of an information checkup is a lot like that part of the physical where you get your clothes back on and go sit in the doctor's office. The consultant will review the findings. He or she will show you the graphs depicting the survey results, review the focus group discussions, and provide a summary of the interviews. Then the consultant will relate all this information back to your goals and the specific questions you wanted answered.

The consultant's conclusions and recommendations may surprise you.

If this happens, take time to review all the data provided. Make a list of questions and call the consultant. You may not agree with everything the consultant recommends, but you should understand the basis for the recommendations. Of course the consultant doesn't know that the IT department will never agree to make the library site the default home page. How do you know it? One of the reasons for any information checkup is to validate your assumptions about how information is really gathered and used within your organization. Don't be afraid to acknowledge that your assumptions were wrong. If they were, you have the opportunity to change your direction and make sure your services are relevant to your organization. If your assumptions were correct, you now have the evidence of that from an impartial expert. That clean bill of health should give you the confidence to take on new challenges.
 
 
Do It Yourself or Hire a Consultant?

Some aspects of the checkup you may be able to handle yourself or turn to other departments within your company for help either in whole or in part. Some critical tasks, however, are best performed by an outsider.

Interviews

Depending on the goal of your checkup, you may be able to conduct some or all of the interviews. If you only want to know what information people use or want, you may get the answers. However, if you want to know how people perceive you, your staff, your resources, and the services you provide, you'll need a consultant to ask the questions. Most people will not say anything negative directly to you. They don't want to hurt your feelings or there may be other political reasons for being reticent. A consultant can promise anonymity, is not part of the political structure, and is not affected by criticism. You will also need a consultant if you want to gather evidence of the value your library provides. A report filled with compliments that you have gathered yourself will be dismissed as self-serving. One written by a paid consultant should carry more credibility.

Focus Groups

You will obtain much more information as an observer than as a discussion leader. Your familiarity with the participants will allow you to learn much from body language and the interaction of participants. However, if the discussion may become critical of the library, participants will be more candid if you are not in the room. There may be individuals in your human resources or marketing departments who have experience in conducting focus groups. As long as these departments and individuals are well-regarded and your organization is not rife with interdepartmental rivalries, you can utilize such internal consultants.

Surveys

This is the technique where using in-house expertise makes the most sense. Your internal market research department may be able to construct an electronic survey for you and may have the software to provide sophisticated statistical analysis. You may still want to run your questions by a consultant to be sure they are unambiguous and will yield the results you seek. Your consultant can also help you interpret the results.

The most important reason for hiring a consultant to conduct the information checkup is time. Interviews take 30-60 minutes and require twice as much time for transcribing and analyzing notes. Focus group discussion guides require thoughtful development. Transcription and analysis of the session take much more time than the discussion itself. Designing and testing a good survey takes time as well. Don't you already have a full-time job?

Hiring a consultant does not mean you will not be involved in the process. Be sure your consultant has a clear understanding of your goals. Don't be afraid to share information on the individuals to be interviewed or invited to a focus group. The more the consultant understands your organization, the better he or she can probe for information and interpret responses. The consultant should be willing to listen to your suggestions. If interviews are spread over several days, the consultant should share impressions and preliminary results with you to insure that the right questions are being asked. Don't be alarmed, however, if the consultant disregards some of your corporate taboos. That's one of the reasons you bring in an outsider.
 
 
Right and Wrong Survey Questions to Measure Importance

Wrong

Please rank the following services with 1 being the most important to you:

 ____ Browsing the book collection ____ Receiving routed journal issues ____ Access to ProQuest from my computer

Right

How important to you is the ability to browse the book collection?

____ Not important ____ Somewhat important ____ Extremely important

How important to you is receiving routed journal issues?

____ Not important ____ Somewhat important ____ Extremely important

How important to you is having access to ProQuest from your computer?

____ Not important ____ Somewhat important ____ Extremely important

Speaking of Surveys...

While we're asking questions What about a career checkup?

When you wake up in the morning, do you want to go to work?

Did you learn something new this week?

In the evening, do you tell your significant other, roommate, spouse about your day?

Are you sleeping well?

Do you laugh frequently?

If you answer "no" to any of these questions, it may be time for a job change, career change, or changes to your job. None of these is easy, but do you really want to live this way?

First, determine what's wrong with your current situation. Is it your boss? If you like your work and like most of the people you work with, but are just tired of battling for your job, your budget, and recognition, you have two choices. You can wait and hope your boss gets promoted (or demoted) or you can change jobs. I know one librarian who has worked at the same company for 15 years and outlasted at least nine really horrible bosses. Along the way she's also had a few great ones. If supervising the librarian is where your company lets new managers train, you may want to stay put and hope the next boss will be better. Or, you might want to take a look around. Looking for a job is a lot easier when you already have one and you're not totally depressed and/or desperate.

Is it the work? When was the last time you enjoyed your job? What were you doing then? Did you really like being a reference librarian, but now all you do is deal with personnel problems? If you've been pushed up the ladder into management, think about climbing down. If you stay with the same organization, you will have to give up control and let someone else make the decisions. If that would be too difficult, look for another job where you can be a contributor rather than a manager.

Are you just bored? No matter how challenging your job used to be, at some point you may just need to make a change. One of the downsides of developing expertise is the ability to do your job without thinking about it. If you work in a large organization, you might be able to change jobs without leaving. Do an inventory of your skills. What are the things you can do that you like to do? What are the things you've heard about and longed to do? Talk to people in other departments. Investigate opportunities for using your skills in a new position and acquiring new skills.

No job is ever perfect but you shouldn't let one ruin your disposition or deaden your brain. We spend too much of our lives at our jobs to settle for ones that don't challenge us, reward us, and give us some fun...most of the time.

Chris Dobson's e-mail address is chris.dobson@f1servicesinc.com.

Table of Contents Previous Issues Subscribe Now! ITI Home
© 2001