SUBSCRIBE NOW!

Vol. 27 No. 5 — Sep/Oct 2003

Nonetheless, information on its own is neither inherently good nor bad. It is often a sequence of events that leads to the consequences of what we simplistically refer to as bad information. One striking example is the Johns Hopkins clinical trials case, in which an insufficient search in PubMed resulted in a death [1]. It is relatively rare to be able to track a chain of events so directly to poor quality information capture, dissemination, or retrieval. However, there are a few examples when blame has been contested in the courts, with varying results [2]. In one case, Dun & Bradstreet was found liable for having inaccurately reported that Greenmoss Builders had filed for voluntary bankruptcy [3], while another case in Germany against a medical publisher concerned a missing decimal point (advice to "inject a 2.5% NaCl infusion" appeared as a "25% NaCl infusion") [4]. If these cases prove anything, it is that the "big, bad Internet" is not behind every instance of poor quality information: The problem predates that platform and goes much deeper.

Before considering any solutions to the problems presented to information professionals by imperfect information, we need to understand the nature of the problem—or, more accurately, multiple problems—at the heart of "when information is wrong." Only then can we make progress towards modifying our processes in order to cope better in future.

DEFINING "WRONG" IN WRONG INFORMATION

It is vital to realize that each point in the information dissemination chain is equally prone to breakage. The fault, if it can be isolated at all, can lie anywhere along the complex processes of publication, collection, storage, dissemination, retrieval, or utilization. Blaming the quality of the source data may be temptingly easy, but simplistic. The Johns Hopkins case was due to searcher shortcomings, and, to some extent, the dissemination platform. Decisions were taken not because the relevant information did not exist, or was wrong, but because it was not found. Hence, the more we can strengthen each individual link in the information chain, the greater the chance of ensuring successful transfer of the right information.

I can perceive at least five distinct ways in which information can be wrong.

Inappropriate Quality

During the 1980s and '90s, when Total Quality Management (TQM) was all the rage, many and various definitions of "quality" were put forward. One of the most succinct is simply that quality is "fitness for purpose," in other words, the output matches the specification or requirements of the putative user. It thus becomes possible to speak of a "quality" Trabant car as well as a "quality" Rolls Royce. The issue is not where the vehicle fits on some hypothetical scale of reliability or comfort, but whether it meets the reasonable expectations of the user and is built in accordance with a set of criteria known to both producer and consumer.

In information terms, this means that information items geared towards one set of consumers may be perceived as poor quality when located by a different set. For example, most teenage mathematics students know that the quantity pi has been defined to a million decimal places, but the home handyman will still be happy to use the approximation of 22/7 to calculate most circular perimeters. Likewise, my theoretical chemistry tutor spent much of his professional life working on the problem of whether benzene has six equal ring bonds or three short and three long ones—yet in organic tutorials, we were not criticized for drawing a regular hexagon to represent the compound.

Problems start when it becomes difficult to discern the intended user of a piece of information, or when users expecting one quality level encounter information built to a different quality level. In past years, a textbook became recognized over several editions as an authoritative source by means of a process of (more or less rigorous) testing in the marketplace. Works tended to be focused on a defined user community. A classic example is the CRC Press Handbook of Chemistry and Physics, a 6-inch-thick tome of data tables that has held open many a chemistry undergrad's room door. The result of this process of targeted marketing and testing was to establish a tacit hierarchy of information quality—the experienced user came to know what was reliable and what was not.

The Internet publishing phenomenon has demolished much of this information hierarchy—anyone can now produce a "world-leading" reference Web site or an apparently authoritative Weblog. As the classic New Yorker cartoon puts it, "On the Internet, no one knows you're a dog." The information market has widened substantially; the same information is now being offered by multiple sources, increasing the risk that information that is of perfectly suitable quality for one purpose is being offered to, accessed, and used by users with a totally different intended purpose. This leaves the door open to both innocent mistakes and deliberate abuse.

Ambiguous or Deliberately Fraudulent

Writing as a chemist, it is tempting to believe that some types of information (such as physical properties) can be measured by an absolute standard, and that this standard becomes more precise as time goes on. Yet recent discussions on the impact of the United States' Data Quality Act [5] have highlighted the fallacy of this belief. As a consequence of the Act, the Environmental Protection Agency (EPA) has drafted a set of "Assessment Factors for Evaluating the Quality of Information from External Sources" [6], which attempt to establish standards for data quality. However, to illustrate the scale of the problem, a recent U.S. Geological Survey report evaluated the variation in a single data item (the octanol-water partition coefficient for certain pesticides) reported in the chemical literature. More than 700 papers published over a 55-year period showed "up to 4 orders of magnitude variation with no convergence with time" for the same data [7]. Possibly an extreme case, but hardly an inspiring starting point.

In addition to "innocent" variation in information quality as an artifact of the method of presentation, there are known examples of deliberate fraud. One of the most recent is that of J. H. Schön of Bell Laboratories who "engaged in scientific misconduct by falsifying and fabricating experimental data between 1998 and 2001" [8]. His publication record is extensive and, although we now know his data is made up, there remains an electronic trail in scientific/technical databases.

Biased or Non-Objective

Prior to the advent of the Internet, professional users assumed that information retrieved from an electronic service was a faithful reflection of their search strategy. The implementation of paid-for links on Internet search engines has demolished that assumption. We can no longer assume that the results presented to us as "the most relevant" are the best fit to our strategy.

A failure to be objective in reporting is one of the most subtle corruptions of information quality, since it is one of the most difficult to detect by anyone other than a specialist peer group. It is also one of the most difficult to correct, if it manages to enter the information chain. Consider the editorial written by Trevor Laird, editor of Organic Process Research & Development [9]. He notes that some authors in his journal appear to deliberately skew their citation listing, avoiding work by competitors in the apparent hope that reviewers will not be aware of competing or contradicting articles. Such a failure of objective reporting can perpetuate itself through the bibliographic system and result in the undermining of the basic assumptions surrounding the citation process. This can have particularly damaging effects when citation analysis is used in many research evaluation techniques and grant-awarding considerations.

Incomplete

On a visit to a film studio in California during the 1980s, Pope John-Paul II made a speech to a group of film directors and producers. This was at a time when the portrayal of violence and sex in movies was being debated as an issue touching upon the exercise of the right to freedom of speech. The Pope made the comment that "the proper exercise of the freedom of speech demands that what is communicated is complete and true." In the context of quality information, I take this to mean that the distribution of a piece of information, whether it's visual, verbal, or printed, can become just as misleading through what it fails to communicate as by what it actually attempts to show.

Consider the following examples. In the house journal of the Royal Society of Chemistry, a researcher reported that an accident had occurred in which the lid of a steel drum, used for disposal of laboratory solvents, had been blown off, apparently due to an increase in internal pressure [10]. The contents of the drum were known, as was the composition of the most recent additions. The most likely cause was considered to be a reaction between acetonitrile and ethanolamine. However, one of the points of the letter was to highlight that "the MSDS [Materials Safety Data Sheets] for ethanolamine and acetonitrile...contain some confusing information." Clearly, the author of the letter was a technically qualified person, yet the very information source that should have helped him to assess risk was not able to convey the information in a concise and technically relevant manner. One possibility is that the process of distilling information into a usable note resulted in information loss, which led to the accident.

A similar failure to ensure the communication of the whole picture can be cited from my own field of patent information. The U.S. Patent and Trademark Office (USPTO) has a well-regarded Web site covering many years' worth of U.S. patent documents. However, an unskilled user can come away from this site with a misleading picture of the patent situation. Under USPTO practice, it is possible for an applicant to apply for a Re-issue certificate (Re) if new information affecting the proper scope of their patent becomes available after the grant. If the Re-issue certificate is granted, the predecessor document is treated as withdrawn for the purpose of determining the owner's enforceable rights. However, although the later (Re) document cites its "parent," there is no link in the other direction—from the withdrawn document to its successor. Only a specific search designed to locate Re documents will reveal the true picture. By contrast, value-added commercial databases, such as IFI Claims and Derwent's WPI, assist the process by linking the two documents. For the searcher who knows what can happen, either source is acceptable—but for the nonskilled searcher, the USPTO database's structure, allied with insufficient searching knowledge, can lead to the wrong answer.

Out-of-Date

The final possibility for defining "wrong" information is that although a search strategy may be adequate, the source unbiased, and the results technically accurate, the answer may be out-of-date. This is clearly a live issue when dealing with dynamic information such as financial or business data. Knowledge of database update policy is important to assess the usefulness of that database to your particular type of search. In some instances, no harm is done by a failure to identify the most current data, as the situation will be rectified later on in the supply chain. For example, a book search may identify the second edition of a title. If the requestor places an order, the book supplier should alert the buyer that the third edition is now available. However, as more and more end users are searching directly for information, there may be no safety net of third-party intervention, and out-of-date information may be recalled and used as if it was the most current.

The best control over this eventuality is the application of searcher skill and database knowledge, although sometimes information suppliers can assist the process. During the 1980s the British Standards Institution (BSI), which publishes technical standards for manufacturing in the U.K., experimented with printing its standards in black text on red paper. This was an attempt to control photocopying and hence prevent the perpetuation of out-of-date technical standards; anyone other than the original purchaser of the standard had to refer to the BSI to obtain a new master copy. That reference process ensured that the most up-to-date edition was supplied.

FAILURE TO LEARN

Even in the best-regulated system, there's a failing that is extremely hard to make provision for—the failure to learn from experience. There are myriad examples of information users reinventing the wheel—or more commonly, making the same mistake—as a result of a failure to retrieve and utilize a comprehensive set of teaching from previous recorded literature. In one sense, the Johns Hopkins case illustrates this. The search strategy may have been right, but the fault lay in the failure to search comprehensively, back to the earliest relevant item.

The phenomenon of"literature half-life" has been recognized for many years. The frequency of citation and/or retrieval decreases in an exponential fashion over time. It becomes very easy to assume that the long "tail" of older items retrieved in a search will become less and less relevant, particularly within a fast-moving technical field, where received wisdom would state that "the relevant material can only have been published in recent years." Today, this natural tendency is exacerbated by a generation of searchers who seem to assume that all information activities started on the same day as the Internet, and that old non-electronic information is passé. This is anathema, particularly to patent searching, when the law requires a presumption that literature of any age has novelty-destroying potential. Consider two examples of"bibliographic amnesia."

• In September 2001, there was a fatal explosion at an ammonium nitrate plant owned by Atofina. In subsequent months, various letters appeared [11] speculating that the explosion was caused by the same or similar mechanism to that which occurred in the harbor of Texas City in April 1947, involving the same chemical. One of the letters noted that a possible mechanistic explanation of the Texas City explosion had been in the literature since 1960.

• There has been much interest recently in so-called "economy-class syndrome," a form of deep-vein thrombosis (DVT) apparently linked to sitting for long periods in cramped conditions, such as on long-haul aircraft flights. A letter [12] noted the existence of a "travel advisory" strikingly similar in content to the current airline "well-being programs," such as taking periodic breaks to move around and exercising limbs in a sitting position. The author concerned had been writing about travel by stagecoach in 1789.

THE PROCESS OF DISSEMINATION

In addition to the above two factors—"wrong" information being retrieved and a failure to learn from what could be retrieved—there is a further risk: the perpetuation of incorrect data. This is an acute problem now that electronic communication has vastly increased the speed at which new information can be added to the corpus of knowledge, disseminated widely to both skilled and unskilled users, reproduced, and transmitted again. The speed with which an "urban myth" becomes established is astonishing.

The damage is bad enough when something that started out as correct information goes through a process of "electronic Chinese whispers," but the wary user at least has the chance to cross-check against earlier sources, as a guard against corruption. But if the error is very early in the dissemination chain, no amount of faithful onward communication can alter it. Sometimes the error may remain in the information system for decades or even centuries. One example is the story of the clarification of the chemical structure of Perkin's mauve, the first synthetic dyestuff. It was originally synthesized in 1856 and had been assigned an incorrect chemical structure since at least 1924. It was not until 1994 that modern analysis of an original sample proved the correct structure [13].

A similar example in more recent years comes from the field of biochemistry. Authors McDonagh and Lightner wrote to Chemical & Engineering News [14] to note that the chemical structure of bilirubin had been incorrectly illustrated in an earlier article in the same periodical. The author of the original paper responded to their letter, conceding that they were correct, but observing moreover that all three major graduate-level biochemistry textbooks in the U.S. [15, 16, 17] also reported the incorrect structure in the current editions! Interestingly, in the course of verifying the latter references, I discovered a Web page entitled Howlers in General Biochemistry Textbooks [http://bip.cnrs-mrs.fr/bip10/howler.htm] that also laments the short-term influence of publishing information about errors.

Peer review in the primary literature should guard against the perpetuation of wrong data, backed up by a periodic consolidation into secondary and tertiary sources. A single paper in a reputable chemical journal would be reviewed by one or more independent and qualified peers before publication. At intervals, review articles (secondary literature) collected all the related primary papers into a more-or-less comprehensive bibliography, with or without informed comment upon their content. Over time, the teaching of individual papers would be progressively distilled into the tertiary literature of handbooks, encyclopedias and databooks, and student textbooks.

Bottle [19] presents data suggesting that the length of time between initial publication and first appearance of the same information in a high school textbook is decreasing. However, the speed of consolidation of information—with concomitant possibilities for error—is not the only concern. There appears to be a parallel reduction in the use of the entire primary-secondary-tertiary information chain. Access to primary literature is now so easy and so powerful that users are tempted to merely re-run searches against a primary source at regular intervals instead of utilizing the slower process of independent data consolidation. For example, the Beilstein Handbook of Organic Chemistry was for decades acknowledged as a slow-publishing but fanatically accurate data compendium. In the last 20 years, it has moved away from its position of strength in an effort to re-invent itself as a more rapid service, abstracting the primary literature.

The dangers of this time pressure, leading to a breakdown in the data distillation process, are twofold:

• The primary literature now contains a larger proportion of material that has not been peer-reviewed at all.

• The quality audit provided by
the secondary and tertiary services, which attempted to
place the primary literature in its proper context, has largely been swept away.

Therefore, it becomes more difficult to see primary literature in its proper context, geared towards a specific user community, generated to a specific quality standard. Open access can merely mean that it is open to misuse. The second consequence is the failure of the mechanism that has provided a useful filter at the source (identifying "citation classics" and de-emphasizing ephemeral items) and a form of bibliographic "version control," collating any corrections and revisions into a common format that ensures their subsequent retrieval.

IMPROVING THE SYSTEM

How can information science deal efficiently with correcting information in this imperfect world? What mechanisms do we have on hand that can help us cope with errors and control or prevent their perpetuation?

The simple answer, at least in relation to much of the scientific literature, seems to be, "Not many." Consider the instances of known fraud. Although individual journals have noted the "withdrawal" of the Schön papers, the bibliographic databases rarely exert any form of quality control, and references to these misleading papers remain to be retrieved in future years, alongside all the legitimate ones. There is no "health warning" attached to the records. There is a fair likelihood that their very notoriety will ensure that the Schön papers continue to be cited, and probably more heavily than would otherwise have happened. There is no control in the scientific citation system to distinguish between a critical citation and an approving one. Hence the presence of these papers in the scientific literature system is perpetuated. [Editor's note: It's distressing that not much has changed since I wrote about problems with identifying corrected articles. See Ojala, Marydee, "Oops! Retractions, Corrections, and Amplifications in Online Environments." Searcher, vol. 4, no. 1 (January 1996): pp. 30-41.]

There are a few examples of more formal correction and quality control mechanisms, and it is worth considering them in turn, for the extent to which they are currently used and the possibility of expanding their application.

Publishing Retractions and Errata

Many established journals publish retractions or errata notices. For example, the publishers of Online Information Review in volume 26 number 2 published a retraction of certain remarks made by one of their columnists in volume 25 number 4. The later notice appeared several issues after the original article. This might have been some help to a regular reader of the paper journal, but is next to useless for electronic retrieval. Searchers are most unlikely to be alerted to the existence of a retraction notice when they locate the original article, since no link is made to the later correction.

The hazard for the searcher remains; unless they know, or have reason to suspect, in advance that a correction or retraction has been issued, it does not normally fall out in the standard search process. Surely it cannot be right that modern retrieval mechanisms for a later item which corrects an acknowledged mistake lag so far behind those which serve to locate the erroneous item? This would seem to be exactly what the hyperlink was invented for—but journal publishers and secondary service providers do not so far seem to have been applying it for this purpose.

Capturing Ephemera

Many of the same remarks on the issue of published retractions also apply to a situation where a first author writes a full article, which is subsequently commented upon by a second author through the letter pages of the same journal. Not all bibliographic databases cover the letters pages, even in respected journals, so even if the later author cites the earlier article (which is not guaranteed), there is a risk that the later comments will be lost to the searcher. Yet these pages can contain not only valuable comment upon papers, but also original items. The letter pages of Chemistry in Britain, the house journal of the U.K. Royal Society of Chemistry, often contain items along the lines of"I tried out the synthesis method of Smith et al. and it blew up." It is notable that one of the frequent respondents to these letters is the editor of Bretherick's Handbook of Reactive Chemical Hazards, which suggests that the publishers have a regular policy of monitoring these informal reports.

Managing Expectations

In light of my comments on data quality, it is clear that one hazard is the same piece of information being used by many different communities. What is "quality" to one is not to another. It is worth noting the efforts that database producers have put into trying to tackle this problem. The INSPEC bibliographic database of engineering literature, produced by the Institution of Electrical Engineers (IEE) in the U.K., commendably devised a system of Treatment Codes, applied to each entry in the database. The intent is to expand the normal Document Type code to include an explicit indication of the "expertise level" of each record. In a similar fashion, the huge esp@cenet Web server of patent information is intended as a "first-cut" source for small businesses and provides only limited search possibilities and limited help in interpreting the output. At the urging of the professional user community, the service now comes with a "health warning" that the output from searches on this site cannot be considered authoritative on questions of patentability.

Publishing Explicit Correction Documents

Returning to my own specialty of patents, there is an interesting example of a policy for dealing with corrections. After many years of negotiation, the World Intellectual Property Organisation (WIPO) promulgated its bibliographic standard ST.50 in 1998. The full text is available at the WIPO Web site [19]. The purpose of ST.50 is to establish a recommended mechanism whereby patent offices ensure that corrections are published. Corrections have to be notified in a way that ensures they are retrieved by any search that would have retrieved the original erroneous document. The principal mechanism for this ensures that the document identifiers (specifically the publication number) of both the original and correction document are substantially identical, with merely a modified suffix to denote the point in the information chain to which the document belongs. For example, published patent application EP 1266661 A1 is superseded by a reprinted version EP 1266661 A9. All other bibliographic data (applicant name, inventors' names, classification) are identical between the two versions (unless of course it is an error in one of these fields that the correction document is replacing).

The details of ST.50 allow for the possibility of different errors entering the information chain at different points in the dissemination process—for example, the paper document may be correct but its corresponding facsimile edition on CD-ROM may be corrupted. Specific document codes enable the user to identify the nature of the correction made, and hence to determine which is the "authority" version appropriate to the search.

It cannot be claimed that the system is perfect, especially as (with all WIPO standards) it is advisory rather than mandatory and thus far has been adopted by relatively few patent offices. However, it is being applied to European Patent Office documents, and correction documents are now entering the bibliographic system through Web databases such as esp@cenet and conventional online files such as Derwent's World Patent Index (WPI). I suggest that it would be instructive for major database producers to examine the principles and operation of the standard, with a view to seeing whether it could be applied to other forms of literature.

Utilizing the Tertiary Literature

The example of Perkin's mauve is striking not only in the longevity of the error, but also in the argument that it makes for intelligent use of a literature distillation process. In all the years when the primary literature was perpetuating the error, the Beilstein Handbook remained uncommitted as to the structure of mauveine. At the present time, there seems to be a consistent view that the "compendium" approach to information is slow and expensive; it is cheaper and easier simply to re-search and re-collate the primary literature. I believe that this attitude overlooks the very valuable contribution made by compilations—that of applying a second set of eyes to the data, without time pressure, to produce a considered opinion on the veracity of each piece of primary information. It is true that the process is labor-intensive and slow, but the question is ultimately one of information quality—how important to you is the answer?

DO YOU WANT IT NOW OR DO YOU WANT IT TO BE CORRECT?

Today every searcher, be they expert or part-time, is expected to produce results at high speed with a high degree of accuracy. I believe that part of the role of the information professional is to ask this pertinent question: "Do you want an incorrect answer quickly or a correct one more slowly?" This is not a Luddite argument against technology—it is an argument for quality, in the full meaning of the word. There will be times when we need an answer, any answer, and serve our user communities best by providing an adequate response in a timely fashion. But equally, there will be times when important commercial consequences hang upon information provision, when we may serve our users better by maintaining professional standards and a rigorous approach to information retrieval. If this means that we have to ask them to wait while we check before delivering an answer, then so be it.

How Thomson ISI Handles Corrected and Retracted Articles Thomson ISI has, for all practical purposes, cornered the market on citation reference searching. How does the company handle corrections to articles in its databases? If either the publisher of the article or Thomson ISI makes an error in any of the bibliographic elements of an article, Thomson ISI will correct the data. No general notice is given for this type of correction. If an individual researcher notifies Thomson ISI of an error in the bibliographic elements of an article or citation referring to their work, Thomson ISI will correct the data and notify the researcher that the correction has been made. If the publisher corrects the content of a previously published article and publishes this subsequent correction, Thomson ISI indexes the new, corrected item, including the original volume, page, and year of publication, and lists the document type as a "correction" in the Web of Science. When a publisher retracts a previously published article and publishes the retraction, Thomson ISI indexes it in the same manner, assigns a document type of "correction," but also includes the word "retraction" in the article title. Thomson ISI also modifies the title of the original article record to include the statement: "(Retracted article. See vol X, pg XXX, YYYY)." As Thomson ISI evaluates journals, the overall quality of the science included is determined by a number of factors, including the application of peer review by the publisher, the prestige of the sponsoring organization if present, and, to some extent, the reputation of the publisher. Obviously, it would be impossible to replicate research from millions of articles as part of this process. Citations to published articles may occur for either positive or negative reasons. Thomson ISI records all citations without further qualification.

REFERENCES

[1] Perkins, Eva, "Johns Hopkins Tragedy: Could Librarians Have Prevented a Death?" [www.infotoday.com/newsbreaks/nb010806-1.htm]

[2] Denis, S. and Y. Poullet, "Questions of Liability in the Provision of Information Services." EUSIDIC Newsletter (Newsidic) No. 100 (Apr. 1990): pp. 7-19.

[3] Dun & Bradstreet versus Greenmoss Builders. 472 US 749 (1985).

[4] Bundesgerichthof, Neue Juristische Wochenschrift (1970), 1963.

[5] U.S. Public Law No. 106-554

[6] Hogue, C., "Assessing Data for Quality." Chemical & Engineering News (February 10, 2003): pp. 21-22.

[7] Pontolillo, J. and R. P. Eganhouse, "The Search for Reliable Aqueous Solubility (Sw) and Octanol-Water Partition Coefficient (K_ow) Data for Hydrophobic Organic Compounds: DDT and DDE as a Case Study." Water-Resources Investigations Report No. 01-4201, U.S. Geological Survey (2001).

[8] Available online at http://www.lucent.com/news_events/researchreview.html

[9] Laird, T., "The Importance of Adequate and Accurate References." Organic Process Research & Development [http://dx.doi.org/10.1021/op030004z].

[10] Letters to the Editor. Chemistry in Britain, vol. 38, no. 7 (July 2002): p. 20

[11] Letters to the Editor. Chemistry in Britain, vol. 38, no. 2 (February 2002): p. 20 and vol. 38 no. 4 (April 2002): p. 22.

[12] Letters to the Editor. Chemistry in Britain, vol. 38, no. 4 (April 2002): p. 24.

[13] Meth-Cohn, O. and A. S. Travis, "The Mauveine Mystery." Chemistry in Britain, vol. 31, no. 7 (July1995): pp. 547-549.

[14] McDonagh, A. F. and D. A. Lightner, "Attention to Stereochemistry." Chemical & Engineering News (February 3, 2003): p. 2.

[15] Voet, D. and J. Voet, Biochemistry. New York: Wiley, 1995 (2nd edition).

[16] Berg, J., J. Tymoczko, and L. Stryer, Biochemistry. New York: Freeman, 2002 (5th edition).

[17] Lehninger, A. L., D. L. Nelson, and M. M. Cox, Principles of Biochemistry. New York: Worth, 1993 (2nd edition).

[18] Bottle, R. T., "Changes in the Communication of Chemical Information. II: An Updated Model." Journal of Information Science, vol. 6 (1983): pp. 109-113.

[19] Corrections, Alterations and Supplements Relating to Patent Information. WIPO Standard ST.50 (1998) [www.wipo.int/scit/en/standards/pdf/st_50.pdf].

Comments? E-mail letters to the editor to marydee@xmission.com.