An objective method of “weeding” bibliographic databases

As computerized bibliographic databases have become larger (and older, on average), storage and search costs and the costs of scanning the larger search outputs to identify the most useful records have all also become greater. A method is described for identifying periodically those database records with minimal utility to a database's clientele, so that such records can be removed to a batch‐search database, thereby reducing the costs of on‐line database storage and use and enhancing the average utility of query responses. The technique, which was proved experimentally long ago and which requires a database with records indexed using a controlled vocabulary, identifies (for retention in an on‐line database) those documents most often indexed by terms most often used in recent queries and enables a quantitative measure of overall utility to be calculated for each document. Because the distribution of numbers of records versus document utility is heavily skewed toward low utility, extraction from the database of even a large fraction of its records will still leave many records with a low value of calculated utility, thus serving as a “fail‐safe” mechanism. Means for selecting rationally and easily the “cutoff point” as to document utility (as to those records which should be retained on line) are described. Several potential serendipitous benefits of such a weeding operation are suggested. The costs of such an operation should be modest. © 1986 John Wiley & Sons, Inc.