HelpfulMed: Intelligent searching for medical information over the internet

Medical professionals and researchers need information from reputable sources to accomplish their work. Unfortunately, the Web has a large number of documents that are irrelevant to their work, even those documents that purport to be “medically‐related.” This paper describes an architecture designed to integrate advanced searching and indexing algorithms, an automatic thesaurus, or “concept space,” and Kohonen‐based Self‐Organizing Map (SOM) technologies to provide searchers with fine‐grained results. Initial results indicate that these systems provide complementary retrieval functionalities. HelpfulMed not only allows users to search Web pages and other online databases, but also allows them to build searches through the use of an automatic thesaurus and browse a graphical display of medical‐related topics. Evaluation results for each of the different components are included. Our spidering algorithm outperformed both breadth‐first search and PageRank spiders on a test collection of 100,000 Web pages. The automatically generated thesaurus performed as well as both MeSH and UMLS—systems which require human mediation for currency. Lastly, a variant of the Kohonen SOM was comparable to MeSH terms in perceived cluster precision and significantly better at perceived cluster recall.