Treffer: TREC-3 retrieval evaluation using expert network

In Mayo Clinic's first year of participation at the Text Retrieval Conference (TREC-3, Category B), our system takes a completely automatic approach to both routing and ad-hoc retrieval, using a combination of a statistical learner (Expert Network or ExpNet) and a shared-word-based matcher (STR). Our focus is to examine how much a Nearest Neighbor approach to query expansion can improve retrieval performance, given the kind of relevance information available in TREC. We found ExpNet effective in the routing test because large amounts of relevant documents are available for each query. In contrast, we found ExpNet often not representative of the testing queries. Therefore, relevance information about such training queries is not very useful for statistical learning about query expansion in general. A realistic strategy for the TREC collections then is to use shared-word-based matching as a basic approach to relevance judgment, and use statistical learning about human judgements for additional evidence. Our experiments show that combining ExpNet and STR leads to better results than using either alone.