The Design and Implementation of the Chinese IR Benchmark
Resource
資訊傳播與圖書館學，6(3)，61-80
Journal
資訊傳播與圖書館學
Journal Volume
6
Journal Issue
3
Pages
61-80
Date Issued
2000
Date
2000
Author(s)
江玉婷
Abstract
The research and development of information retrieval has made considerable progress recently.
However, there is not any applicable test mechanism for system evaluation in the Chinese research
society. This paper reports our research on the design and implementation of the first Chinese
information retrieval benchmark. According to the framework and contents of the existing foreign
benchmarks, we develop a methodology to establish the Chinese IR benchmark. An IR benchmark
consists of three parts: document set, topic set, and relevance judgments. Our document set contains
132,207 documents collected from news web sites, the topic set contains 50 topics transformed from
real users’ information needs, and each topic has on the average16.34 related documents as a result of
the relevance judgments. The results of our research show that the quantity of document set is valid
from the viewpoint of sampling statistics. The topics reveal multiple kinds of information need, and
they also reflect certain real retrieval environment. Besides, the judgments given by three judges have
exhibited significant consistency, so we conclude their reliability. Although the benchmark is in its first
edition, it possesses a complete structure and medium scale. On this basis, it is readily feasible to
expand this benchmark's current scale to a proper large one in the near future.
Subjects
IR Benchmark（IR Test Collection）
IR Evaluation
Document Set
Topic
Relevance Judgment
Publisher
臺北市:國立臺灣大學圖書資訊學系
Type
journal article
File(s)![Thumbnail Image]()
Loading...
Name
jicls1999.pdf
Size
435.68 KB
Format
Adobe PDF
Checksum
(MD5):07852fbb687a4f82fde2a14fa5c67370