Mining Web informative structures and contents based on entropy analysis
Journal
IEEE Transactions on Knowledge and Data Engineering
Journal Volume
16
Journal Issue
1
Pages
41-55
Date Issued
2004
Date
2004
Author(s)
Abstract
In this paper, we study the problem of mining the informative structure of a news Web site that consists of thousands of hyperlinked documents. We define the informative structure of a news Web site as a set of index pages (or referred to as TOC, i.e., table of contents, pages) and a set of article pages linked by these TOC pages. Based on the Hyperlink Induced Topics Search (HITS) algorithm, we propose an entropy-based analysis (LAMIS) mechanism for analyzing the entropy of anchor texts and links to eliminate the redundancy of the hyperlinked structure so that the complex structure of a Web site can be distilled. However, to increase the value and the accessibility of pages, most of the content sites tend to publish their pages with intrasite redundant information, such as navigation panels, advertisements, copy announcements, etc. To further eliminate such redundancy, we propose another mechanism, called InfoDiscoverer, which applies the distilled structure to identify sets of article pages. InfoDiscoverer also employs the entropy information to analyze the information measures of article sets and to extract informative content blocks from these sets. Our result is useful for search engines, information agents, and crawlers to index, extract, and navigate significant information from a Web site. Experiments on several real news Web sites show that the precision and the recall of our approaches are much superior to those obtained by conventional methods in mining the informative structures of news Web sites. On the average, the augmented LAMIS leads to prominent performance improvement and increases the precision by a factor ranging from 122 to 257 percent when the desired recall falls between 0.5 and 1. In comparison with manual heuristics, the precision and the recall of InfoDiscoverer are greater than 0.956.
Subjects
Anchor text; Entropy; Hubs and authorities; Information extraction; Informative structure; Link analysis
Other Subjects
Algorithms; Electronic commerce; HTML; Information retrieval systems; Intelligent agents; Marketing; Problem solving; Search engines; Websites; Authorities; Hubs; Information extraction; Information structures; Link analysis; World Wide Web
Type
journal article
File(s)![Thumbnail Image]()
Loading...
Name
27.pdf
Size
1.86 MB
Format
Adobe PDF
Checksum
(MD5):f420a5ce8339ed3b96746ec5f321f539