Transactions of the Japanese Society for Artificial Intelligence
Online ISSN : 1346-8030
Print ISSN : 1346-0714
ISSN-L : 1346-0714
Original Paper
Finding and Classifying Near-Duplicate Pages based on Identical Sentences Detection
Tomohide ShibataNaun KangSadao Kurohashi
Author information
JOURNAL FREE ACCESS

2010 Volume 25 Issue 1 Pages 224-232

Details
Abstract
The recent explosive increase of Web pages has made it possible for us to obtain a variety of information with a search engine. However, by some estimates, as many as 40% of the pages on the Web are duplicates of the other pages. Therefore, there is a problem that some search results contain duplicate pages. This paper proposes a method for finding similar pages from a huge amount of Web pages: hundred million Japanese Web pages. Similar pages are defined as two pages that share some sentences, and are classified into mirror pages, citation pages and plagiaristic pages, etc. First, in each page, its content region is extracted since sentences in a non-content region do not tend to be utilized for the similar page detection. From the content region in each page, relatively long sentences are extracted. This is because two pages tend to be relevant when they share relatively long sentences. A pair of pages that has the identical sentences is regarded as similar pages. Next, similar pages are classified based on several information such as an overlap ratio, the number of inlinks/outlinks, and the URL similarity. We conducted the similar page detection and classification on the large scale Japanese Web page collection, and can find some mirror pages, citation pages, and plagiaristic pages.
Content from these authors
© 2010 JSAI (The Japanese Society for Artificial Intelligence)
Previous article
feedback
Top