Development of a web-scale Chinese word N-gram corpus with parts of speech information
Journal
8th International Conference on Language Resources and Evaluation, LREC 2012
Pages
320-324
ISBN
9782951740877
Date Issued
2012
Author(s)
Abstract
Web provides a large-scale corpus for researchers to study the language usages in real world. Developing a web-scale corpus needs not only a lot of computation resources, but also great efforts to handle the large variations in the web texts, such as character encoding in processing Chinese web texts. In this paper, we aim to develop a web-scale Chinese word N-gram corpus with parts of speech information called NTU PN-Gram corpus using the ClueWeb09 dataset. We focus on the character encoding and some Chinese-specific issues. The statistics about the dataset is reported. We will make the resulting corpus a public available resource to boost the Chinese language processing.
Subjects
ClueWeb09
Encoding detection
Part-of-speech n-grams
Type
conference paper