Progressive Email Spam Detection Techniques Based on Content Structure Abstraction and Social Networks
Date Issued
2010
Date
2010
Author(s)
Tseng, Chi-Yao
Abstract
Email communication is indispensable nowadays, but the email spam problem continues growing drastically. The major challenge of this problem is that spammers will always develop new sophisticated approaches to attack spam filters. This adversarial and evolving nature makes a static spam filter difficult to constantly retain high detection performance. In this dissertation, we study how to design more robust systems based on the essential and enduring spam-sending characteristics. Moreover, we consider the progressive update issue that is significant but less discussed in the literature.
Based on the observation that spams with identical or similar content are usually sent in large quantities and successively, the notion of collaborative spam filtering with near-duplicate similarity matching scheme has been widely discussed. The primary idea of the similarity matching scheme for spam detection is to maintain a known spam database, formed by user feedback, to block subsequent near-duplicate spams. To better catch the evolving nature of spams, we propose a novel email abstraction scheme, which considers using email layout structure to represent emails. This newly-devised abstraction can more effectively capture the near-duplicate phenomenon of spams. Moreover, we design a complete spam detection system Cosdes (standing for COllaborative Spam DEtection System), which possesses an efficient near-duplicate matching scheme and a progressive update scheme. The progressive update scheme enables system Cosdes to keep the most up-to-date information for near-duplicate detection.
On the other hand, motivated by the fact that spammers are prone to have unusual behavior and specific patterns of email communication, another central topic of this dissertation is to explore email social networks to detect spams. Previous works related to this topic generally suffer from two problems: (1) the system is not robust in diverse environments, and (2) no update scheme is provided to catch the feature changes of evolving networks. To remedy these problems, we propose an incremental support vector machine (SVM) model for spam detection on dynamic email social networks. A complete spam detection system MailNET is devised to better adjust to diverse networks. Several features of each user in the network are extracted to train an SVM model. Moreover, to catch the evolving nature of email communication, we employ an incremental update scheme that enables MailNET to efficiently re-train an approximate SVM model when a set of new emails added into the network.
In addition, we also examine the feasibility of distinguishing spam nodes from normal users in email social networks by a power iteration algorithm. This algorithm generates a reputation score for each node to determine the possibility of being a spammer. Since we do not intend to produce a ranking list but to separate suspicious nodes from normal ones, relaxed constraints are introduced to expedite the convergence of the proposed PageRank-like algorithm. On the basis of this algorithm, we design a spam detection system ProMail that models email communication as a network and calculates a reputation score for each node. Furthermore, to capture the dynamic nature of email interactions, a progressive update scheme is proposed to not only include newly arrived emails but also delete obsolete ones. The designed power iteration algorithm has the progressive update capability, and thus can update the reputation scores of associated nodes.
We conduct extensive experiments over our studies. To better simulate the real email environment, we use university-scale email streams as the evaluation datasets. The experimental results show that the designed systems Cosdes, MailNET, and ProMail are effective and can be applicable to the real-world environment.
Subjects
Email Spam Detection
Progressive Update
Near-duplicate Matching
Social Networks
Supervised Learning Model
Type
thesis
File(s)![Thumbnail Image]()
Loading...
Name
ntu-99-F93921037-1.pdf
Size
23.32 KB
Format
Adobe PDF
Checksum
(MD5):d584e7f789417c11ffbad0d110a35eca