Author
Astekin, M., Zengin, H., Sözer, Hasan
Publication Date
2018
Publication Place
-
IEEE
Subject
Log analysis, Distributed systems, Parallel processing, Anomaly detection, Big data, Machine learning
Type
Document
Language
English
Digital
Yes
Manuscript
No
Library
Özyeğin University
Library Asset ID
978-153865035-6
Record ID
9d1d1493-2379-40d2-9e1b-8af75e88c82e
Library Location
Computer Science
Date
2018
Notes
Cloud Computing and Big Data Laboratory (B3LAB) of TUBITAK-BILGEM ; Software Research Laboratory (SRL) of Ozyegin University
Sample Text
Anomaly detection is a valuable feature for detecting and diagnosing faults in large-scale, distributed systems. These systems usually provide tens of millions of lines of logs that can be exploited for this purpose. However, centralized implementations of traditional machine learning algorithms fall short to analyze this data in a scalable manner. One way to address this challenge is to employ distributed systems to analyze the immense amount of logs generated by other distributed systems. We conducted a case study to evaluate two unsupervised machine learning algorithms for this purpose on a benchmark dataset. In particular, we evaluated distributed implementations of PCA and K-means algorithms. We compared the accuracy and performance of these algorithms both with respect to each other and with respect to their centralized implementations. Results showed that the distributed versions can achieve the same accuracy and provide a performance improvement by orders of magnitude when compared to their centralized versions. The performance of PCA turns out to be better than K-means, although we observed that the difference between the two tends to decrease as the degree of parallelism increases.
DOI
10.1109/BigData.2018.8621967
