Evaluation of distributed machine learning algorithms for anomaly detection from large-scale system logs: a case study

Name: Evaluation of distributed machine learning algorithms for anomaly detection from large-scale system logs: a case study
Author: Astekin, M., Zengin, H., Sözer, Hasan

İsim	Evaluation of distributed machine learning algorithms for anomaly detection from large-scale system logs: a case study
Yazar	Astekin, M., Zengin, H., Sözer, Hasan
Basım Tarihi:	2018
Basım Yeri	- IEEE
Konu	Log analysis, Distributed systems, Parallel processing, Anomaly detection, Big data, Machine learning
Tür	Belge
Dil	İngilizce
Dijital	Evet
Yazma	Hayır
Kütüphane:	Özyeğin Üniversitesi
Demirbaş Numarası	978-153865035-6
Kayıt Numarası	9d1d1493-2379-40d2-9e1b-8af75e88c82e
Lokasyon	Computer Science
Tarih	2018
Notlar	Cloud Computing and Big Data Laboratory (B3LAB) of TUBITAK-BILGEM ; Software Research Laboratory (SRL) of Ozyegin University
Örnek Metin	Anomaly detection is a valuable feature for detecting and diagnosing faults in large-scale, distributed systems. These systems usually provide tens of millions of lines of logs that can be exploited for this purpose. However, centralized implementations of traditional machine learning algorithms fall short to analyze this data in a scalable manner. One way to address this challenge is to employ distributed systems to analyze the immense amount of logs generated by other distributed systems. We conducted a case study to evaluate two unsupervised machine learning algorithms for this purpose on a benchmark dataset. In particular, we evaluated distributed implementations of PCA and K-means algorithms. We compared the accuracy and performance of these algorithms both with respect to each other and with respect to their centralized implementations. Results showed that the distributed versions can achieve the same accuracy and provide a performance improvement by orders of magnitude when compared to their centralized versions. The performance of PCA turns out to be better than K-means, although we observed that the difference between the two tends to decrease as the degree of parallelism increases.
DOI	10.1109/BigData.2018.8621967

Kaynağa git Özyeğin Üniversitesi

Aramaya Dön

Özyeğin Üniversitesi

Kaynağa git

Evaluation of distributed machine learning algorithms for anomaly detection from large-scale system logs: a case study

Yazar Astekin, M., Zengin, H., Sözer, Hasan

Basım Tarihi 2018

Basım Yeri - IEEE

Konu Log analysis, Distributed systems, Parallel processing, Anomaly detection, Big data, Machine learning

Tür Belge

Dil İngilizce

Dijital Evet

Yazma Hayır

Kütüphane Özyeğin Üniversitesi

Demirbaş Numarası 978-153865035-6

Kayıt Numarası 9d1d1493-2379-40d2-9e1b-8af75e88c82e

Lokasyon Computer Science

Tarih 2018

Notlar Cloud Computing and Big Data Laboratory (B3LAB) of TUBITAK-BILGEM ; Software Research Laboratory (SRL) of Ozyegin University

Örnek Metin Anomaly detection is a valuable feature for detecting and diagnosing faults in large-scale, distributed systems. These systems usually provide tens of millions of lines of logs that can be exploited for this purpose. However, centralized implementations of traditional machine learning algorithms fall short to analyze this data in a scalable manner. One way to address this challenge is to employ distributed systems to analyze the immense amount of logs generated by other distributed systems. We conducted a case study to evaluate two unsupervised machine learning algorithms for this purpose on a benchmark dataset. In particular, we evaluated distributed implementations of PCA and K-means algorithms. We compared the accuracy and performance of these algorithms both with respect to each other and with respect to their centralized implementations. Results showed that the distributed versions can achieve the same accuracy and provide a performance improvement by orders of magnitude when compared to their centralized versions. The performance of PCA turns out to be better than K-means, although we observed that the difference between the two tends to decrease as the degree of parallelism increases.

DOI 10.1109/BigData.2018.8621967