Incremental analysis of large-scale system logs for anomaly detection

Name: Incremental analysis of large-scale system logs for anomaly detection
Author: Astekin, M., Özcan, S., Sözer, Hasan

İsim	Incremental analysis of large-scale system logs for anomaly detection
Yazar	Astekin, M., Özcan, S., Sözer, Hasan
Basım Tarihi:	2019
Basım Yeri	- IEEE
Konu	Log analysis, Distributed systems, Parallel processing, Anomaly detection, Big data, Machine learning
Tür	Belge
Dil	İngilizce
Dijital	Evet
Yazma	Hayır
Kütüphane:	Özyeğin Üniversitesi
Demirbaş Numarası	978-1-7281-0857-5
Kayıt Numarası	3797ac85-8095-4a36-882c-5e9d8833b99c
Lokasyon	Computer Science
Tarih	2019
Örnek Metin	Anomalies during system execution can be detected by automated analysis of logs generated by the system. However, large scale systems can generate tens of millions of lines of logs within days. Centralized implementations of traditional machine learning algorithms are not scalable for such data. Therefore, we recently introduced a distributed log analysis framework for anomaly detection. In this paper, we introduce an extension of this framework, which can detect anomalies earlier via incremental analysis instead of the existing offline analysis approach. In the extended version, we periodically process the log data that is accumulated so far. We conducted controlled experiments based on a benchmark dataset to evaluate the effectiveness of this approach. We repeated our experiments with various periods that determine the frequency of analysis as well as the size of the data processed each time. Results showed that our online analysis can improve anomaly detection time significantly while keeping the accuracy level same as that is obtained with the offline approach. The only exceptional case, where the accuracy is compromised, rarely occurs when the analysis is triggered before all the log data associated with a particular session of events are collected.
DOI	10.1109/BigData47090.2019.9006593

Kaynağa git Özyeğin Üniversitesi

Aramaya Dön

Özyeğin Üniversitesi

Kaynağa git

Incremental analysis of large-scale system logs for anomaly detection

Yazar Astekin, M., Özcan, S., Sözer, Hasan

Basım Tarihi 2019

Basım Yeri - IEEE

Konu Log analysis, Distributed systems, Parallel processing, Anomaly detection, Big data, Machine learning

Tür Belge

Dil İngilizce

Dijital Evet

Yazma Hayır

Kütüphane Özyeğin Üniversitesi

Demirbaş Numarası 978-1-7281-0857-5

Kayıt Numarası 3797ac85-8095-4a36-882c-5e9d8833b99c

Lokasyon Computer Science

Tarih 2019

Örnek Metin Anomalies during system execution can be detected by automated analysis of logs generated by the system. However, large scale systems can generate tens of millions of lines of logs within days. Centralized implementations of traditional machine learning algorithms are not scalable for such data. Therefore, we recently introduced a distributed log analysis framework for anomaly detection. In this paper, we introduce an extension of this framework, which can detect anomalies earlier via incremental analysis instead of the existing offline analysis approach. In the extended version, we periodically process the log data that is accumulated so far. We conducted controlled experiments based on a benchmark dataset to evaluate the effectiveness of this approach. We repeated our experiments with various periods that determine the frequency of analysis as well as the size of the data processed each time. Results showed that our online analysis can improve anomaly detection time significantly while keeping the accuracy level same as that is obtained with the offline approach. The only exceptional case, where the accuracy is compromised, rarely occurs when the analysis is triggered before all the log data associated with a particular session of events are collected.

DOI 10.1109/BigData47090.2019.9006593