Author
Astekin, M., Özcan, S., Sözer, Hasan
Publication Date
2019
Publication Place
-
IEEE
Subject
Log analysis, Distributed systems, Parallel processing, Anomaly detection, Big data, Machine learning
Type
Document
Language
English
Digital
Yes
Manuscript
No
Library
Özyeğin University
Library Asset ID
978-1-7281-0857-5
Record ID
3797ac85-8095-4a36-882c-5e9d8833b99c
Library Location
Computer Science
Date
2019
Sample Text
Anomalies during system execution can be detected by automated analysis of logs generated by the system. However, large scale systems can generate tens of millions of lines of logs within days. Centralized implementations of traditional machine learning algorithms are not scalable for such data. Therefore, we recently introduced a distributed log analysis framework for anomaly detection. In this paper, we introduce an extension of this framework, which can detect anomalies earlier via incremental analysis instead of the existing offline analysis approach. In the extended version, we periodically process the log data that is accumulated so far. We conducted controlled experiments based on a benchmark dataset to evaluate the effectiveness of this approach. We repeated our experiments with various periods that determine the frequency of analysis as well as the size of the data processed each time. Results showed that our online analysis can improve anomaly detection time significantly while keeping the accuracy level same as that is obtained with the offline approach. The only exceptional case, where the accuracy is compromised, rarely occurs when the analysis is triggered before all the log data associated with a particular session of events are collected.
DOI
10.1109/BigData47090.2019.9006593
