Machine Learning for your Infrastructure: Anomaly Detection with Elastic + X-Pack (2023)

Machine Learning for your Infrastructure: Anomaly Detection with Elastic + X-Pack (1)

The world continues to go through the digital transformation at an accelerating pace. Modern applications and infrastructure continue to expand and operational complexity continues to grow. According to a recent ManageEngine Application Performance Monitoring Survey:

  • 28 percent use ad-hoc scripts to detect issues in over 50 percent of their applications.
  • 32 percent learn about application performance issues from end users.
  • 59 percent of trust monitoring tools to identify most performance deviations.

Most enterprises and web-scale companies have instrumentation & monitoring capabilities with an ElasticSearch cluster. They have a high amount of collected data but struggle to use it effectively. This available data can be used to improve the availability and effectiveness of performance and uptime along with root cause analysis and incident prediction.

Here is the main question: How to make sense of the huge piles of collected data? The first step towards making sense of data is to understand the correlations between the time series data. But only understanding will not work since correlation does not imply causation. We need a practical and scalable approach to understand the cause-effect relationship between data sources and events across the complex infrastructure of VMs, containers, networks, micro-services, regions, etc.

It’s very likely that due to one component something goes wrong with another component. In such cases, operational historical data can be used to identify the root cause by investigating through a series of intermediate causes and effects. Machine learning is particularly useful for such problems where we need to identify “what changed”, since machine learning algorithms can easily analyze existing data to understand the patterns, thus making easier to recognize the cause. This is known as unsupervised learning, where the algorithm learns from the experience and identifies similar patterns when they come along again.

Let’s see how you can setup Elastic + X-Pack to enable anomaly detection for your infrastructure & applications.

1. Setup Elasticsearch:

According to Elastic documentation, it is recommended to use the Oracle JDK version 1.8.0_131. Check if you have required Java version installed on your system. It should be at least Java 8 if required install/upgrade accordingly.

  • Download elasticsearch tarball and untar it
  • It will then create a folder named elasticsearch-5.5.1. Go into the folder.
  • Install X-Pack into Elasticsearch
  • Start elasticsearch

2. Setup Kibana

Kibana is open source analytics and visualization platform designed to work with Elasticsearch.

  • Download kibana tarball and untar it
  • It will then create a folder named kibana-5.5.1. Go into the directory.
  • Install X-Pack into Kibana
  • Running kibana
  • Navigate to Kibana at http://localhost:5601/
  • Log in as the built-in user elastic and password changeme.
  • You will see the below screen:
Machine Learning for your Infrastructure: Anomaly Detection with Elastic + X-Pack (2)
(Video) A walk through anomaly detection using Elastic's Machine Learning

3. Metricbeat:

Metricbeat helps in monitoring servers and the services they host by collecting metrics from the operating system and services. We will use it to get CPU utilization metrics of our local system in this blog.

  • Download Metric Beat’s tarball and untar it
  • It will create a folder
  • By default, Metricbeat is configured to send collected data to elasticsearch running on localhost. If your elasticsearch is hosted on any server, change the IP and authentication credentials in metricbeat.yml file.
  • Metric beat provides the following stats:

-System load

-CPU stats

-IO stats

-Per filesystem stats

-Per CPU core stats

-File system summary stats

-Memory stats

-Network stats

(Video) Automated Anomaly Detection with Elasticsearch Using Machine Learning

-Per-process stats

  • Start Metricbeat as daemon process

Now, all setup is done. Let’s go to step 2 to create machine learning jobs.

  • Real-time data: We have metricbeat providing us the real-time series data which will be used for unsupervised learning. Follow the below steps to define index pattern metricbeat-* in Kibana to search against this pattern in Elasticsearch:
    - Go to Management -> Index Patterns
    - Provide Index name or pattern as metricbeat-*
    - Select Time filter field name as @timestamp
    - Click Create

You will not be able to create an index if elasticsearch did not contain any metric beat data. Make sure your metric beat is running and output is configured as elasticsearch.

Machine Learning for your Infrastructure: Anomaly Detection with Elastic + X-Pack (4)
  • Saved Historic data: Just to see quickly how machine learning detect the anomalies you can also use data provided by Elastic. Download sample data by clicking here.

-Unzip the files in a folder: tar -zxvf server_metrics.tar.gz

-Download this script. It will be used to upload sample data to elastic.

-Provide execute permissions to the file: chmod +x upload_server-metrics.sh

-Run the script.

-As we created an index pattern for metricbeat data, in same way create index pattern server-metrics*

There are two scenarios in which data is considered anomalous. First, when the behavior of key indicator changes over time relative to its previous behavior. Secondly, when within a population behavior of an entity deviates from other entities in population over a single key indicator.

(Video) Elasticsearch Machine learning multi metric job anomaly detection

To detect these anomalies, there are three types of jobs we can create:

  1. Single Metric job: This job is used to detect Scenario 1 kind of anomalies over only one key performance indicator.
  2. Multimetric job: Multimetric job also detects Scenario 1 kind of anomalies but in this type of job we can track more than one performance indicators, such as CPU utilization along with memory utilization.
  3. Advanced job: This kind of job is created to detect anomalies of type 2.

For simplicity, we are creating following single metric jobs:

  1. Tracking CPU Utilization: Using metric beat data
  2. Tracking total requests made on server: Using sample server data

Follow the below steps to create single metric jobs:

  • Saved Historic data: Just to see quickly how machine learning detect the anomalies you can also use data provided by Elastic. Download sample data by clicking here.
  • Unzip the files in a folder: tar -zxvf server_metrics.tar.gz
  • Download this script. It will be used to upload sample data to elastic.
  • Provide execute permissions to the file: chmod +x upload_server-metrics.sh
  • Run the script.
  • As we created an index pattern for the metricbeat data, in same way create index pattern server-metrics*

There are two scenarios in which data is considered anomalous. First, when the behavior of key indicator changes over time relative to its previous behavior. Secondly, when within a population behavior of an entity deviates from other entities in population over a single key indicator.

To detect these anomalies, there are three types of jobs we can create:

  1. Single Metric job: This job is used to detect Scenario 1 kind of anomalies over only one key performance indicator.
  2. Multimetric job: Multimetric job also detects Scenario 1 kind of anomalies but in this type of job we can track more than one performance indicators, such as CPU utilization along with memory utilization.
  3. Advanced job: This kind of job is created to detect anomalies of type 2.

For simplicity, we are creating following single metric jobs:

  1. Tracking CPU Utilization: Using metric beat data
  2. Tracking total requests made on server: Using sample server data

Follow the below steps to create single metric jobs:

Job1: Tracking CPU Utilization

Job2: Tracking total requests made on the server

  • Go to http://localhost:5601/
  • Go to Machine learning tab on the left panel of Kibana.
  • Click on Create new job
  • Click Create single metric job
  • Select index we created in Step 2 i.e. metricbeat-* and server-metrics* respectively
  • Configure jobs by providing the following values:
  1. Aggregation: Here you need to select an aggregation function that will be applied to a particular field of data we are analyzing.
  2. Field: It is a drop down, will show you all field that you have w.r.t index pattern.
  3. Bucket span: It is an interval time for analysis. Aggregation function will be applied on the selected field after every interval time specified here.
  • If your data contains so many empty buckets i.e. data is sparse and you don’t want to consider it as anomalous check the checkbox named sparse data (if it appears).
  • Click on Use full <index pattern> data to use all available data for analysis.
Machine Learning for your Infrastructure: Anomaly Detection with Elastic + X-Pack (5)
  • Click on the play symbol
  • Provide job name and description
  • Click on Create Job

After creating a job the data available will be analyzed. Click on view results, you will see a chart which will show the actual and upper & lower bound of predicted value. If actual value lies outside of the range, it will be considered as anomalous. The Color of the circles represents the severity level.

Machine Learning for your Infrastructure: Anomaly Detection with Elastic + X-Pack (7)
(Video) Machine Learning Forecasting with Elasticsearch, Elastic Stack (ELK Stack)
Machine Learning for your Infrastructure: Anomaly Detection with Elastic + X-Pack (8)
  • Click on the machine learning tab in the left panel. The jobs we created will be listed here.
  • You will see the list of actions for every job you have created.
  • Since we are storing every minute data for Job1 using metricbeat. We can feed the data to the job in real time. Click on play button to start the data feed. As we get more and more data prediction will improve.
  • You see details of anomalies by clicking Anomaly Viewer.
Machine Learning for your Infrastructure: Anomaly Detection with Elastic + X-Pack (9)
Machine Learning for your Infrastructure: Anomaly Detection with Elastic + X-Pack (10)

We have seen how machine learning can be used to get patterns among the different statistics along with anomaly detection. After identifying anomalies, it is required to find the context of those events. For example, to know about what other factors are contributing to the problem? In such cases, we can troubleshoot by creating multimetric jobs.

*******************************************************************

This post was originally published on Velotio Blog.

Velotio Technologies is an outsourced software product development partner for technology startups and enterprises. We specialize in enterprise B2B and SaaS product development with a focus on artificial intelligence and machine learning, DevOps, and test engineering.

Interested in learning more about us? We would love to connect with you on our Website, LinkedIn or Twitter.

*******************************************************************

(Video) Introducing Machine Learning for the Elastic Stack

FAQs

Which algorithm is best for anomaly detection? ›

Local outlier factor (LOF)

Local outlier factor is probably the most common technique for anomaly detection. This algorithm is based on the concept of the local density. It compares the local density of an object with that of its neighbouring data points.

Is Elasticsearch machine learning free? ›

A Powerful Skill at Your Fingertips Learning the fundamentals of machine learning features of elastic search. It puts a powerful and very useful tool at your fingertips. Elastic Search is free, easy to learn, has excellent documentation.

What is Elasticsearch anomaly detection? ›

Anomaly detectionedit

Anomaly detection runs in and scales with Elasticsearch, and includes an intuitive UI on the Kibana Machine Learning page for creating anomaly detection jobs and understanding results.

What is Elasticsearch in machine learning? ›

Elasticsearch is a feature-rich, open-source search engine built on top of Apache Lucene, one of the most important full-text search engines on the market.

What are the three 3 basic approaches to anomaly detection? ›

There are three main classes of anomaly detection techniques: unsupervised, semi-supervised, and supervised. Essentially, the correct anomaly detection method depends on the available labels in the dataset.

Which machine learning technique can be used for anomaly detection? ›

The most commonly used algorithms for this purpose are supervised Neural Networks, Support Vector Machine learning, K-Nearest Neighbors Classifier, etc.

Is Elasticsearch difficult to learn? ›

Elasticsearch is a highly scalable open source search engine. Although it started as a text search engine, it is evolving as an analytical engine, which can support not only search but also complex aggregations. Its distributed nature and ease of use makes it very easy to get started and scale as you have more data.

Is Elasticsearch worth Learning? ›

Elasticsearch is great, I can store huge amounts of data and for simple searches get an answer in under a tenth of a second, without big powerful servers. It really isn't suited to small projects however, there is a very steep learning curve and the service is really quite harsh to newbies.

What is Elasticsearch best for? ›

Elasticsearch allows you to store, search, and analyze huge volumes of data quickly and in near real-time and give back answers in milliseconds. It's able to achieve fast search responses because instead of searching the text directly, it searches an index.

What is the purpose of anomaly detection? ›

Anomaly detection aims at finding unexpected or rare events in data streams, commonly referred to as anomalous events. Detecting anomalies could be useful directly or as a first insight to find new knowledge in the data.

What are the difficult in a anomaly detection? ›

Challenges in anomaly detection include appropriate feature extraction, defining normal behaviors, handling imbalanced distribution of normal and abnormal data, addressing the variations in abnormal behavior, sparse occurrence of abnormal events, environmental variations, camera movements, etc.

What are the data types of anomaly detection? ›

The three different types of time series anomalies
  • Global outliers. Also known as point anomalies, these outliers exist far outside the entirety of a data set.
  • Contextual outliers. ...
  • Collective outliers.

Which algorithm is used in Elasticsearch? ›

The default scoring algorithm used by Elasticsearch is BM25. There are three main factors that determine a document's score: Term frequency (TF) — The more times that a search term appears in the field we are searching in a document, the more relevant that document is.

Is Elasticsearch a programming language? ›

Elasticsearch is developed in Java and is dual-licensed under the source-available Server Side Public License and the Elastic license, while other parts fall under the proprietary (source-available) Elastic License.
...
Elasticsearch.
Original author(s)Shay Banon
Written inJava
Operating systemCross-platform
TypeSearch and index
9 more rows

What type of data is Elasticsearch? ›

Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene and was first released in 2010 by Elasticsearch N.V. (now known as Elastic).

What are 3 things that can be anomalies? ›

Anomalies can be classified into the following three categories:
  • Point Anomalies. If one object can be observed against other objects as anomaly, it is a point anomaly. ...
  • Contextual Anomalies. If object is anomalous in some defined context. ...
  • Collective Anomalies.
10 Apr 2018

How many types of anomaly are there? ›

There are three types of anomalies: update, deletion, and insertion anomalies. An update anomaly is a data inconsistency that results from data redundancy and a partial update.

How do you select anomaly detection features? ›

The anomaly detection/feature selection is done by simply flagging each metric as a zero or a 1 if its value is inside or outside normal range (0 for within normal range; 1 for outside of normal range). I also calculate a “flag ratio” that expresses how far outside of normal the value is.

Which algorithm is used for detection? ›

Popular algorithms used to perform object detection include convolutional neural networks (R-CNN, Region-Based Convolutional Neural Networks), Fast R-CNN, and YOLO (You Only Look Once). The R-CNN's are in the R-CNN family, while YOLO is part of the single-shot detector family.

Which algorithm is used for machine learning? ›

Below is the list of Top 10 commonly used Machine Learning (ML) Algorithms:
  • Linear regression.
  • Logistic regression.
  • Decision tree.
  • SVM algorithm.
  • Naive Bayes algorithm.
  • KNN algorithm.
  • K-means.
  • Random forest algorithm.
14 Nov 2022

What type of learning and machine learning techniques are used for object detection? ›

Object Detection Using Deep Learning

You can use a variety of techniques to perform object detection. Popular deep learning–based approaches using convolutional neural networks (CNNs), such as R-CNN and YOLO v2, automatically learn to detect objects within images.

How long it will take to learn Elasticsearch? ›

If you have no coding experience whatsoever, six months to a year of studying languages like SQL and Java should be sufficient preparation for adding Elasticsearch to your toolbox.

Is there anything better than Elasticsearch? ›

ArangoDB is a great alternative to Elasticsearch. It is open source under the Apache v2 license. It does search engine duties similar to Elasticsearch but packs more — document, key-value, and graph stores. And the best part is it uses a single query language, AQL, for all these different data stores.

Is Elasticsearch suitable for big data? ›

Elasticsearch is the main product of a company called 'Elastic'. It is used for web search, log analysis, and big data analytics. Often compared with Apache Solr, both depend on Apache Lucene for low-level indexing and analysis.

Is Elasticsearch good for Career? ›

On top of it, this technology allows an extremely quick data retrieval and storage process. A wide variety of career opportunities are also budding, coiling to this technology.

What is Elasticsearch not good for? ›

Elasticsearch is not a good data store as other options such as MongoDB, Hadoop, etc. It performs well for small use cases, but in case of streaming of TB's data per day, it either chokes or loses the data. It is a flexible and powerful data storage search engine, but it is a bit difficult to learn.

Should I use Elasticsearch as a database? ›

Elasticsearch is a standalone database. Its main use case is for searching text and text and/number related queries such as aggregations. Generally, it's not recommended to use Elasticsearch as the main database, as some operations such as indexing (inserting values) are more expensive compared to other databases.

Is Elasticsearch still used? ›

Since its release in 2010, Elasticsearch has quickly become the most popular search engine and is commonly used for log analytics, full-text search, security intelligence, business analytics, and operational intelligence use cases.

Is Elasticsearch frontend or backend? ›

If you are using AWS elasticsearch and you need some kind of authentication, you should let the backend talking to elasticsearch (backend using IAM roles to generate the AWS Signature). If you do it via frontend, you might have to expose your secrets which is obviously not a good strategy.

Which companies are using Elasticsearch? ›

3874 companies reportedly use Elasticsearch in their tech stacks, including Uber, Shopify, and Udemy.
  • Uber.
  • Shopify.
  • Udemy.
  • Instacart.
  • Robinhood.
  • Slack.
  • LaunchDarkly.
  • CRED.

What is the advantage of anomaly detection? ›

Perhaps the most significant benefit of anomaly detection is the automation of KPI analysis. For most businesses, KPI analysis is still a manual task of sorting through all of their digital channel's data across different dashboards.

What is an example of anomaly? ›

An anomaly is an abnormality, a blip on the screen of life that doesn't fit with the rest of the pattern. If you are a breeder of black dogs and one puppy comes out pink, that puppy is an anomaly.

What is the advantage of anomaly-based detection? ›

The major benefit of the anomaly-based detection system is about the scope for detection of novel attacks. This type of intrusion detection approach could also be feasible, even if the lack of signature patterns matches and also works in the condition that is beyond regular patterns of traffic.

How can anomaly detection be improved? ›

How to Develop an Anomaly Detection System That Shortens Claim Resolution Time by 30%
  1. Exploratory data analysis.
  2. Data pre-processing and data cleansing.
  3. Data enrichment.
  4. Selecting machine learning algorithms for anomaly detection.
  5. Model training.
  6. Anomaly detection model performance evaluation.
8 Sept 2021

Which algorithm is best for outliers? ›

Isolation Forest Algorithm

Isolation forest is a tree-based algorithm that is very effective for both outlier and novelty detection in high-dimensional data.

Which deep learning model is best for anomaly detection? ›

Unsupervised learning

Because, as mentioned previously, labeled anomalous data is relatively rare, unsupervised approaches are more popular than supervised ones in the anomaly detection field.

How do you handle data anomaly? ›

When you want to do Multivariate anomaly detection you have to first normalize the values in the data so that algorithm can give correct predictions. Normalization or Standardization is essential when dealing with continuous values.

What language is Elasticsearch written? ›

Elasticsearch

What DB is used in Elasticsearch? ›

Completely open source and built with Java, Elasticsearch is a NoSQL database. That means it stores data in an unstructured way and that you cannot use SQL to query it.

Does Elasticsearch use SQL? ›

Elasticsearch has the speed, scale, and flexibility your data needs — and it speaks SQL. Use traditional database syntax to unlock non-traditional performance, like full text search across petabytes of data with real-time results.

Why use Elasticsearch instead of SQL? ›

You plan to use Elasticsearch when you're doing a lot of text search, where traditional RDBMS databases are not performing really well (poor configuration, acts as a black-box, poor performance). Elasticsearch is highly customizable, extendable through plugins.

Is Elasticsearch a ETL tool? ›

No, Elasticsearch is not an ETL tool. It is a free and open-source search engine for text, numeric, geospatial, structured, and unstructured data. Elasticsearch is mostly used in business intelligence, security intelligence, and operational intelligence. There are separate ETL tools available for Elasticsearch.

Is Kibana the same as Elasticsearch? ›

Kibana is the official interface of Elasticsearch.

Users of Elasticsearch will find Kibana to be the most effective interface for discovering data insights and performing active management of the health of their Elastic Stack.

Is Elasticsearch an algorithm? ›

Fast search usually boils down to data organization, which is why Elasticsearch is based on an inverted index. But sometimes speed comes from clever algorithms. Last year we looked at four such algorithms, but there are dozens more. In this talk we'll explore a new set of interesting algorithms in Elasticsearch.

Is Elasticsearch a DBMS? ›

Elasticsearch. Elasticsearch is a NoSQL, document-oriented database management system having a full-text search engine at its heart. Built on the Apache Lucene library, it stores data as a JSON file, supports RESTful APIs, and uses a powerful analytical engine for faster data retrieval.

How much data can Elasticsearch handle? ›

Aim to keep the average shard size between at least a few GB and a few tens of GB. For use-cases with time-based data, it is common to see shards between 20GB and 40GB in size.

Which is better for anomaly detection supervised or unsupervised? ›

We conclude that unsupervised methods are more powerful for anomaly detection in images, especially in a setting where only a small amount of anomalous data is available, or the data is unlabeled.

What is the best algorithm for object detection? ›

Most Popular Object Detection Algorithms. Popular algorithms used to perform object detection include convolutional neural networks (R-CNN, Region-Based Convolutional Neural Networks), Fast R-CNN, and YOLO (You Only Look Once). The R-CNN's are in the R-CNN family, while YOLO is part of the single-shot detector family.

Why is SVM good for anomaly detection? ›

One-class SVM, or unsupervised SVM, is an algorithm used for anomaly detection. The algorithm tries to separate data from the origin in the transformed high-dimensional predictor space. ocsvm finds the decision boundary based on the primal form of SVM with the Gaussian kernel approximation method.

Is SVM used for anomaly detection? ›

Anomaly detection typically uses data mining and machine learning methods for detecting abnormal activities in systems. Many anomaly detection techniques have been developed, including Support Vector Machines (SVM), which can solve classification and regression problems.

What are the difficulties in anomaly detection? ›

Challenges in anomaly detection include appropriate feature extraction, defining normal behaviors, handling imbalanced distribution of normal and abnormal data, addressing the variations in abnormal behavior, sparse occurrence of abnormal events, environmental variations, camera movements, etc.

Which type of analytics is used to detect anomalies? ›

With time series, an anomaly detection algorithm will based on historical data identify observations that does not conform to the expected. This is the type of anomaly detection that Google uses for Google Analytics.

What type of analytics is anomaly detection? ›

Anomaly detection is a statistical technique that Analytics Intelligence uses to identify anomalies in time-series data for a given metric, and anomalies within a segment at the same point of time.

How much data do you need for object detection? ›

Conclusion. Here is what we learned from the experiments: The minimum number of image data for training is around 150–500.

Which language is best for object detection? ›

C++ is considered to be the fastest programming language, which is highly important for faster execution of heavy AI algorithms. A popular machine learning library TensorFlow is written in low-level C/C++ and is used for real-time image recognition systems.

Which dataset is best for object detection? ›

The 10 Best Public Datasets for Object Detection in 2022
  • ImageNet.
  • COCO (Microsoft Common Objects in Context)
  • PASCAL VOC.
  • BDD100K (UCBerkeley "Deep Drive")
  • Visual Genome.
  • nuScenes.
  • DOTA v2. ...
  • KITTI Vision Benchmark Suite.
13 Sept 2022

What are the applications of anomaly detection? ›

Applications of anomaly detection include fraud detection in financial transactions, fault detection in manufacturing, intrusion detection in a computer network, monitoring sensor readings in an aircraft, spotting potential risk or medical problems in health data, and predictive maintenance.

Is anomaly detection classification or regression? ›

Anomaly detection is not binary classification because our models do not explicitly model an anomaly. Instead, they learn to recognize only what it is to be normal. In fact, we could use binary classification if we had a lot of anomalies of all kinds to work with… But then, they wouldn't be anomalies after all!

What kind of data is suitable for SVM? ›

SVMs are used in applications like handwriting recognition, intrusion detection, face detection, email classification, gene classification, and in web pages. This is one of the reasons we use SVMs in machine learning. It can handle both classification and regression on linear and non-linear data.

Is anomaly detection deep learning? ›

Deep learning methods offer the opportunity to model complex, nonlinear relationships within data, and leverage this for the anomaly detection task. The performance of deep learning models can also potentially scale with the availability of appropriate training data, making them suitable for data-rich problems.

Videos

1. Security Analytics: Using the Elastic Stack and X-Pack features - Demo
(Elastic)
2. Anomaly Detection using Machine Learning in Elastic Stack - Daily Elastic Byte S03E15
(Official Elastic Community)
3. Elasticsearch Machine learning single metric job anomaly detection
(Vinay Nataraja)
4. IT Operations with Machine Learning Demo
(Elastic)
5. ElasticCC: Machine learning with elastic stack
(Official Elastic Community)
6. Machine Learning Tutorial - Lab 2 - Creating a Multi-Metric Job
(Elastic)
Top Articles
Latest Posts
Article information

Author: Terrell Hackett

Last Updated: 10/11/2022

Views: 5979

Rating: 4.1 / 5 (52 voted)

Reviews: 83% of readers found this page helpful

Author information

Name: Terrell Hackett

Birthday: 1992-03-17

Address: Suite 453 459 Gibson Squares, East Adriane, AK 71925-5692

Phone: +21811810803470

Job: Chief Representative

Hobby: Board games, Rock climbing, Ghost hunting, Origami, Kabaddi, Mushroom hunting, Gaming

Introduction: My name is Terrell Hackett, I am a gleaming, brainy, courageous, helpful, healthy, cooperative, graceful person who loves writing and wants to share my knowledge and understanding with you.