Anomaly Detection, A Key Task for AI and Machine Learning, Explained - KDnuggets (2023)

comments

By Sciforce.

It is true that the Industrial Internet of Things will change the world someday. So far, it is the abundance of data that makes the world spin faster. Piled in sometimes unmanageable datasets, big data turned from the Holy Grail into a problem pushing businesses and organizations to make faster decisions in real-time. One way to process data faster and more efficiently is to detect abnormal events, changes or shifts in datasets. Thus, anomaly detection, a technology that relies on Artificial Intelligence to identify abnormal behavior within the pool of collected data, has become one of the main objectives of the Industrial IoT.

Anomaly detection refers to identification of items or events that do not conform to an expected pattern or to other items in a dataset that are usually undetectable by a human expert. Such anomalies can usually be translated into problems such as structural defects, errors or frauds.

Examples of potential anomalies:

  • A leaking connection pipe that leads to the shutting down of the entire production line;
  • Multiple failed login attempts indicating the possibility of fishy cyber activity;
  • Fraud detection in financial transactions.

Why is it important?


Modern businesses are beginning to understand the importance of interconnected operations to get the full picture of their business. Besides, they need to respond to fast-moving changes in data promptly, especially in case of cybersecurity threats. Anomaly detection can be a key for solving such intrusions, as while detecting anomalies, perturbations of normal behavior indicate a presence of intended or unintended induced attacks, defects, faults, and such.

Unfortunately, there is no effective way to handle and analyze constantly growing datasets manually. With the dynamic systems having numerous components in perpetual motion where the “normal” behavior is constantly redefined, a new proactive approach to identify anomalous behavior is needed.

Statistical Process Control


Statistical Process Control, or SPC, is a gold-standard methodology for measuring and controlling quality in the course of manufacturing. Quality data in the form of product or process measurements are obtained in real-time during the manufacturing process and plotted on a graph with predetermined control limits that reflect the capability of the process. Data that falls within the control limits indicates that everything is operating as expected. Any variation within the control limits is likely due to a common cause — the natural variation that is expected as part of the process. If data falls outside of the control limits, this indicates that an assignable cause might be the source of the product variation, and something within the process needs to be addressed and changed to fix the issue before defects occur. In this way, SPC is an effective method to drive continuous improvement. By monitoring and controlling a process, we can assure that it operates at its fullest potential and detect anomalies at early stages.

(Video) A review of machine learning techniques for anomaly detection - Dr David Green

Introduced in 1924, the method is likely to stay in the heart of industrial quality assurance forever. However, its integration with Artificial Intelligence techniques will be able to make it more accurate and precise and give more insights into the manufacturing process and the nature of anomalies.

Tasks for Artificial Intelligence


When human resources are not enough to handle the elastic environment of cloud infrastructure, microservices and containers, Artificial Intelligence comes in, offering help in many aspects:

Anomaly Detection, A Key Task for AI and Machine Learning, Explained - KDnuggets (1)

Tasks for Artificial Intelligence

Automation: AI-driven anomaly detection algorithms can automatically analyze datasets, dynamically fine-tune the parameters of normal behavior and identify breaches in the patterns.

Real-time analysis: AI solutions can interpret data activity in real time. The moment a pattern isn’t recognized by the system, it sends a signal.

Scrupulousness: Anomaly detection platforms provide end-to-end gap-free monitoring to go through minutiae of data and identify smallest anomalies that would go unnoticed by humans

Accuracy: AI enhances the accuracy of anomaly detection avoiding nuisance alerts and false positives/negatives triggered by static thresholds.

(Video) Introduction to Explainable AI (ML Tech Talks)

Self-learning: AI-driven algorithms constitute the core of self-learning systems that are able to learn from data patterns and deliver predictions or answers as required.

Learning Process of AI Systems


One of the best things about AI systems and ML-based solutions is that they can learn on the go and deliver better and more precise results with every iteration. The pipeline of the learning process is pretty much the same for every system and comprises the following automatic and human-assisted stages:

  • Datasets are fed to an AI system
  • Data models are developed based on the datasets
  • A potential anomaly is raised each time a transaction deviates from the model
  • A domain expert approves the deviation as an anomaly
  • The system learns from the action and builds upon the data model for future predictions
  • The system continues to accumulate patterns based on the preset conditions

Anomaly Detection, A Key Task for AI and Machine Learning, Explained - KDnuggets (2)

Learning Process of AI Systems

As elsewhere in AI-powered solutions, the algorithms to detect anomalies are built on supervised or unsupervised machine learning techniques.

Supervised Machine Learning for Anomaly Detection


The supervised method requires a labeled training set with normal and anomalous samples for constructing a predictive model. The most common supervised methods include supervised neural networks, support vector machine, k-nearest neighbors, Bayesian networks and decision trees.

Probably, the most popular nonparametric technique is K-nearest neighbor (k-NN) that calculates the approximate distances between different points on the input vectors and assigns the unlabeled point to the class of its K-nearest neighbors. Another effective model is the Bayesian network that encodes probabilistic relationships among variables of interest.

Supervised models are believed to provide a better detection rate than unsupervised methods due to their capability of encoding interdependencies between variables, along with their ability to incorporate both prior knowledge and data and to return a confidence score with the model output.

(Video) Spotting and solving everyday problems with machine learning | Session

Unsupervised Machine Learning for Anomaly Detection


Unsupervised techniques do not require manually labeled training data. They presume that most of the network connections are normal traffic and only a small amount of percentage is abnormal and anticipate that malicious traffic is statistically different from normal traffic. Based on these two assumptions, groups of frequent similar instances are assumed to be normal and the data groups that are infrequent are categorized as malicious.

The most popular unsupervised algorithms include K-means, Autoencoders, GMMs, PCAs, and hypothesis tests-based analysis.

Anomaly Detection, A Key Task for AI and Machine Learning, Explained - KDnuggets (3)

The most popular unsupervised algorithms

SciForce’s Chase for Anomalies


Like probably any company specialized in Artificial Intelligence and dealing with solutions for IoT, we found ourselves hunting for anomalies for our client from the manufacturing industry. Using generative models for likelihood estimation, we detected the algorithm defects, speeding up regular processing algorithms, increasing the system stability, and creating a customized processing routine which takes care of anomalies.

For anomaly detection to be used commercially, it needs to encompass two parts: anomaly detection itself and prediction of future anomalies.

Anomaly detection part


For the anomaly detection part, we relied on autoencoders — models that map input data into a hidden representation and then attempt to restore the original input from this internal representation. For regular pieces of data, such reconstruction will be accurate, while in case of anomalies, the decoding result will differ noticeably from the input.

Anomaly Detection, A Key Task for AI and Machine Learning, Explained - KDnuggets (4)

Results of our anomaly detection model. Potential anomalies are marked in red.

(Video) Transformer Neural Networks - EXPLAINED! (Attention is all you need)

In addition to the autoencoder model, we had a quantitative assessment of the similarity between the reconstruction and the original input. For this, we first computed sliding window averages for sensor inputs, i.e. the average value for each sensor over a 1-min. interval each 30 sec. and fed the data to the autoencoder model. Afterwards, we calculated distances between the input data and the reconstruction on a set of data and computed quantiles for distances distribution. Such quantiles allowed us to translate an abstract distance number into a meaningful measure and mark samples that exceeded a present threshold (97%) as an anomaly.

Sensor readings prediction


With enough training data, quantiles can serve as an input for prediction models based on recurrent neural networks (RNNs). The goal of our prediction model was to estimate sensor readings in future.
Though we used each sensor to predict other sensors’ behavior, we had trained a separate model for each sensor. Since the trends in data samples were clear enough, we used linear autoregressive models that used previous readings to predict future values.

Similarly to the anomaly detection part, we computed average each sensor values over 1-min. interval each 30 sec. Then we built a 30-minute context (or the number of previous timesteps) by stacking 30 consecutive windows. The resulting data was fed into prediction models for each sensor and the predictions were saved as estimates of the sensor readings for the following 1-minute window. To expand over time, we gradually substituted the older windows with predicted values.

Anomaly Detection, A Key Task for AI and Machine Learning, Explained - KDnuggets (5)

Results of prediction models outputs with historical data marked in blue and predictions in green.

It turned out that the context is crucial for predicting the next time step. With the scarce data available and relatively small context windows we could make accurate predictions for up to 10 minutes ahead.

Conclusion


Anomaly detection alone or coupled with the prediction functionality can be an effective means to catch the fraud and discover strange activity in large and complex datasets. It may be crucial for banking security, medicine, marketing, natural sciences, and manufacturing industries which are dependent on the smooth and secure operations. With Artificial Intelligence, businesses can increase effectiveness and safety of their digital operations — preferably, with our help.

(Video) Artificial Intelligence Full Course | AI Full Course | AI And ML Full Course 2022 | Simplilearn


Original. Reposted with permission.

Related:

  • What is Benford’s Law and why is it important for data science?
  • How to Become a (Good) Data Scientist – Beginner Guide
  • The Machine Learning Puzzle, Explained

More On This Topic

  • How to use Machine Learning for Anomaly Detection and Conditional…
  • Automated Anomaly Detection Using PyCaret
  • Introducing MIDAS: A New Baseline for Anomaly Detection in Graphs
  • Crop Disease Detection Using Machine Learning and Computer Vision
  • Machine Learning Key Terms, Explained
  • Machine Learning Algorithms Explained in Less Than 1 Minute Each

FAQs

What does anomaly detection do in machine learning? ›

Anomaly detection is identifying data points in data that don't fit the normal patterns. It can be useful to solve many problems including fraud detection, medical diagnosis, etc. Machine learning methods allow to automate anomaly detection and make it more effective, especially when large datasets are involved.

What is anomaly detection in AI? ›

Anomaly detection is a technique that uses AI to identify abnormal behavior as compared to an established pattern. Anything that deviates from an established baseline pattern is considered an anomaly. Dynatrace's AI autogenerates baseline, detects anomalies, remediates root cause, and sends alerts.

What are the three 3 basic approaches to anomaly detection? ›

There are three main classes of anomaly detection techniques: unsupervised, semi-supervised, and supervised. Essentially, the correct anomaly detection method depends on the available labels in the dataset.

What happens in anomaly detection Mcq? ›

What happens in anomaly detection? - ' b)Build Machine Learning algorithms ' is the correct answer. 'Anomaly detection' is the 'identification of rare events ', 'items', or 'observations' that are 'suspicious' as a result of they take issue considerably from customary behaviors or patterns.

What algorithm is used for anomaly detection? ›

Isolation Forest is an unsupervised anomaly detection algorithm that uses a random forest algorithm (decision trees) under the hood to detect outliers in the dataset. The algorithm tries to split or divide the data points such that each observation gets isolated from the others.

What is an example of an anomaly? ›

An anomaly is an abnormality, a blip on the screen of life that doesn't fit with the rest of the pattern. If you are a breeder of black dogs and one puppy comes out pink, that puppy is an anomaly.

What are the characteristics of anomaly detection? ›

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. These nonconforming patterns are often referred to as anomalies, outliers, discordant observations, exceptions, aberrations, surprises, peculiarities, or contaminants in different application domains [2].

What are 3 things that can be anomalies? ›

Anomalies can be classified into the following three categories:
  • Point Anomalies. If one object can be observed against other objects as anomaly, it is a point anomaly. ...
  • Contextual Anomalies. If object is anomalous in some defined context. ...
  • Collective Anomalies.
10 Apr 2018

What is anomaly detection What are the different types of anomalies? ›

Anomaly detection or outlier detection is the process of identifying rare items, observations, patterns, outliers, or anomalies which will significantly differ from the normal items or the patterns. Anomalies are sometimes referred to as outliers, novelties, noise, deviations or exceptions.

Why do we need anomaly detection? ›

Anomaly detection (aka outlier analysis) is a step in data mining that identifies data points, events, and/or observations that deviate from a dataset's normal behavior. Anomalous data can indicate critical incidents, such as a technical glitch, or potential opportunities, for instance, a change in consumer behavior.

How can we prevent anomaly? ›

The simplest way to avoid update anomalies is to sharpen the concepts of the entities represented by the data sets. In the preceding example, the anomalies are caused by a blending of the concepts of orders and products. The single data set should be split into two data sets, one for orders and one for products.

Which of the following is an advantage of anomaly detection? ›

Which of the following is an advantage of anomaly detection? Explanation: Once a protocol has been built and a behavior defined, the engine can scale more quickly and easily than the signature-based model because a new signature does not have to be created for every attack and potential variant.

What is anomaly detection example? ›

A single instance of data is anomalous if it deviates largely from the rest of the data points. An example is Detecting credit card fraud based on “amount spent.”

How do you make an anomaly detection model? ›

The anomaly detection process consists of the following phases:
  1. Exploratory data analysis.
  2. Data pre-processing and data cleansing.
  3. Data enrichment.
  4. Selecting machine learning algorithms for anomaly detection.
  5. Model training.
  6. Anomaly detection model performance evaluation.
8 Sept 2021

What is an anomaly detection? ›

Anomaly detection is the process of finding outliers in a given dataset. Outliers are the data objects that stand out amongst other objects in the dataset and do not conform to the normal behavior in a dataset.

Is anomaly detection a classification problem? ›

Using anomaly detection, no region in data space with a good number of observations from the reference class will be classified as anomaly, so this means that in your two class problem the minority class cannot be found anywhere where the majority class is present strongly enough.

What is the change detection problem in anomaly detection? ›

Abstract. For the anomalous change detection problem, you have a pair of images, taken of the same scene, but at differ- ent times and typically under different viewing conditions. You are looking for interesting differences between the two images.

Why do anomalies exist? ›

Most anomalies are found to be associated with faulty interpretations or systematic errors in the experiments. Recent examples for such outcomes involve the experimental claims for faster-than-light neutrinos and unusually strong gravitational waves from cosmic inflation.

Is an anomaly good or bad? ›

Although the word 'anomaly' may have negative connotations, it does not necessarily indicate something bad has happened. An anomaly can also mean that something very good has happened – a result better than expected – which has completely skewed the results.

Do anomalies exist? ›

There exists nonperturbative global anomalies classified by cyclic groups Z/nZ classes also known as the torsion part. It is widely known and checked in the late 20th century that the standard model and chiral gauge theories are free from perturbative local anomalies (captured by Feynman diagrams).

Which type of anomalies are there? ›

There are three types of anomalies: update, deletion, and insertion anomalies.

Is anomaly detection supervised or unsupervised? ›

We conclude that unsupervised methods are more powerful for anomaly detection in images, especially in a setting where only a small amount of anomalous data is available, or the data is unlabeled.

How do you deal with data anomaly? ›

When you want to do Multivariate anomaly detection you have to first normalize the values in the data so that algorithm can give correct predictions. Normalization or Standardization is essential when dealing with continuous values.

What is another term for anomaly? ›

1 : something different, abnormal, peculiar, or not easily classified : something anomalous. 2 : deviation from the common rule : irregularity.

Which functions work with anomaly detection? ›

Functions for Anomaly Detection. You can use simple functions, prediction-based functions, or statistical functions to examine trends that might indicate an anomaly. Simple functions can give insight into the rate of change and trends.

Which machine learning technique is used to detect outliers? ›

Code for Outlier Detection Using Interquartile Range (IQR)

You can use the box plot, or the box and whisker plot, to explore the dataset and visualize the presence of outliers. The points that lie beyond the whiskers are detected as outliers. You can generate box plots in Seaborn using the boxplot function.

Why do we need anomaly detection? ›

Anomaly detection (aka outlier analysis) is a step in data mining that identifies data points, events, and/or observations that deviate from a dataset's normal behavior. Anomalous data can indicate critical incidents, such as a technical glitch, or potential opportunities, for instance, a change in consumer behavior.

What is anomaly detection example? ›

A single instance of data is anomalous if it deviates largely from the rest of the data points. An example is Detecting credit card fraud based on “amount spent.”

What are the characteristics of anomaly detection? ›

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. These nonconforming patterns are often referred to as anomalies, outliers, discordant observations, exceptions, aberrations, surprises, peculiarities, or contaminants in different application domains [2].

How is anomaly detection different from classification? ›

Anomaly detection is not binary classification because our models do not explicitly model an anomaly. Instead, they learn to recognize only what it is to be normal. In fact, we could use binary classification if we had a lot of anomalies of all kinds to work with… But then, they wouldn't be anomalies after all!

What are 3 things that can be anomalies? ›

Anomalies can be classified into the following three categories:
  • Point Anomalies. If one object can be observed against other objects as anomaly, it is a point anomaly. ...
  • Contextual Anomalies. If object is anomalous in some defined context. ...
  • Collective Anomalies.
10 Apr 2018

What are the difficulties in anomaly detection? ›

Challenges in anomaly detection include appropriate feature extraction, defining normal behaviors, handling imbalanced distribution of normal and abnormal data, addressing the variations in abnormal behavior, sparse occurrence of abnormal events, environmental variations, camera movements, etc.

Which of the following is an advantage of anomaly detection? ›

Which of the following is an advantage of anomaly detection? Explanation: Once a protocol has been built and a behavior defined, the engine can scale more quickly and easily than the signature-based model because a new signature does not have to be created for every attack and potential variant.

Is anomaly detection supervised or unsupervised? ›

We conclude that unsupervised methods are more powerful for anomaly detection in images, especially in a setting where only a small amount of anomalous data is available, or the data is unlabeled.

What is the change detection problem in anomaly detection? ›

Abstract. For the anomalous change detection problem, you have a pair of images, taken of the same scene, but at differ- ent times and typically under different viewing conditions. You are looking for interesting differences between the two images.

Which machine learning technique is used to detect outliers? ›

Code for Outlier Detection Using Interquartile Range (IQR)

You can use the box plot, or the box and whisker plot, to explore the dataset and visualize the presence of outliers. The points that lie beyond the whiskers are detected as outliers. You can generate box plots in Seaborn using the boxplot function.

Which type of anomalies are there? ›

There are three types of anomalies: update, deletion, and insertion anomalies.

How do you deal with data anomaly? ›

When you want to do Multivariate anomaly detection you have to first normalize the values in the data so that algorithm can give correct predictions. Normalization or Standardization is essential when dealing with continuous values.

How do you identify anomalies in time series data? ›

The procedure for detecting anomalies with ARIMA is: Predict the new point from past datums and find the difference in magnitude with those in the training data. Choose a threshold and identify anomalies based on that difference threshold. That's it!

Is anomaly detection predictive technique? ›

For predictive maintenance of machines, anomaly detection tasks are the most relevant. Examples of mathematical concepts for unsupervised learning include PCA (Principal component analysis), SOM (Self organizing maps), Neural Networks, k-means clustering etc.

Is anomaly detection classification or regression? ›

As you might see by now, supervised anomaly detection is actually classification, but overall they are two distinct machine learning problems. The two key factors for differentiating them are if you have labeled classes and whether it is an imbalanced dataset or not.

What is classification based anomaly detection? ›

The typical anomaly detection setting is a one class classi- fication task, where the objective is to classify data as normal or anomalous. The importance of the task stems from being able to raise an alarm when detecting a different pattern from those seen in the past, therefore triggering further inspection.

Videos

1. Supervised, Unsupervised and Reinforcement Learning in Artificial Intelligence in Hindi
(Gate Smashers)
2. 🔥Machine Learning Algorithms Full Course For 2022 | AI and Machine Learning Algorithms | Simplilearn
(Simplilearn)
3. One trick to find almost any dataset for Data Science project -Free Datasets | Search FREE Datasets
(The AI University)
4. Frontiers in Machine Learning: Machine Learning Reliability and Robustness
(Microsoft Research)
5. Artificial Intelligence Full Course | Artificial Intelligence Tutorial for Beginners | Edureka
(edureka!)
6. Predictive Analytics Pt 1
(Gary Janchenko)
Top Articles
Latest Posts
Article information

Author: Melvina Ondricka

Last Updated: 01/19/2023

Views: 6259

Rating: 4.8 / 5 (68 voted)

Reviews: 83% of readers found this page helpful

Author information

Name: Melvina Ondricka

Birthday: 2000-12-23

Address: Suite 382 139 Shaniqua Locks, Paulaborough, UT 90498

Phone: +636383657021

Job: Dynamic Government Specialist

Hobby: Kite flying, Watching movies, Knitting, Model building, Reading, Wood carving, Paintball

Introduction: My name is Melvina Ondricka, I am a helpful, fancy, friendly, innocent, outstanding, courageous, thoughtful person who loves writing and wants to share my knowledge and understanding with you.