Machine Learning Algorithms for Anomaly Detection in Particles Accelerators Technical Infrastructure
Download as pptx, pdf0 likes143 views
The document discusses the use of machine learning algorithms for anomaly detection in particle accelerators' technical infrastructure, aiming to predict failures before they occur. It presents various approaches, including Mahalanobis distance and LSTM networks, as well as a framework that utilizes neural networks for real-time monitoring of device behavior. The methods focus on improving reliability, reducing downtime, and facilitating predictive maintenance through advanced data analysis techniques.
Machine Learning Algorithms for Anomaly Detection in Particles Accelerators Technical Infrastructure
- 1. Use of Machine Learning Algorithms for Anomaly Detection in Particles Accelerators Technical Infrastructure Lorenzo Giusti [email protected] 23/04/2020 1
- 2. Problem 2 • We want to exploit upcoming failures among the accelerators’ technical infrastructure • In the current situation, a device is going to be checked if its internal temperature rises over a certain threshold level (37.5 °C) • The monitoring framework is constantly supervised by the engineers, which have thousands of devices to take care and sometimes they can miss one alarm out of millions received everyday • Thermal runaways cannot be exploited by human supervision, leading to a repentine increasing of the internal temperature which leads to an explosion 23/04/2020
- 3. Motivation 3 • Devices are not flawless: • Reliability and performances decreases with time • Faults can suddenly put out of services and impact overall availability • Preventive and corrective maintenance is expensive and impact operating time • Predictive maintenance and anticipated interventions improve: • Reduce risks • Decreases downtime • Increase reliability and overall availability 23/04/2020
- 4. Approaches 4 • Mahalonobis Distance: • Measures the distance between a point and a distribution • If the distance is above a certain threshold, the point is considered anomalous • Isolation Forest: • Split the regions “randomly” until a point is isolated • The probability of that point being non-anomalous is proportional of the number of splits • Residual Autoregressive Score: • Compute the norm between the actual time series and the one predicted with an autoregressive model (eg. AIRMA) • If the gradient of the norm is a monotonic increasing function, then the time series is classified as anomalous 23/04/2020
- 5. Solution 5 • Machine Learning based anomaly detection: • Real time monitoring with unsupervised detection • Device independent with generalized algorithms • Independent of environmental and periodic operational conditions • State of the art artificial intelligence algorithms • Faults are predicted with significant lead time (i.e. day to weeks before failure) • Different type and thresholds of anomalies are detected 23/04/2020
- 6. Failure Analysis Framework Architecture 6 Noisy temperature signal as input Extract, Load and Transform Engineered features as output • Our framework is developed using only the significant features of the devices • i.e. only the temperature sensors for the collimators • Extract Load Transform Pipeline: • Identify and remove seasonal components from the signal • Filter out the environmental noise (eg. Gaussian Smoothing) • Derive additional features in order to gain more insights on physical phenomena of interest • At the end, we homogenize the extracted data range and variability 23/04/2020
- 7. Neural Networks 7 • Neural Networks are computing systems that are inspired by the biological brain. • They learn to perform tasks by considering examples, without being programmed with task-specific rules • The original goal of the neural approach was to solve problems in the same way that a human brain would, now the attention moved to performing specific tasks McCulloch, W.S., Pitts, W., (1943), “A logical calculus of the ideas immanent in nervous activity”, Bulletin of Mathematical Biophysics. 23/04/2020
- 8. Recurrent Neural Networks 8 • In handling generic sequences of data, simple feed forward neural networks have very tough limitations, especially for: • Sequences with variable lengths (e.g. environmental processes) • Sequences with long-term dependencies • Recurrent neural networks (RNNs) are a class of neural networks naturally able to exhibit temporal dynamic behavior D. E. Rumelhart, G. E. Hinton, and R. J. Williams, (1986), “Learning internal representations by error propagation”. • RNNs are susceptible to vanishing gradients in processing long sequences 23/04/2020
- 9. Recurrent Neural Networks 9 • Sharing parameters across the models allow us to extends the model to samples of different forms and generalize across them: • The output is produced using the same update rule applied to the previous outputs • RNNs introduce the concept of cycles in the computational graph: • Cycles model the influence of the value at time t on the value at time t+ 𝜏 I. Goodfellow, Y. Bengio, and A. Courville, (2015), “Deep Learning”. • The hidden state at time t can be considered as a summary of all the previous values processed by the RNN 23/04/2020
- 10. LSTM Networks 10 • The forget gate’s vector of activations allows the network to better control the gradients values, preventing eventual vanishings of it • Well-suited to classifying, processing and making predictions based on time series, since there can be lags of unknown duration between important events in a time series • LSTM Networks are a type of recurrent models that have been used in many sequence learning tasks like speech recognition and time series forecasting Sepp Hochreiter; Jürgen Schmidhuber (1997). "Long short-term memory". In Neural Computation. 23/04/2020
- 11. Bidirectional Networks 11 • Bidirectional LSTM Networks are made by putting two independent networks together • The input sequence is given in normal time order for one network, and in reversed time order for the other one. • This structure allows the networks to have both backward and forward information about the sequence at every time step M. Schuster and K. K. Paliwal, (1997) ,”Bidirectional recurrent neural networks" in IEEE Transactions on Signal Processing. 23/04/2020
- 12. Autoencoders 12 • The principal aim is to learn a representation for a set of data, by training the network to ignore signal noise • By learning to replicate the most salient features in the model is encouraged to learn how to precisely reproduce the most frequent characteristics: • When facing anomalies, the model should fail the reconstruction • Reconstruction error of a data point, is used as an anomaly score to detect anomalies Kramer, M.A., (1991), “Nonlinear principal component analysis using autoassociative neural networks”, AIChE J. 23/04/2020
- 13. Autoencoders 13 • Formally, Autoencoders are feedforward neural networks whose aim is learning to copy its input to its output: ℐ(𝐱) = 𝐱 • As it is, the identity function itself isn’t so useful, unless we force the model to prioritize which aspects of the input should be copied, leading the model to learn useful properties of the input data • The main idea is to separate the identity function in two parts: ℐ(𝐱) = 𝑔(𝑓(𝐱)) • 𝐡 = 𝑓(𝐱) is the encoder function, which maps the input to an internal representation or code • 𝐫 = 𝑔(𝐡) is the decoder function, which produces a reconstruction of the input as a function of the code taken from the previous mapping• One way to capture the most salient features of the input, the encoder/decoder functions must be of the form: 𝑓: 𝑅 𝑛 → 𝑅 𝑚 , 𝑚 < 𝑛 𝑔: 𝑅 𝑚 → 𝑅 𝑛 , 𝑚 < 𝑛 23/04/2020
- 14. Autoencoders 14 • An Autoencoder whose code dimension is less than the input dimension is called undercomplete • The learning process of an undercomplete autoencoder is described as: 𝐖∗ = arg𝑚𝑖𝑛 𝐖 𝐿(𝐱, 𝑔(𝑓(𝐱)) = arg𝑚𝑖𝑛 𝐖 𝐱 − 𝑔(𝑓(𝐱)) 2 23/04/2020
- 15. Autoencoders 15 • Using proper activation functions lead the autoencoder to learn a generalization of the principal component analysis under a kernel function: • If the activations of the neurons are linear and the loss function is the mean squared error, it can be shown that the encoder function is just an approximation of the principal component analysis technique • There are a lot of fancy variants of the undercomplete autoencoders: • Regularized Autoencoders: • The loss function has additional terms which force the model to have other properties besides the ability to reproduce the identity function • Sparse Autoencoders: • The loss function has an additional sparsity penalty 𝛀(𝐡) (usually the L1 reg.) which induce sparsity in the code • Denoising Autoencoders: • Before the training phase, the training data is altered by some form of noise: • The model is more stable and robust to the induced corruption • Variational Autoencoders: • Make variational inference on the distribution of the training data ☠ 23/04/2020
- 16. How to detect extreme rare events 16 • Bidirectional Long Short Term Memory Autoencoder overcome with a modelling approach: • Learns and reconstruct the nominal behaviour of a time series • Use the signal versus reconstruction error to detect anomalies • Devices normal behaviour is often affected by external factors or variables which are not evident by analysing signals behaviour with time due to: • Unmonitored or unknown environmental conditions • Additional harsh conditions that add noise, i.e. radiation dose • Measurements and data acquisition errors, i.e. the difference between the measured value of a quantity and its true value 23/04/2020
- 17. Extreme rare events detection 17 Bidirectional LSTM LSTM Autoencoder Bidirectional LSTM Autoencoder 23/04/2020
- 18. Anomalies as Outliers in the Reconstruction Error Distribution 18 • The model learns how to encode the normal behavior of our devices • We set a threshold as an extreme value in the distribution of the reconstruction errors on the data we claim have no anomalies • Points with the reconstruction error over the threshold are anomalies Normal Behavior Abnormal Behavior • Subsequent anomalies trigger a critical alarm on the monitoring framework 23/04/2020
- 19. Results 19 No anomalies detected Anomaly detected on 06-19-2019 Temp. over threshold on 07-23-2019Temp. never goes over the crit. level 23/04/2020
- 20. Conclusions & Future developments 20 • With this techniques we have shown that is possible to predict well in advance if a device deviates from its nominal behavior thus predicting a potential fault • Is also possible to assess the criticality of the detected anomaly • We aim to generalize the framework to a multi-system anomaly detector, for the following systems: • Uninterruptible Power Supplies and more generally batteries✔ • Collimators ✔ • Electrical Transformers ✘ • Hydraulic pumps ✘ • Compressors ✘ • Future features will also be added to infer the type of anomaly and the extent (sensor, component, sub-system, system) 23/04/2020
- 21. Refereces 21 Zhang, C. et al., (2018), “A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data“, arXiv:1811.08055v1. Marchi, E. et al., (2015), “A Novel Approach for Automatic Acoustic Novelty Detection Using a Denoising Autoencoder with Bidirectional LSTM Neural Networks”, ICASSP. Sakurada, M., Yairi, T., (2014), “Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction”, ACM. Zhou, C., Paffenroth, R. C., (2017), “Anomaly Detection with Robust Deep Autoencoders”, KDD. Malhotra P., et al., (2016), “LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection”, ICML. Gong, D. et al., (2019) “Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection”, arXiv:1904.02639v2. Majid S. alDosari, (2016) “Unsupervised Anomaly Detection in Sequences Using Long Short Term Memory Recurrent Neural Networks”. 23/04/2020