Control System Data Analysis Current Issues and Solution

Control System Data Analysis Current Issues and Solution 20 November 2013 CERN - Openlab Workshop Author: Filippo Tilaro Supervised by: Axel Voitier

Summary § Current “production” state § Use-cases under analysis GAS alarms breakdown § Control System Health § Statistical Analysis of Alarms § § Issues and Current limitations Openlab Workshop – 20 November 2013 2

Current “production” State Services § Control and Monitoring system § Huge data volume: § OS logs, performances metrics, § Alerting and reporting system § § Manually configured Based on threshold trespassing pattern § device status, Measurements, Alarms … but not efficiently exploited yet MOON (Monitoring) Supervision layer TN DIM/CMW Process layer PLCs OPC High Voltage Fieldbus Field layer Sensors & Actuators Openlab Workshop – 20 November 2013 3

Use cases under analysis § GAS system breakdown: system fault analysis and pattern extraction § § § Events sequence pattern matching Post-mortem analysis so far Fault prediction based on recognizable trails of events § Control Systems Health § Pattern matching and correlations of multivariate time series § Structured (i. e. measurements) and unstructured (i. e. logs) data § Alarms statistical analysis § Extract statistical indexes from the list of raised alarms § Pragmatic approach: automatic threshold discovery and learning § Strategy: § § Use and extend the Siemens Watch. CAT and other open-source analysis tools to extract possible patterns and discover new insights hidden in the control data Take advantage of the huge amounts of control data produced by CERN facilities Openlab Workshop – 20 November 2013 4

Gas System use-case 9 Apps 1 Data Server 7 Apps 1 Data Server 28 Applications (Sub Detector) 6 Apps 1 Data Server Multi-wire chamber Openlab Workshop – 20 November 2013 5

Gas System Analysis Extraction Events List XML Conversion Complex Diagnostic: § Alarm flooding, “domino effect” § A single fault can stop the whole process § The 1 st alarm is not necessarily the most relevant for the diagnosis § The alarm list depends on the system status § a knowledge-based model is not sufficient! Siemens Watch. CAT Pattern Extraction: § Fault Signature § Sequence Alignment Simulation of Physical Control System: § Complex System: more than 9000 equations to model all the system § Validated against the real system § Includes fault model! Openlab Workshop – 20 November 2013 6

Example: Distribution fault Bubbler (safety device broken) line 2: § Initial impact on the Pump module, then on the Distribution § The Distribution seems to not have alarms yet § The Entire Control Process collapses Openlab Workshop – 20 November 2013 7

Offline Control System Health § Goal: control system faults/anomalies detection and diagnosis Application Win. CC OA Systems Parameters (Million dpes) ALICE 100 3 ATLAS 130 12 CMS 90 10 LHCb 160 10 Accelerator Complex 120 10 § System architecture under analysis: § 16 Control Applications § § QPS, n. QPS, CRYO, CIET, CIS, PIC, WIC, LHC-CIRCUIT, PSEN … Linux control PCs : ~120 PLCs: ~300 FECs: ~100 Openlab Workshop – 20 November 2013 8

Offline Control System Health Analysis Pre-Data Analysis MOON § § Lemon § § § long term storage diagnostic data, alarms, devices status Performances metrics Exceptions Status information I II III • Data Extraction • XML-Conversion • Data Cleaning / Completion LOGs § § Win. CC OA logs Sys logs Repository UNICOS § Unified Control System Alarms CMW FECs § FECs logs (from Splunk) Openlab Workshop – 20 November 2013 Siemens Watch. CAT 9

Offline Control System Health: Status § Issues: § Huge amount of data [~130 GB + LHC] § Different data types: § Structured/Not Structured § Numerical / Boolean / Plain-text § Gaps, missing some metadata § Unsynchronized data sources § Different relationships among the subsystems § … § Initial conclusions § no single framework out of the box to analyze numerical data and not (next version of Watch. CAT) § Necessary a combination of tools for a complete data analysis (log processing, statistical analysis, pattern recognition…) § Split this use-case into smaller ones: § signal analysis use-case (next version of Watch. CAT will provide predictive trending capabilities) § automatic extraction of statistical metrics and thresholds Openlab Workshop – 20 November 2013 10

Alarms Analysis Flow Filtering & Aggregation MOON Alarms List POJOs Conversion Extraction Injection Reporting § CEP engine § Open-source rules engine § declarative paradigm Openlab Workshop – 20 November 2013 11

Typical Issues Necessary actions: § Access to the data (i. e. sensible or protected information) § Deal with data heterogeneity: file formats, units of measure, date formats, data structures § Data synchronization § Several different data sources § Data enhancement: data classification, data completeness, improve time resolution … § Data selection / filtering § Data input/output representations § … Openlab Workshop – 20 November 2013 12

Any Questions Thank you for attending! Openlab Workshop – 20 November 2013 13