Power Grid Research at Pacific Northwest National Laboratory

Power Grid Research at Pacific Northwest National Laboratory Moe Khaleel Laboratory Fellow and Director Computational Sciences and Mathematics Division September 10, 2009 HPC User Forum 1

Overview Introduction Problems on the electrical grid Need for HPC on the electrical grid Operations Integration of renewables Cyber security Need for HPC at vision level State estimation Multithreaded platforms for contingency analysis Looking forward

Transform the way the U. S. generates, transmits, distributes and uses electricity Current U. S. electricity infrastructure is inadequate for national energy priorities in the 21 st century. Three main areas need to be addressed: Capacity Grid management (wide area, real-time) Vulnerability, resiliency, reliability The future state of the grid must be able to: Substantially increase the integration of renewables Reduce carbon emissions Provide flexibility to enable electrification of transportation and reduce dependence on oil imports (substitute electricity for oil) Respond to increased demand Reality: The current grid infrastructure/operation is limited Currently manage/engage grid at sub-optimal (service territory) level; can’t efficiently move electrons across large enough spaces No ability to see performance across grid (lacks transparency) Inability to integrate renewables; need storage, ability to offset, transmission across service territories where generated, load/dispatch renewables System communication has been one-directional: supply to demand Fragmented authority, control, market, function, regulation Need to build new functionality and infrastructure into the grid Future grid must be transformed while maintaining reliability and affordability (serving public good) 3

Transform the way the U. S. generates, transmits, distributes and uses electricity Capacity: Transmission infrastructure cannot meet future load growth and large-scale renewable connectivity to grid Utilities not incentivized to build physical infrastructure Difficult to site & permit new transmission infrastructure Renewable resource physically isolated from high grid transmission infrastructure Grid Management: Unable to manage grid at national, interconnect scale Large scale models that allow examination and optimization of future national grid do not exist Integrated wide area models (variable renewable generation, energy storage, distributed generation, demand management) that describe real-time power flow and predict reliability do not exist Ability to see and understand the grid at interconnections scale are limited; wide area grid performance is not accessible, transparent so can’t optimize supply and demand across limited service areas Transparent real-time monitoring and operation currently not in place Large-scale wind generation introduces significant variability Large scale electric energy storage capability is limited – pumped hydro, flywheels, electro-chemical systems connected to and supporting the power grid Vulnerability and Resiliency Susceptible to cyber, other threats – can we prevent, respond to threats? Resilient to catastrophic events – can we rapidly recover? 4

Power System Elements

Problems on the electrical grid Inadequacy of current control center functions Slow, not able to keep up with the change of the grid Static, no dynamic information for real-time operations Computational Issues with today’s grid operations Real-time grid view: static Not able to capture grid dynamics Low computational efficiency: not keep up with system changing No use of high-performance computing architectures

Problems on the electrical grid

The need for HPC on the electrical grid Major HPC Architectures Shared-memory architecture for extensive data sharing and un-uniform data access Distributed-memory architecture for less data sharing and uniform data access Hybrid architecture - re-configurable architecture: e. g. FPGA + shared-memory Performance of Parallel Algorithms/Programs Speedup/Scalability Reliability

The need for HPC on the electrical grid Parallel Computing is essential Only explicitly parallelized algorithms can take advantage of multi-core parallel computers Parallel Computing is “an art” Parallelization approaches are problem-dependent Computing implementation needs to consider “good match” of computing architecture and the problems

Today’s Electrical Grid Operations Paradigm Normal operations slow static • Asset underutilization • Limited market opportunities • Lead to emergency operations Emergency operations • Blackouts and cascading failures Operator SCADA ~ seconds State Estimation ~ minutes Off-line Transient/Voltage Stability Analysis seasonal Contingency Analysis ~ minutes Market Operation Ratings & Limits Violations ~ hours Constrained solutions

Trends Impacting Control System Security Open Protocols Open industry standard protocols are replacing vendorspecific proprietary communication protocols General Purpose Computing Equipment and Software Standardized computational platforms increasingly used to support control system applications Interconnected to Other Systems Connections with enterprise networks to obtain productivity improvements and information sharing Reliance on External Communications Increasing use of public telecommunication systems, the Internet, and wireless for control system communications Increased Capability of Field Equipment “Smart” sensors and controls with enhanced capability and functionality

The Emerging Cyber Threat Industry has long history of planning for and coping with natural disasters and other reliability events Through industry standard operating procedures, there is much effort expended to reduce likelihood of cascading outages leading to widespread blackouts Historically, cyber security focused on countering unstructured adversaries e. g. , individuals, untargeted malicious software, human error Very little protection against structured adversaries intent on exploiting vulnerabilities to maximize consequences e. g. , terrorist groups, organized crime, nation states Insider threat remains very challenging, can be used as part of structured threat vector New possibilities for widespread sustained outages resulting from cyber attack are now being contemplated But industry still not ready to cope with this threat

The need for HPC: State Estimation Power system State Estimation (PSE) Given: power grid topological information, telemetry on line flows, bus injections or bus voltages Compute: a reliable estimate of the system state (bus voltages), validate model structure and parameter values Calculated using Weighted Least-Squares (WLS) method WLS: minimize Where r = z - h(x), and r is the residual vector, x is the system state, z is a vector of measured quantities, h is a vector function, wi is the weight for residual ri and W is as diagonal matrix. This is a non-linear problem, which is solved using the Newton-Raphson iterative procedure

The need for HPC: State Estimation PSE Every iteration of the method requires solving a large set of sparse linear equations Sparse matrices are derived from the topology of the power grid being analyzed The number of non-zeros per row varies greatly and the matrix is badly conditioned The set of linear equations can be solved using direct solvers such as sparse LU factorization or iterative solvers such as sparse Conjugate Gradient (CG) PSE is a critical element of the software used by power grid control centers Under real-time constraints (< 10 seconds) Commercial PSE solvers are not commonly parallel

Power System State Estimation August 14, 2003 Blackout Situational Awareness? August 13, 2003 Normal Do we know what really happened? Could it be prevented? Source: NOAA/DMSP

Model Validation Need and Challenges Recorded system dynamics vs. simulation results: California and Oregon Intertie (COI) real power flow during the August 10, 1996 event

The need for HPC: Multithreaded Platforms for Contingency Analysis Growing class of scientific applications is becoming memory-bound Many scientific applications exhibit irregular memory access patterns CPU and memory technology trends indicate that the situation will not improve anytime soon Multithreaded architectures offer an appealing alternative for irregular applications Processors tolerate memory access latencies by switching execution context between multiple hardware threads Examples of such architectures are the Cray MTA-2 and XMT systems and the Sun Niagara Latency tolerance mechanisms should improve the performance of irregular, data-intensive applications with abundant fine-grained parallelism

Role of Contingency Analysis From “N-1” to “N-x” To improve situational awareness From Balancing Authorities to a Wide Area Example: 35 BAs in west Further require “N-x” CA To better understand cascading failures N-x Contingency Analysis Result in a large number of cases. “N 5” 1020 cases for the west =~ 1020 seconds + lots of data Needs: better contingency selection and post-processing

Looking forward: PNNL’s vision of advanced contingency analysis Industry Need: Multi-failure (N-x) massive contingency analysis Issue: Massive number of cases Massive amount of data Need: Smart case selection Operator-oriented data processing 10 Z cases Major Tasks: 10 Y cases 10 X cases Contingency Analysis Prediction Our Solution: Data Trendin g Graph Theory Expected Outcomes: Contingency Index Sorting past Visual Analytics & Graph Trending Interactive Graphing An advanced tool for contingency analysis to support the operation of the nation’s critical infrastructures, e. g. power grids Analysis capabilities through generic implementation of algorithms on Cray XMT Thread. Storm processors, applicable to other problems now Time future

Tool Suites for Advanced Power System Simulation SCADA Measurements Phasor Measurements Parallel state estimation Dynamic state estimation Parallel contingency analysis On-line voltage stability Look-ahead dynamic simulation Dyn. contingency analysis Market Monitoring Visualization National Level: monitoring, planning, design Utility Level Applicable to current grid operations Next-generation grid operation tools

Looking forward: PNNL’s long-term vision for electrical grid operations Normal operations fast dynamic • Optimized asset utilization • Enabled regular & ancillary markets • Predict emergencies Emergency operations • Prevent/mitigate failures SCADA & Phasor ~ (sub)seconds Dynamic State Estimation Operator Dynamic CA, Violations On-line TSA/VSA ~ (sub)seconds Dynamic Ratings & Limits Real-Time Constrained Optimized Market solutions Operation < 1 hour

Electricity Infrastructure Operational Center at PNNL Use HPC to analyze sensor data for real-time grid monitoring, prediction, and operation: data-driven models, stochastic simulations, visual analytic tools, secure sensor network infrastructure, data provenance 22

QUESTIONS?