IT Application Monitoring Using PI AF and Notifications

IT Application Monitoring Using PI AF and Notifications Josiah Thoen - AEP © 2008 OSIsoft, Inc. | Company Confidential

AEP – Who We Are – By Assets • One of the largest U. S. electricity generators ( ~ 38, 000 MWs) with a significant cost advantage in many of our market areas • Largest consumer of coal in the Western Hemisphere • Operations within four RTO’s • PJM • SPP • ERCOT • MISO • A leading consumer of natural gas • Major wind power developer (#3 in U. S. in 2005) • 39, 000 miles of transmission • 186, 000 miles of distribution • Coal transportation • 7, 000 railcars • 2, 230 barges and 53 towboats • 5 million customers © 2008 OSIsoft, Inc. | Company Confidential 2

AEP IT - Service Performance l Monitoring AEP IT Assets for over 7 years – Servers • Hardware, Operating Systems (Windows and Unix) – Middleware Platforms • IIS, BEA Web. Logic, etc. – Applications • In house custom applications • Vendor Applications – Black. Berry Server, Lotus Notes, etc. © 2008 OSIsoft, Inc. | Company Confidential 3

AEP IT - PI Infrastructure l PI Server Collective – 2 Servers • 8 CPU, 8 gb RAM, 85 gb SAN Drive Space – Software: l Analytics Server – 2 CPU, 2 gb RAM, 30 gb Local Drive Space – Software: • ACE (Primary) – PI Server • AF & Notification • MS SQL Express • ACE (Backup) l Data Acquisition Node – 2 CPU, 2 gb RAM, 30 gb Local Drive Space – Software: • IT Monitor Interfaces • Custom PISDK monitoring services l Stats: – – – Tags: 94, 049 Modules: 54, 076 AF Elements: ~10, 000 AF Element Templates: ~25 Notifications: 331 Data Storage: • 1 gb every 23 days © 2008 OSIsoft, Inc. | Company Confidential 4

AEP IT and PI – the beginning l PI IT Monitor Pilot – Customer • IT Application Support Teams – Primary • IT Operations – Secondary • Business Units - Secondary – Purpose • Production Server and Application visibility – Result • Infrastructure monitoring approach • Application Monitoring © 2008 OSIsoft, Inc. | Company Confidential 5

AEP IT and PI - Problems l Time Series data is not relational – Convoluted naming conventions and overloaded tag attributes • Network > Datacenter > Server > Components > Application > Middleware > Database > and back l Maintenance nightmare – Everything is manual – Numerous duplicate displays l No Event Management – All this data but no way to knowing when something is broken. © 2008 OSIsoft, Inc. | Company Confidential 6

AEP IT and PI - Solutions l Problem: Time Series data is not relational – Solution: AF – Relational Meta Data on top of Time Series data. l Problem: Maintenance nightmare – Solution: AF – Template based management of elements and their attributes l Problem: No Event Management – Solution: PI Notifications – Template based triggered event delivery © 2008 OSIsoft, Inc. | Company Confidential 7

AF Implementation l Web. Methods Broker. Client Queue. Length monitoring – Software. AG Web. Methods • SOA, B 2 B messaging platform • Pub-Sub Message Brokering – Publisher sends a message to a Broker. – Broker delivers the message to one or more Subscribers of that message. • Used in many Tier 1 and 2 application services and business processes – Facilitates communication with external counter parties – Used Internally to send data between applications. © 2008 OSIsoft, Inc. | Company Confidential 8

Business Need l Monitor the Broker to make sure that client queues are not getting full. – Publishing clients send messages to the Broker – Broker delivers the message to subscribing clients queue. – Subscribing clients retrieve messages from their queue. – Implications of failure: • Client may be unavailable. • Broker may be having problems. © 2008 OSIsoft, Inc. | Company Confidential 9

Functional Requirements l Store the Broker Client Info data. l Alerting: – Warn when queue contains 25+ messages. – Alarm when queue contains 75+ messages. – Alerts must send a Clear message when the alerting condition is gone. l Alerting Recipient: – Email notification to those interested. – AEP IT Command Center: • 24 x 7 Support Center. • All Events go to Central Event Management System. © 2008 OSIsoft, Inc. | Company Confidential 10

Monitoring Process l Data Acquisition – PISDK Custom Monitor – Invoke a service within the Web. Methods environment every 60 seconds. • Parse data. • Do exception testing • Send data to PI. – Automatically Add new Broker Clients to AF as they are discovered. – Automatically create tags for Broker Clients if they do not exist. © 2008 OSIsoft, Inc. | Company Confidential 11

AF Element Templates l Broker. Client. Info: – Defines the attributes of a Broker. Client as it exists in Web. Methods. – Each Attribute references a PI Point. • Appname = \%Server%Broker. Client. Info_%. . Element%_%. . El ement%_%Element%_%Attribute%_Info • Queue. Length = \%Server%Broker. Client. Info_%. . Element%_%. . El ement%_%Element%_%Attribute%_Count © 2008 OSIsoft, Inc. | Company Confidential 12

Broker. Client. Info Template © 2008 OSIsoft, Inc. | Company Confidential 13

Broker. Client. Info Element © 2008 OSIsoft, Inc. | Company Confidential 14

AF Element Structure l Web. Methods – Brokers • <Broker. Server> – <Broker> » <Broker. Client> • <Broker. Server> – <Broker> » <Broker. Client> © 2008 OSIsoft, Inc. | Company Confidential 15

AF Element Structure © 2008 OSIsoft, Inc. | Company Confidential 16

Notification Process - Requirements l Notification is based on the Queue. Length attribute of a Broker. Client. Info. – Needs to be customizable for each client. – Needs to be disabled for certain clients. – Default Warning = 25. – Default Alarm = 75. © 2008 OSIsoft, Inc. | Company Confidential 17

Notification Process - Template l Attribute Template: © 2008 OSIsoft, Inc. | Company Confidential 18

Notification Process – Warn Trigger IF Notification enabled And Warn enabled And Queue. Length >= Warn value And Queue. Length < Alarm value © 2008 OSIsoft, Inc. | Company Confidential 19

Notification Process – Alarm Trigger IF Notification enabled And Alarm enabled And Queue. Length >= Alarm value © 2008 OSIsoft, Inc. | Company Confidential 20

Delivery Channel l Web. Service – Sends Target and Trigger value to our Event. Management Web Service. • Target = AFElement Path • Value = CRITICAL, WARN, OK • Event Management Web Service – Gets additional details from AF using the Target. – Creates an Event Management message and sends it to our Event Management server. – Closes the previous event using the Event. Id attribute from the AF element. – Sends new Event. Id back to AF Targets Event. Id attribute. © 2008 OSIsoft, Inc. | Company Confidential 21

Event Management View l BMC Impact Explorer l Transpara © 2008 OSIsoft, Inc. | Company Confidential 22