Nagios Implementation Case Eastman Kodak Company Eric Loyd

Nagios Implementation Case: Eastman Kodak Company Eric Loyd Founder & CEO Bitnetix Incorporated eric@bitnetix. com www. bitnetix. com 877. BITNETIX

About Eric Loyd and Bitnetix Founder and CEO of Bitnetix Incorporated VOIP services and IT/network consulting 25 Years in IT at places like Eastman Kodak Frontier Communications Global Crossing Bitnetix started its seventh year in July, 2012 Digital Rochester GREAT Award Finalist in Communications Technology Using Nagios to monitor our client equipment, VOIP platform, and still using it at Kodak since 2004 © 2012 Bitnetix Incorporated 2

A History of Eastman Kodak’s kodak. com Web Server Infrastructure (non-confidential)

History of kodak. com Pre-2004 Machines located in Rochester, NY Public Apache servers Reverse proxy Apache servers Application servers (ATG/Dynamo, Tomcat, etc) Database boxes, Production Support, etc. 2004 – Moved ~80 machines from ROC -> ? ? ? ROC <-> ? ? ? Firewalls Bandwidth requirements Minimal user impact Flipped the switch, went live © 2012 Bitnetix Incorporated 4

History of kodak. com Some of the things kodak. com did at the time Consumer store and product information B 2 B portal and wholesaler purchasing “Picture Of The Day” (www. kodak. com/go/potd) Warranty registration Photo lab calibration strips “Phone home” reports for printers, docks, cameras, etc Software/firmware updates Corporate press releases, bios, and regulatory information Reverse proxy for internal information through secure channels Dozens of sitelets for products and campaigns © 2012 Bitnetix Incorporated 5

Why Kodak Chose Nagios to Monitor kodak. com

Why Nagios? No centralized corporate monitoring software Nothing to compete with internally Nothing to build on, either Cost No additional cost beyond existing human resources Framework Nagios worked with firewalls without needing agents Leverage SSH, HTTP and other remote protocols Custom checks and notifications (very important) © 2012 Bitnetix Incorporated 7

Initial Hurdles in the New Complex Server Environment

kodak. com Network © 2012 Bitnetix Incorporated

Initial hurdles Firewalls Public load balancers on external Internet IPs Public Apaches in Zone 1, Kodak network Reverse proxy, app servers in Zone 2, semi-secure Nagios machine in internal Zone 3, most secure Complex “top” and “bottom” checks for web site Is the site working from the user’s perspective (top)? From the application side (bottom)? How to separate apparent from actual failure © 2012 Bitnetix Incorporated 10

Initial hurdles No Internal Nagios Knowledge It was a contractor who set up Nagios (me) Contractors typically have a finite lifespan at Kodak Contractor made custom checks, event handlers, and all Nagios configurations. Uh-oh… Escalation and Paging Screw it – let’s email everyone, every time and let Thunderbird sort it all out Paging done via texting gateway email address Which means email gateway failure = notification failure Twitter API as backup / current primary notification © 2012 Bitnetix Incorporated 11

SSH to Remote Servers

SSH to the rescue One user, one key, infinite access Software apps run as second user, with SSH auth Additional robot accounts can be added at any time Wrap existing checks in an SSH shell Provides additional control, error handling, reporting Allows all checks to submit results to SQL database SQL Database Side Note – all custom scripts executed CLI Perl code that locked a file, logged to it, and unlocked it. A Perl cron job woke up every 5 minutes, locked the file, read it, pushed things to Oracle, unlocked, and deleted log file. A second cron pruned Oracle daily to 400 days of data and collapsed checks older than 30 days so that successive checks with the same status were removed. © 2012 Bitnetix Incorporated 13

Managing Nagios Configuration Files

Configuration Management SCCS Solaris’s “poor man’s CVS” Pre-installed, no additional cost, existing expertise Current configuration is managed through SVN Rsync – the workhorse to move config files Configuration Repository and Push (CRa. P) directory Cfengine Local versus remote execution Post-install, ignore pid files, deploy/restart, etc. Makefile – the “CLI” to the entire process © 2012 Bitnetix Incorporated 15

Common Event Handler

Common Event Handler EKrestart – That Which Does Setup • • • Arguments Conversions do_soft/hard? do_something? do_restart • • • Lock, logs, SQL send_nagios SSH to remote Remote EKrestart Process args do_<service> send_nagios Unlock, log, SQL Terminate © 2012 Bitnetix Incorporated do_<service> • • • Locks (level 2) Instance mapping Port mapping App restart Email & log Exit 17

A Closer Look at EKrestart #!/bin/sh PATH=. . . [ "$1" = "-r" ] && client_code host="$1" service="$2" base. Service=`echo $service | awk -F: '{print $1}'` state="$3" type="$4" tries="$5" perfdata="$6" class="<based on machine name, e. g. , x-y-CLASS-nnn. kodak. com>" number="<based on machine name, e. g. , x-y-class-NNN. kodak. com>" case "$state" in OK) do_fixit; ; WARNING) do_nothing; ; UNKNOWN) do_nothing; CRITICAL) do_something; *) do_nothing; esac © 2012 Bitnetix Incorporated 18

A Closer Look at EKrestart do_fixit() { case "$base. Service" in Workers) do_restart; ; *) do_nothing; ; esac } do_nothing() { $debug && echo "$service is in $state ($type) for $tries. " } do_something() { case "$type" in SOFT) do_soft; ; HARD) do_restart; ; *) do_nothing; ; esac } do_soft() { case "$tries" in 3, 4, 5) do_restart; ; *) do_nothing; ; esac } # Take action before it's too late? # Hard CRITICAL - Our last chance to take action # Okay, let's restart it before it goes hard # Don't restart yet © 2012 Bitnetix Incorporated 19

A Closer Look at EKrestart do_restart() { # <figure some stuff out, set up lock files, send_nagios, log to SQL, etc> ssh $machine <EKrestart> -r do_$service <parameters> # <tear down, unlock, close log, send_nagios, log to SQL, etc> exit } # On the client side, we use the same EKretart script, but start at client_code() { host=`hostname` function="$2" service="$3" # (etc) eval $function exit } # Example function do_Dynamo() { # lock file processing # turn off new sessions, wean existing ones # /etc/init. d/restart_dynamo_$instance # tear down return } © 2012 Bitnetix Incorporated 20

Integrating Nagios into Operational Procedures

Integration with Operations Homebrew API nchart, send_nagios, nlog – all portable to other installations of Nagios on other machines Integrate with start/stop scripts Lock files. Lots of lock files! TOO MANY lock files!! The “Rippler” Leverage EKrestart, cron, and send_nagios Pager / Twitter and lots of private twitter feeds Inter-group notifications Predominately with procmail © 2012 Bitnetix Incorporated 22

Predictive Failure Recovery and a Good Night’s Sleep

Predictive Failure Recovery On ATG/Dynamo (and other) services do_soft triggers do_restart on third failure do_hard always triggers restart Notifications on fourth failure Escalation to pager only on fifth notification Nagios has time to restart things that are bad, or are going bad, prior to sending out notifications Service check dependencies allow us to know whether it’s a bad application, server, or user experience Twitter – follow private tweets with smartphone, use apps to acknowledge problems, and get an even better night’s sleep!! © 2012 Bitnetix Incorporated 24

Questions Eric Loyd Founder & CEO Bitnetix Incorporated eric@bitnetix. com www. bitnetix. com 877. BITNETIX