NOC Theory and Practice Part II The Nuts

NOC Theory and Practice Part II: The Nuts and Bolts (The Practice Of Making a Good NOC) NOC Theory and Practice NANOG 24 - Miami, FL - TCB

DISCLAIMER I am not affiliated with any corporation. My views are my own. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Intro - What’s In A NOC? • Purpose - What Type of Company Are You? • People will always be your key asset. • Systems make the job of people easier. • Managers assist in identifying gaps in both. • Processes allow for flexibility in operations while still maintaining standards. • And, last but not least, the facility is where you. NOCdo your work. Theory and Practice - NANOG 24 - Miami, FL - TCB

The Facility - Think Data Center • Location, location. • Security • Environmental Concerns w This applies to both your inside and outside environment. • Ergonomics • Reliability • Sources of Labor NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

The Goals of a NOC Our conception is: • At the bare minimum, you will receive requests from your customers • You will process those requests in some fashion (engineering, handoff, . . . ) • You will communicate resolution to the customer, and/or keep customer updated on status NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Goals II • NOC as first, second, or third tier support. w Conception of NOC as “do everything” group w The engineers should be smart at designing, configuring, etc. - NOC should be divided into categories of engineers able to resolve issues w A smart issue routing process makes this a very effective model NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

The Process • Receive events from your “inputs” (phone calls, network monitoring, etc) • Process inputs into a problem to be resolved • Route the problem to the correct individual • Resolve the problem as appropriate • Communicate status continually. • Record the problem for later review and training. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

The Key to Success Is By constant re-examination of your operation, your employees, and what inputs you are getting, you can reach incredible efficiency. Communicate this to other teams and your operations will prosper. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

The Facility (cont. ) • Security applies to keeping other people out, but it also applies to making a safe arrival for your employees, at any hour. • Environmental concerns: Just like a datacenter, consider climate control and earthquake/act of god resistance. Disaster planning rules should be in effect. • Sources of labor: Not just engineers, but builders. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

A Word on Portability • As Matt suggests, portability is critical. • Obvious resources: Computers, communications, phones. • Not obvious resources: Vehicles and quick transport/assembly, good cases for shipping or relocation, breakdown/reassembly schedules. • Multiple everything. • Make sure your resources are distributed. If your network is large, your footprint of mobility should be large. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Hiring the Right People • Skillset w but more importantly, flexibility • Stress • Scheduling • Stability NOCs are hard things to staff appropriately. And good people are hard to retain. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Hiring (cont. ) • Make sure they have lives. But know what you can’t ask. • Can they talk? How are their soft skills? • Do they get flustered easily? And remember, org charts suck. Classify people according to their skills, not according to some hierarchy which has no meaning outside of some middle-management mind. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Keeping Your People Happy • Environment • Ergonomics • Creature Comforts w We’re not kidding. Give some thought to a 24 x 7 outlook. This means beds, showers, food, and climate. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Environment - People • Climate controlled centers get cold at night. Make sure you have 24 H control over your space. • When thinking about climate, keep in mind your working conditions are slightly different in a NOC. • 2: 00 AM to 4: 00 AM are the critical hours. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Ergonomics - People • Hire a good ergonomic consultant. • Positions of lights, phones, keyboards, monitors, pens - all this matters. • Observation is key. Your engineers will show you what the problems are. (Cameras) • Seating. If you’re in a chair for eight hours at a stretch, don’t you want it to be a comfortable one? • Screen space. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

About People. About NOCs. • It’s a messy business. It’s high stress. It’s easy to let things get out of hand. • Good management qualities in a NOC: w cool heads w methodical approach to solving problems w having been on the customer side w micromanagement sucks. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Let’s go over retention. • Retention of NOC staff is 100% harder. Very few people live for this kind of job. • So, innovate. Think outside the box. • People want to grow. Know what your people want to do when they grow up. • Remember your core focus. Do operations, not sales, or account services, or anything else. • At the same time, make sure your folks are informed on everything. Everything comes out in the wash. All the dirty clothes come to NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Interviewing NOC folks • You are interested in three things. w The candidate’s willingness to learn. w The candidate’s 1 -2 year outlook on their career. w The candidate’s ability to work funny schedules. • They are interested in three things. w Will you offer them growth potential? w Can you give them enough perks to enjoy their work? w Do you have your stuff together? § Are the procedures well-documented? § Will they be called in outside their shift? § Will they ever wonder who to go to? NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Productivity • Momentum is key. Keeping your people busy is the most important thing towards maintaining productivity. • The other component to momentum is keeping them working on different things. This makes them better engineers, and helps feed ego. • “Everything at your fingertips” - your engineers should never have to make a customer wait more than four minutes on hold for anything. If it has to wait longer than that, tell the customer why, and mark it as something to fix. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

The View of the Customer • A customer wants w Professionalism: your engineers must treat the customer as if they are paying you a lot of money and the engineer wants the customer to continue to do that. w ETR: Your customer wants to know when the problem will be fixed. “I don’t know” is not an acceptable answer. The ETR is the first thing you give to a customer and the one thing you update him on at least every hour (or less if they prefer). NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

The View of the Customer II w A description: Concise or not, the customer wants to know what the problem appears to be. Remember your focus. It is not (typically) your responsibility to engineer your customer’s network for maximum reliability. Your responsibility is the operation of that network. The only thing you should do as a NOC is fix problems when they arise in the most expedient and professional manner possible. A customer wants to be. NOC comfortable, and assured that Theory and Practice - NANOG 24 - Miami, FL - TCB

Things we never, EVER allow a customer to hear • I don’t know when it will be fixed. • I can’t help you. • I’m not qualified to answer that question. • Let me transfer you to someone else. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

It’s not always ideal. But • Where possible, the first person to take the call should be the person who follows it to completion. The customer hates being thrown between different engineers - but not having an answer is worse. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

A Little About Systems • Don’t just focus on ticketing. • Make sure your people have information on everything they need to operate. • If all the architects fell off the planet today, could you rebuild? • It’s about agility. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Policies and Procedures • If you don’t write it down, it isn’t a procedure or a policy. There are no unwritten rules in a NOC. People cycle. • Focus on procedures that affect the handling of a customer. The soft skills are usually the hardest to learn. • Full disclosure? • Troubleshooting Techniques NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

And a word on incidents. • Incidents, tickets, call it what you will. • Whatever happens in the NOC: w gets summarized to the account exec if the issue is of appropriate severity w gets recorded - nothing EVER gets deleted from a customer log w gets archived w gets resolved - how much time will you spend on issues that never get fixed? NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Some final notes - Conclusion • It’s easy to throw a bunch of people in a room. It’s hard to build an efficient, productive, and well-oiled NOC (the same goes for networks) • Record everything. Write everything down. • Try to remain as flexible as possible. • Look to your employees for cues on what their ideal environment should be. • Remember your focus, and know your customer. • Know what not to say. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Credits As far as the practice of NOC building goes, we must thank: • herb@tomobiki. urusei. net - thanks for great private discussions on NOC building • nanog subscribers - for talking about operational stuff NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

And finally • Sean Donelan - for giving network operations folks an impetus for existing (due to his backhoe skills) • Shouts to EFNet IRC #nanog: you know who you are NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Part III: Evaluation Criteria • Social engineer your own NOC. • We like to rate NOC folks on: w Do they sound professional? Do they have the answers? w Do they follow us all the way through to resolution? w Can they tackle difficult as well as simple problems? NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Part III: Evaluation Criteria (cont). • How many calls a day does your NOC take? • Do you survey your customers, Cisco-TAC style? • The three ‘P’s: w Perception w Passion (Emotion) w Presentation NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Evaluation Criteria • They (customer) must perceive you as the people that will help. • You (NOC) must present your solution with total assurance that you can handle the problem. • You (NOC) must have passion in dealing with the customer. Make it friendly, even if it’s a network down emergency. It will come back to pay off. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB

Remember. • Your goal is to be professional, calm, cool and collected, even in the face of serious adversity. • If you don’t have an answer, you know where to get it and how long it will take. • You have to communicate to the customer and keep the customer’s viewpoint in mind, no matter what. • As a NOC manager/builder, you have to make sure your people have the right tools. NOC Theory and Practice - NANOG 24 - Miami, FL - TCB