Outsourcing Res Net One Institutions Experience Ithaca College

Outsourcing Res. Net One Institution’s Experience Ithaca College RESIDENCE HALL COMPUTER NETWORK Copyright David Weil, Ithaca College, 2005. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Welcome! • A copy of this presentation is also available at: www. ithaca. edu/staff/dweil David Weil Director of Web, Systems and Departmental Services Information Technology Services Ithaca College dweil@ithaca. edu

Ithaca College • Private, residential undergraduate liberal arts college in upstate New York • Comprehensive, masters one institution • 5 schools & an Faculty interdisciplinary division Staff • Over 100 degree Students programs In Residence Halls 440 960 6, 200 4, 400

Information Technology Services Director of ITS Information Systems and Services • HR, ERP, SIS • 19 people Network and Communication Services • Networks • Phones • 9 people Technology and Instructional Support Services • Field Service • User Support • 14 people Web, Systems & Departmental Services • Web Dev. • Sys Admin • ITS Business • 18 people

Residence Hall Network • Phased in over 3 years in early to mid 90’s • “One port per pillow” • Today - 53 residential halls with 4, 400 ports

ITS provided Res. Net • Flat network, 10/100 Mbps to the desktop • “Free” – no additional charge for the students • Shared main college Internet connection – Res. Net bandwidth was allocated as a percentage of our main connection – Eventually allocated bandwidth by time of day (shifted more bandwidth to Res. Net at night, less during day) – Shaped Internet traffic (allocated very little bandwidth to identified P 2 P traffic)

Bandwidth Allocated to Res. Net 1998 3 Mb 1999 6 Mb to 12 Mb 2000 16 Mb 2001 24 Mb Spring 2004 38 Mb

Res. Net Pressures Build & Change At first… By 2000 it was about…. And then…. • It was all about support to get student machines configured and connected • Support & • Bandwidth & • P 2 P Management • File sharing (RIAA) & • Bandwidth & • P 2 P Management & • Security - Viruses and Worms, and their impact on network performance

Fall 2003 – It boils over • Viruses, Worms, and other self-propagating nasties – And their significant impact on network performance • • RIAA, MPAA, ESA… Insatiable Bandwidth Desires P 2 P “whack-a-mole” Internal debate regarding the purpose of Res. Net and use of our resources, and what is appropriate use of Res. Net (games, P 2 P, etc) • Staff needed for other projects

Fall 2003 – It boils over • August 2003 Blaster and Nachi strike and Res. Net could not handle the traffic from infected machines • On the Friday before opening weekend we turned off the entire Res. Net to prevent further infections – It ended up being off for 7 days while we developed procedures to deal with the situation and worked with students to get their machines protected and cleaned

Why was it so bad? • Flat network – The entire Res. Net was one subnet, no way to contain an infected machine • Full 10/100 Mbps Ethernet speeds – Allowed viruses and worms to spread like wildfire – An unprotected machine would be infected within 30 seconds of connecting to Res. Net • No firewall protection for Res. Net machines • Individuals were not used to having to worry about computer security or patching

And once past opening weekend… • We continued to take incremental steps to address many of the performance, support and network architecture issues • Had to remain vigilant and constantly turn off ports of infected machines – One or two infected machines would have significant impact on network performance • Saw a significant increase in P 2 P usage • RIAA, MPAA complaints increased significantly • Continued to experience a huge drain on support, network and other ITS staff – many projects were put on hold • Bottom-line: throughout Fall 2003 Res. Net performance was sporadic and ITS staff stretched thin

Students (and Parents) React • Student created “Res. Net Sucks” Website with on-line petition and call-to-action • Editorials and letters in student newspaper • Students deliver a petition to ITS with 1, 175 signatures demanding a better Res. Net • And of course, angry emails and phone calls to the helpdesk, ITS management, and the college administration

www. evanm. com/resnet/indexold. htm

www. ithaca. edu/ithacan/articles/0311/06/news/LSresnet_revol. htm

Something had to change • If we were to continue to support Res. Net ourselves, we would need to: – – Significantly change and upgrade the network Install better traffic monitoring and compliance systems Provide additional network and support staff Find additional space for student computer repairs • However, we were concerned about the ability to find funding and staff resources to support this for the long term

Thoughts turned to outsourcing • Initial requirements: – Tiered service plan including a base no-cost option for the students – On-site support – Increased bandwidth – Robust network architecture that could better withstand viruses, Trojans and worms – A true partner interested in a collaborative relationship and who understands higher education

We solicited proposals • A major nation-wide telecom • The local cable-TV / broadband franchise • A small company specializing in supporting residence hall networks

The nation-wide telecom • Limited experience with this type of network and service • Limited experience with the academic environment

Local cable-TV / broadband provider • Would provide a significant discount for their broadband service – Based upon a guarantee # of subscribers from us – The college would need to make one payment to them, and have students pay the college for the service • Would not provide tiered or free service options • No direct on-site support

Local cable-TV / broadband provider • Would need to build an entirely new network infrastructure in the residence halls – Current cable-TV infrastructure could not support broadband service – College would be unable to support this cable-TV based network should the partnership end

Small Residence Hall Provider • Agreed to meet all of our requirements – Tiered service options (including “free” option) – On-site support – Increased bandwidth / network redesign – Understood residence hall networks and the academic culture • CEO very effective at talking with the students about their concerns and issues – Interested in a partnership-type of relationship

Small Residence Hall Provider • Concerns: – Located in Austin Texas with only two current customers – We would be their largest deployment (and farthest away) – Small company, operating “lean”

Selected Apogee Telecom • We felt that these concerns were manageable • Selected the small residence hall provider, Apogee Telecom • Felt they were the best overall fit for us

What we outsourced… • Redesign of the Residence Hall network • Network monitoring, troubleshooting, repair and replacement • Custom service levels and prices that met our particular needs and budgets • Management of the account sign up process • Student billing • Marketing and PR

What we outsourced… • An on-site technician for troubleshooting student network issues • A call center open 7 days-a-week • The ISP of record for DMCA and other complaints – Work collaboratively with the college for responding to DMCA complaints – College gets copies of all notices and still may impose judicial sanctions

Apogee Telecom – The Network • Redesigned our residence hall network and “over built it” – Provided 3. 5 times the bandwidth (OC-3, 155 Mbps total) – Used our switches, but replaced most other components – Installed two Allot Communications Net. Enforcers to control network traffic and bandwidth allocation – Significantly limited the number of computers on any one subnet • Students allocated bandwidth based upon their service level – Could use that bandwidth for whatever they wanted (any “legal use”)

Fall 04 Apogee Service Plans Plan Cost per semester On Campus Resources Basic Free High speed* 64 Kbps Bronze $49. 95 High speed 1 Mbps 128 Kbps Gold $69. 95 High speed 3 Mbps 256 Kbps Internet Access Download Upload *High Speed = 10/100 Mbps to all ithaca. edu resources

Apogee Service Costs • The College pays Apogee a base fee per month to cover: • • 25 Mbps administrative internet service “Free” 64 Kbps basic-level Internet service for all students High-speed access to on-campus resources for all students Apogee help desk support for Res. Net students • This base fee is roughly equal to the amount we used to pay to our old campus network provider for the T 3 service – so in the end, there was no increase in costs to the college

The Implementation March – August 2004

Timeline Contract signed Communicated plans to campus Technical site visits and design work Cable trenching Implementation sub-teams created Support procedures created Student antivirus strategy determined Network design finalized Onsite support staff interviewed and hired Legal / policy issues agreed upon Site coordinator starts Network infrastructure upgraded Res. Net by Apogee Live March April May June July August

Campus Communication • Apogee CEO came to campus and met with administrators, student government, and conducted an open town hall meeting • Announcement letters sent to students and parents detailing changes • Ads placed in the student paper • On-going dialogs with student leaders and parents to respond to concerns…

www. evanm. com/resnet/index 2. htm

A petition to ITS with 1, 175 signatures responding to a February 26, 2004 student newspaper article. They request that we work with a vendor such as Apogee to fix Res. Net and not impose any additional fees for what they consider a basic and necessary service.

Account Sign-ups • No matter what service level, people need to sign up for an Apogee Res. Net account • Advance sign-up mailings sent over the summer to all students living in residence halls • Freshman sign-ups at orientation (June/July) • Could sign up via Web, phone or paper form • 2, 892 people signed up in advance (67%)

Fall 04 Apogee Subscriptions Basic 1, 937 Bronze 1, 120 Gold 1, 214 Total Subscribers 4, 271 As of December 28, 2004

Activation Process • A student plugs their computer into a Res. Net port • Automatically directed to Apogee login page – Includes a link to the account sign-up page if needed • Student activates computer using Apogee supplied username and password • Only need to do this once per semester

Cutover and Move In – Here’s Murphy • Last of the new equipment cutovers occurred shortly before opening weekend • Major fiber-cut 60 miles away took down service the week before opening weekend • A few days before opening weekend a network problem developed where students were not getting their subscribed-to level of service – Would result in a MAJOR PR problem if it didn’t work! – Apogee was able to put a workaround in place late at night on the Friday of opening weekend

Opening Weekend • Our helpdesk and offices were quiet (it was strange!) • Apogee’s call center was busy, on-site staff were very busy and some students had to wait a few days for onsite help • Overall things went smoothly – Few if any complaints ever came to us regarding problems • ITS staff were unsettled by not having the ability to directly monitor Res. Net performance or call volume to the Apogee support center

Early Fall • A few scattered equipment problems • One particular residence hall experienced a few interruptions due to someone repeatedly plugging an improperly configured wireless router into the network • Overall, things went well

Other (minor) issues that came up along the way… • Different expectations for who would be on-site during opening weekend • ITS staff and management had some trouble “letting go”….

How’s it working? • ITS has had time to focus on other things – “This has been my most productive fall in a very long time” • Students seem to like the service – Apogee conducted a customer survey in late fall. Out of 4, 193 survey’s sent out, received 1, 067 responses (25. 4%)

“Connection is stable and reliable” Total responses: 1, 067

“I am satisfied with the speed I signed up for” Total responses: 1, 067

“The Apogee call-center support staff was knowledgeable in troubleshooting my issue” Total responses: 425

And even our friend Evan, seems to like the service….

Where do we go from here….

Spring 2005 • Continuation “as-is” • Continue to optimize network settings and support procedures

Future • Upgrade a few more network components • Look to add possible other services in the future: – Provide subscription-based TV-like services over the network – Wireless in residence halls – Better automatic virus and patching updating and quarantine services • Revisit price-points

Is Outsourcing Res. Net For You? • Depends on a number of factors such as: – In-house staffing levels and expertise – Your view of the roll / mission of Res. Net – Ability to continually refine the network and services to meet changing needs – Cost structure • It’s really a decision that must fit with the values, expectations, and unique situation of each institution • For us, so far the answer has been yes

Other Resources • EDUCAUSE Live! Broadcast – Two other team members from Ithaca College will discuss more about our decision and process – 1 pm (Eastern) Thursday, January 27 th – Details at will be posted at www. educause. edu • The Chronicle of Higher Education – “Outsourcing the Dorm Network” – December 3, 2004

Outsourcing Res. Net One Institution’s Experience Thanks for your time! A copy of this presentation is available at: www. ithaca. edu/staff/dweil Res. Net Email: dweil@ithaca. edu Ithaca College RESIDENCE HALL COMPUTER NETWORK