Hosting Millennium Project AIUG 16 November 2012 Lesley

Hosting Millennium Project AIUG 16 November 2012 Lesley Sharp Manager, Online Strategy

Background SA Government State. Net SLSA

Issues - 2008 -2009 • Millennium server • 7 years old and needed to be replaced • Not compliant with SA Government Security requirements (database and web server on same server) • DMZ environment • Even if the web serving module was deployed and application and database servers separated, approval for SSH access would be refused • III Support Requirements vs State. Net Support Requirements • III preferred support method = SSH (Port 22) • State. Net = VPN, two-factor authentication and no split tunnelling • Testing showed that two-factor authentication process raised other issues for III, and couldn’t realistically be achieved (clash of IP numbers)

PATRON INTERNET 1. SLSA 2. DMZA 3. DMZA 4. III 5. HSP Co-located Hosting 6. 7. HSP Dedicated Hosting (VS) SLSA INTERNET STAFF 8. Role Based DMZ

Option 4 • October 2009 – commenced discussions with III about their willingness to host our Millennium system in Emeryville • Commenced a one month trial from early December 2009 of hosting Millennium with III

Show-stopper • Serial check-in card! • Drawing the 84 -box check-in card on the screen consistently took 20 seconds • III staff testing in Emeryville – instantaneous • III staff tested from Ohio and the response time reduced to 9 -14 seconds • Investigations with Dimension Data to check if packet-shaping of data through State. Net was causing this problem

Show-stopper cont’d • Tested from different parts of the network with no change in the delay time • Dimension Data traced the data and advised there was no packet loss and the delay was caused by latency while traversing networks and/or the application itself may be the cause • End result that Hosting Millennium with III in the USA was not a viable option • Project closed 23 March 2010

Option 5 – Co-Located Hosting • SA Government Hosting Panel established 2008 • SLSA issued RFQ in June 2010 for “Un-managed Co-Location” services i. e. Place our new server in rented rack space outside of State. Net and have SA Government DCSS vendor manage the server for us • No formal response to this RFQ, general lack of rack space in Adelaide at that time

Option 6 – Virtual Server - Hosted • 9 July 2010 – Issued RFQ for a Virtual Private Server in a dedicated hosting environment • One response – from Internode • 19 August 2010 – Customer Agreement established • 16 September 2010 – Virtual Private Server commissioned • Hooray – all our problems are over!

Migration • Opened a new project with III, engaged a project manager • Held a series of planning teleconferences • Migration date of 26 October 2010 agreed • Outage notifications were issued a week prior to staff and customers – advising up to 72 hours outage, with possibility of early release

Issues • Required to convert from Method I indexing to Method II indexing for Advanced Word Searching before the migration could occur • III Project Manager unavailable • III had forgotten that the data would need to be converted from Sun to Linux • Release 2009 A and Linux 64 bit create performance problems • Unplanned immediate upgrade to Release 2009 B • Caused issues with acquisitions data

Issues cont’d • Firewall not set up correctly, needed changes • Host name not set up correctly, not fully qualified • Issues with testing the Enterprise Backup API • Communications time lag between USA and SA • SLSA Information Desk staff provided access to the catalogue despite an official “outage” still being in place

Issues cont’d • When testing commenced, not all staff had the most recent Millennium client • Staff did not follow the test plan • Other than seeing how quickly the serials check-in card rendered • Found it was impossible to get the required number of people testing at the same time • We were too far down the track to consider rolling back

Issues cont’d • And as we segued from testing to normal operations we started to encounter Performance Issues Disconnections

Issues cont’d • PC power settings adjusted • Higher than acceptable network collision rate • Commumications cupboard re-cabled • Millennium clients freezing; response times slow • Upgraded server with memory and CPU • Ongoing performance issues • III check VMWare settings

Issues cont’d • Internode add VMWare Tools • Internode apply Resource Groups 18 February 2011 Performance issues resolved

Issues cont’d • Disconnections • Dimension Data - Packet tracing • State. Net Intrusion Prevention System – change required to bypass rules 10 March 2011 Disconnect problem declared ‘solved’

Other issues • 16 March 2011 • Kernel problems commence • Related to install of Encore Reporter? • Resolved by booting into “old” kernel • 25 March 2011 (a Friday of course) • Performance issues – slowness and Server Not Responding messages • This turns out to be caused by an incident at Internode, unrelated to anything else (phew!) • No other outages caused by hosting environment

Benefits of hosting No more State. Net! Better backup regime Ability to increase server resources quickly and simply III can provide support using their preferred method Have been able to deploy our backlog of modules (RSS, WAM, Encore Reporter, ERM, Content Cafe) Can now plan upgrades such as to Sierra with more confidence

Thank you Lesley Sharp Manager, Online Strategy Sharp. lesley@slsa. gov. au State Library of South Australia, North Terrace, Adelaide SA 5000 Phone: (08) 8207 7250 - www. slsa. gov. au