Building a research cloud with things you already

Building a research cloud with things you already have in your kitchen Angus Griffin / Guido Aben

Cloud. Stor: The Original Ad Hoc Large File Transfer Service 2

Cloud. Stor: Uploads over last year uq unimelb monash deakin uwa unsw uws latrobe utas murdoch qut jcu uts 3 23292 9469 8790 6168 5870 5357 3983 3605 3143 2013 1867 1469 1389 griffith uow unisa rmit anu adelaide curtin flinders mq scu cqu usyd acu 1265 1181 1098 982 700 691 511 443 438 363 307 299 258 une swin vu usc nd usq ecu cdu canberra newcastle csu ballarat 251 156 135 88 86 71 61 58 34 17 7

Repeat Questions “Can we keep our files up permanently? ” “Can I upload multiple files? ” “Why is it called cloudstor when it’s really for sending ? ” 4

Cloud. Stor: All About Workflow Now TWO applications: & 5

Cloud. Stor+ Main Features • • 6 Open Source Licenced: own. Cloud Access, Sync, Share Single Sign On (AAF) National distribution

Cloud. Stor+ Use & Proposed Cost Personal Working Storage for Researchers: • 100 GB is free (Tier 1 institutional support required) • 1 TB @ ~$30/month or ~$300/year 7

Why build your own? • On the surface, analogous to Dropbox, Sky. Drive, Google Drive et al. • Different to the usual suspects: – Pre-baked authentication and authorisation integration, – Close to the user – geolocation of files via data placement – Proximity means low latency means performance – Scale-up costs predictable, not beholden to fickle licensing: built on commodity hardware with open source software

Ingredients found in the kitchen • Frontend: own. Cloud (with a few tweaks) – OSS, free, very active community • Backend: Map. R HDFS (based on Apache Hadoop) – Strongest NFS implementation we could find. however costs a few $ may replace Map. R with direct own. Cloud-HDFS integration • Initially 3 boxes in 3 existing Po. Ps • own. Cloud instances anycast-advertised • Hadoop synchs filesystem & databases

Pots, pans and recipes • Cheap, cheap. No “special sauce” storage. • The “Google” approach - design for failure – Replicate at least 3 times, in different places • Each node 108 TB (so, 36 TB useable) – 36 x 3 TB drives – Based on commodity hardware – Connected at 10 Gbps or 40 Gb directly to the backbone • Next set of nodes will be 72 x 3 TB drives (in 4 U) – That’s 72 TB useable!!! – Fit 2 PB raw, 720 TB useable in a rack. That’s what we call “really dense”.

Distributed Architecture

Cloud. Stor+ Speed Test Comparison (from Sydney HQ) Cloud. Stor+ Sugar. Sync Sky. Drive Upload Speed 30 MB/s 0. 3 MB/s 0. 75 MB/s Google Drive Box. net 0. 75 MB/s 0. 15 MB/s To be fair, it’s still in beta … 12 Amazon Drop. Box Cloud. Drive 0. 2 MB/s 0. 55 MB/s

The Future • International nodes? • Or even wilder: a Science CDN? • Research-specific apps! – Metadata – Collab editing / annotating – Specialised file viewers (microscopy… genomics…) • Sharing from the shell?