Facilities and How They Are Used ORNLProbe Randy

Facilities and How They Are Used ORNL/Probe Randy Burris Dan Million – facility administrator Mike Gleicher – development contractor Florence Fowler – network research and linux support OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Overview q Background q Facilities for Data Analysis q Facilities for file transfers and Grid q Facilities for file system integration and improved access to tertiary storage OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Backgrouhd – Goals q Provide an environment for SDM ISIC activities q Provide an environment for Sci. DAC applications § Climate § Terascale Supernova Initiative § High Energy Nuclear Physics q Provide an environment that is appropriate for production deployment q Support and perform R&D on improved tertiary storage access OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Background – Supplying the environments q Predict § § § equipment needs, assuming: Large memory Large storage capacity – disk and tape Good bandwidth – network and storage Moderate CPU power AIX or Red. Hat Linux 7. 2 operating systems q Provide basic software (compiler) q Respond to subsequent requests OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Facilities for Data Analysis Production HPSS Probe HPSS External Networks ESnet OC 12 OC 192 AIX S 80 6 -processor 467 MHz RS 64 -II, 2 GB AIX RS/6000 p 630 4 -processor 1 GHz Power 4; 4 GB AIX RS/6000 H 70 4 -processor 340 MHz RS 64 -II; 4 GB Linux 4 processor 1. 4 GHz Xeon 8 GB 200 GB SCSI RAID 1 TB Fibre. Channel 360 GB T 3 Fibre. Channel 360 GB internal 180 GB 1 TB Fibre. Channel U. S. DEPARTMENT OF ENERGY Each Dual P-III 240 GB Argonne cluster PVFS Fibre. Channel Switch 180 GB OAK RIDGE NATIONAL LABORATORY Alice cluster OSCAR; PVFS Each: One Athlon 480 GB

Data Analysis Software Environment AIX and linux Java R data analysis GGobi data visualization gnu Fortran, C OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY AIX Fortran, C

Facilities for file transfers and Grid Unassigned or floating External Networks ESnet OC 12 OC 192 2 proc 2 GB linux IBM B 80 2 proc, 1 GB AIX (Grid nodes) IBM 44 P 1 proc, 512 AIX Sun E 250 512 MB Solaris 300 GB SCSI 2 proc 2 GB linux 1 TB Fibre. Channel 300 GB SCSI 180 GB Fibre. Channel Globus 2. 0 on AIX Globus 2. 0 on Solaris HRM research on Solaris OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY Sun E 450 512 MB Solaris

HRM Research Environment ORNL Production HPSS NERSC Production HPSS Disk Cache STKSTK Library Sleepy Sun E 250 ORNL Probe HPSS pftp, hsi, Grid. FTP Globus 2. 0 Probe Production OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY Two-stripe STK Tape Drive 180 GB Sun STK Tape Fibre. Channel Drive RAID 180 GB Sun Fibre. Channel RAID

HRM Research Environment, soon Production HPSS NERSC Production HPSS Disk Cache Sleepy Sun E 250 Brookhaven Production HPSS Probe Production OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY pftp, hsi, Grid. FTP STKSTK Library Probe HPSS Globus 2. 0 1 TB Fibre. Channel RAID Selected STAR data 3 TB Tape storaage

Improving tertiary access q q q Improve end-user access (hsi) Investigate linking PVFS and HPSS Study HPSS I/O involving Unix files Investigate use of HPSS linux movers Monitor (and evaluate when available) HPSS I/O redirection Coordinate with a proposed project § Shared very large disk storage § Shared file system § HPSS integrated with file system and sharing the disk capacity OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

HSI improvements q Added Grid authentication service q Implemented partial file transfers q Added scheduling mechanism § Find all files to be sent § Sort by tape and location on tape OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

File system integration 2 proc 2 GB linux Probe HPSS Alice cluster OSCAR; PVFS Each Dual P-III 240 GB 1 TB Fibre. Channel “Argonne” cluster PVFS Each Dual Athlon 480 GB OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

HPSS, Unix files and linux RAIDzone 2 proc, 1 GB 2 TB RAID XFS; linux RAIDzone – use to study: HPSS I/O involving Unix files HPSS linux mover NFS with HPSS archive Alice cluster OSCAR; PVFS Each Dual P-III 240 GB Argonne cluster PVFS Each Dual Athlon 480 GB OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 2 proc 2 GB linux 1 TB Fibre. Channel

Since 10 PM yesterday… q To support § Sci. DAC SDM and Sci. DAC Applications § Research (I/O to/from Cray X 1) § Production q New equipment on order; hope to get this month § Two more Dell 2650 linux processors • 2. 4 GHz, 2 GB, 5 73 GB disks § Four more 1 TB Fibre. Channel RAID arrays § Two more IBM servers § Tape cartridges sufficient for 30 TB using existing tape drives OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Questions and answers q What goals/accomplishments by Feb 2003? § § § q All equipment installed and operational HPSS/RAIDzone linux mover testing complete Link between PVFS and HPSS (linux mover and/or hsi) tested Star data residing in ORNL HPSS (Probe or production) Some data passed over OC 192 link Why are the goals important (and to whom)? § Data analysis staff need big-resource node § Visualization activities in support of TSI need disk cache and flexible computational resources § HPSS I/O of Unix files is central to many HPSS plans § Users of PVFS should have an archive capability § HRM needs three data sites for thorough testing of selection algorithms § OC 192 bandwidth can satisfy a lot of bulk WAN transfer needs OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Questions and answers, continued Which scientific domain? HENP and TSI, mostly q Why is the work significant? Who will use it? q § Visualization of massive data is necessary for scientific research. This facility targets storage and movement of massive data. § Application scientists and ISIC researchers Are others doing the same work? No. q Status at end of three years? Achievable? q § HPSS and filesystems will be better integrated § Access to HPSS will be faster and easier § WAN throughput will be greatly improved q Will unsolved problems remain? Proposals? § There is never enough bandwidth or effective throughput § Filesystems and equipment are evolving rapidly; keeping up is tough § Distributed operations – data analysis, visualization – will remain challenges OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Discussion? OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY