CHEP 04 Performance Analysis of Cluster File System

Outline Introduction Review of cluster file system Data access model Performance analysis formula Performance

Introduction Cluster systems made up with PCs are more and more popular The improvement

Architecture of a cluster system job Compute node 1 job • • • Compute

Cluster file system review one of the most important methods to share information of

Data access model IO node 1 Disk IO node 2 ● ● ● Disk

Some assumptions Data is processed only in each client Storage nodes only provide storage

Performance analysis formula T = max (D*c/N, D/(N*I), D/(M*I), D/(P*R) ) S = D/T

In above formula, if c is very small, the formula becomes: T = max

Some cases N=1, M>=1 (or N>=1 and M=1), R>I S depends on I N=1,

Test environment Twelve PCs I/O nodes, Manager nodes and clients P 4 2. 8

Pre-test Test tools Netperf 2. 2 pl 3 Iozone 3. 217 Local area network

CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

One client one server Only one client access files Only one I/O nodes in

Results FS Record size (KB) 64 128 256 512 1024 2048 4096 8192 16384

Multi-process test Only one client and one I/O node Many processes access one I/O

Results NFS AFS PVFS CASTOR 1 10372 7878 10806 10680 2 10362 7889 10752

Multi-client to multi-server Multiple clients read/write files Multiple I/O nodes provide file storage The

Results CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Some useful methods In theory, good cluster file system the data is physically balanced

Use high-speed network, for example Gigabit Ethernet or Myrinet Use or develop high performance

Summary Cluster file system review Performance analysis formula Performance test Some methods to improve

Thank you!! CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Slides: 23

Download presentation

CHEP 04 Performance Analysis of Cluster File System on Linux Yaodong CHENG IHEP, CAS chyd@ihep. ac. cn

Outline Introduction Review of cluster file system Data access model Performance analysis formula Performance test Some useful methods CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Introduction Cluster systems made up with PCs are more and more popular The improvement of commodity hardware and software CPU, memory, hard disk, network Linux software technology How to use the our existing hardware and software more efficiently CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Architecture of a cluster system job Compute node 1 job • • • Compute node N disk High speed network I/O Node 1 disk • • • I/O Node N disk tape CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Cluster file system review one of the most important methods to share information of cluster system General characteristics: Single-system image Transparency Good scalability High performance Structure C/S, share-disk, virtual share-disk CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Data access model IO node 1 Disk IO node 2 ● ● ● Disk Meta Data Server IO node N Client 1 Network I/O Servers Disk Client 2 ● ● ● Client N Manager Node CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Some assumptions Data is processed only in each client Storage nodes only provide storage capacity and deal with file operations The traffic between clients and management nodes is very small The time for dealing with requests of clients is far smaller than the time consumed by transferring data CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Performance analysis formula T = max (D*c/N, D/(N*I), D/(M*I), D/(P*R) ) S = D/T = min (N/c, N*I, M*I, P*R) c: the CPU time to compute each byte; D: the total of data; I: network speed; M: the number of I/O nodes; N: the number of clients; P: the number of disks in parallel; R: disk speed T: the minimum access time to total data S: the maximum aggregate bandwidth Limitation: P/M >=1 CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

In above formula, if c is very small, the formula becomes: T = max (D/(N*I), D/(M*I), D/(P*R) ) S = D/T = min (N*I, M*I, P*R) and this formula is the basis of performance analysis in this work CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Some cases N=1, M>=1 (or N>=1 and M=1), R>I S depends on I N=1, M>=1 (or N>=1 and M=1), R<I S depends on I and P*R N>1, M>1, R>I S depends on the number of clients and I/O nodes CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Test environment Twelve PCs I/O nodes, Manager nodes and clients P 4 2. 8 G/512 M/Disk. WD 80 G-8 M-7200 RPM OS CERN Linux 7. 3. 3 Kernel: 2. 4. 20 -18. 7. cernsmp Local file system: ext 3 Network: 100 M Ethernet Cluster file system Open. AFS 1. 2. 9, NFS v 3, PVFS, CASTOR 1. 6. 1. 2 CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Pre-test Test tools Netperf 2. 2 pl 3 Iozone 3. 217 Local area network bandwidth (I): 100 M Ethernet: about 94. 11 Mbits/sec Local file system measurement (R). /iozone -Rab local. xls -g 2048 M Recompile IOzone linked with CASTOR RFIO library CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

One client one server Only one client access files Only one I/O nodes in server configuration Write performance measurement file size: 512 MB record size: 64 KB-16 MB output unit: KB/sec CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Results FS Record size (KB) 64 128 256 512 1024 2048 4096 8192 16384 NFS 11101 10803 11054 11125 11083 11042 11045 11109 11047 AFS 5173 PVFS 9953 5342 5239 5137 5148 5335 5212 5175 5353 10158 10103 10239 10759 10603 10662 10948 10976 CAST 10209 10335 10530 10622 10697 10722 10723 10705 10678 OR CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Multi-process test Only one client and one I/O node Many processes access one I/O node simultaneously. Write performance measurement File size: 100 MB Record size: 512 KB Process number: 1 10 Output unit: KB/sec CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Results NFS AFS PVFS CASTOR 1 10372 7878 10806 10680 2 10362 7889 10752 11255 3 10323 10841 10751 11221 4 10311 1020 10686 11450 5 10257 9358 10707 11430 6 10258 9142 10690 11441 7 10255 8120 10696 11390 8 10173 8545 10697 11440 9 10240 8652 10696 11442 10 10250 7305 10698 11430 Number of process FS CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Multi-client to multi-server Multiple clients read/write files Multiple I/O nodes provide file storage The output is aggregate bandwidth Only measure CASTOR and PVFS Write performance The size of each file: 200 M Record size: 2 MByte Output unit: MB/sec CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Results CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Some useful methods In theory, good cluster file system the data is physically balanced among the I/O devices the data requirements are balanced among the application’s tasks network has enough aggregate bandwidth to pass the data between the two without saturating In practice, the following methods are useful CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Use high-speed network, for example Gigabit Ethernet or Myrinet Use or develop high performance network file transfer protocol Use multi-server to improve the aggregate bandwidth Improve the read/write speed of disks File stripping and parallel I/O Good file system design Improve the processing ability of manager nodes CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Summary Cluster file system review Performance analysis formula Performance test Some methods to improve the performance CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland

Thank you!! CHEP'04 Sep 27 - Oct 1, 2004 Congress Zentrum Interlaken, Switzerland