FAD File Access Data Open VMS Boot Camp

FAD – File Access Data Open. VMS Boot Camp 2015 Greg Jordan Fact. Set September 2015 gjordan@factset. com

Agenda + FAD Overview + LCK Tracing + Using FAD + What FAD Can Tell you + Caveats + FAD Parsers to provide Cluster Views

File System Performance + The File System on Open. VMS can often contribute to a variety of performance issues. – Determining what is happening is not always that easy + FAD was created as a utility to provide insight into what is going on in the file system – A large percentage of performance issues that arose on our large clusters were File System related – Our DECps data would frequently report long response times for a disk volume • But there was not any performance issue happening within the IO Subsystem

FAD Overview + FAD is an SDA extension that leverages data form the LCK Trace facility – The file system operations use the VMS Lock Manager for synchronization • Most operations are serialized!! – By analyzing the LCK trace data • The frequency of file system operations per file can be determined. • The latency for the operations can be determined – Most locking operations can be mapped back to a process

File System Locking + The File System uses locking operations for: – – – Opening Files Close Files Deleting Files Extending Files Truncating Files Window Turns + Read/write IO to a file typical does not result in a file system locking operation – Unless the file needs to be extended or a window turn is needed

LCK Trace + The LCK trace facility traces events in the VMS lock manager – There about 8 different events that it will trace + The events are written to a ring buffer in memory – When the end of the ring buffer is hit, the tracing starts writing at the front again + FAD is only interested in 3 of the events – New lock requests – Convert requests – Deque requests + FAD is primarily tracing file system serialization locks – F 11 B$s<file id> – These are taken PW mode and so only 1 can be granted at

Starting LCK TRACE + The LCK trace is started from with SDA $ ANAL/SYS SDA> LCK LOAD SDA> LCK START TRACE/NOALL/NEW/CONV/DEQ + The above starts tracing for the lock manager + and limits the events to just the ones that FAD will look at The trace uses a 1280 page ring buffer – This can hold 109200 trace events – Increase with /buffer=pages

FAD Commands + FAD has 3 basic commands – FAD START COLLECT • Begin looking at the LCK TRACE data • FAD will gather process the LCK TRACE data once a second – FAD SHOW COLLECT • Display the collected data – FAD STOP COLLECT • Stop collecting data

FAD$SDA Help SDA> FAD File Access Data - Quick Help Information ----------------------FAD commands: 1. 3 FAD START COLLECT /INDEX=x /NEW /DEVICE=name FAD STOP FAD SHOW COLLECT /TOP=n /PROCESS=n /LATENCY /HISTOGRAM /NONAMES - start to collect file access data - the process to collect data on - can be a index, ipid or epid. The index will be extracted from the later - Only collect new trace entries that are created after the collection was started. Default is to collect all existing data in the LCK trace buffer. - Only collect data for file system operations on the specified volume name. - stops collecting file access statistics - display file access statistics number of files to display (max 500), default=20 number of processes to display sort based on latency, default is access count output a latency histogram - do not translate file ids to names

FAD SHOW COLLECT SDA> FAD SHOW COLLECT FDSA 08 Access Data from 24 -SEP 14: 39: 48 to 14: 39: 56 delta is 00: 07 -----------------------------------Access Top Process Tot Ave Count e. PID Count Name Procs Lat File Name --------------- -----------------------4702 30 EA 5941 4700 BATCH_1266 2 0. 4 DISK$I 64 VMS 84 T 1: [VMS$COMMON. SYSEXE]INSTALL. EXE; 2 2446 30 EA 5941 2446 BATCH_1266 1 0. 4 DISK$ENG_PERL: [FDS 0. BUILD. LOG]INSTALL_FDSD 07. LOG; 2 2352 30 EA 5941 2350 BATCH_1266 2 0. 0 DISK$JAMS: [JAMS. TEMP 2]INSTALL. RON 050 F 47 E 5; 1 1608 30 EE 555 E 1608 BATCH_981 1 0. 3 DISK$PRD_FIB: [000000]FDS 0. DIR; 1 1016 30 ECCAFF 1014 SWARI 2 0. 3 DISK$PRD_PROGS: [FDS 0. PRODUCTION. PROGS]BUILD. DIR; 1 955 30 ECCAFF 714 SWARI 13 0. 4 DISK$ENG_PERL: [PERL-5_8_7_I 64_ROOT. LIB]5_8_7. DIR; 1 608 30 ECCAFF 420 SWARI 13 0. 3 DISK$ENG_PERL: [PERL-5_8_7_I 64_ROOT]LIB. DIR; 1 603 30 EE 555 E 603 BATCH_981 1 0. 4 DISK$PRD_FIB_3: [000000]FDS 0. DIR; 1 603 30 EE 555 E 603 BATCH_981 1 0. 2 DISK$BENCHMARK: [000000]FDS 0. DIR; 1 402 30 EE 555 E 402 BATCH_981 1 0. 2 DISK$DATABASE 1: [DATABASE]SP. DIR; 1 402 30 EE 555 E 402 BATCH_981 1 0. 2 DISK$DATABASE 1: [000000]DATABASE. DIR; 1 402 30 EE 555 E 402 BATCH_981 1 0. 1 DISK$BENCHMARK: [FDS 0]EXG. DIR; 1 402 30 EE 555 E 402 BATCH_981 1 0. 1 DISK$DATABASE 14: [FDS 0. FTSE_INDEX]V 6. DIR; 1 307 30 EE 555 E 302 BATCH_981 3 0. 2 DISK$FACTSET: [FDS 0]FACTSET. DIR; 1 202 30 EE 555 E 201 BATCH_981 1 0. 5 DISK$BUILDARCH 3: [FDS 0. CANARY]FCR. DIR; 1 201 30 EE 555 E 201 BATCH_981 1 0. 3 DISK$FACTSET: [FDS 0. FACTSET]GUI. DIR; 1

/HISTOGRAM and /PROCESS SDA> FAD SHOW COL/HISTOR/TOP=3/PROC=3 -----------------------------------Access Top Process Tot Ave Count e. PID Count Name Procs Lat File Name --------------- -----------------------4702 30 EA 5941 4690 BATCH_1266 26 4. 2 DISK$I 64 VMS 84 T 1: [VMS$COMMON. SYSEXE]INSTALL. EXE; 2 Histo: 4702<10 ms 0<20 ms 0<50 ms 0<100 ms 0<500 ms 0<1 s 0<3 s 0<10 s 0>10 s 30 ECCAFF 10 SWARI 3. 9 long: 30. 6 time: 24 -SEP 14: 47: 06. 509927 30 EAF 943 2 SERVER_0277 16. 6 long: 45. 1 time: 24 -SEP 14: 46: 18. 376893 2446 30 EA 5941 2446 BATCH_1266 Histo: 2030<10 ms 10<20 ms 1 6<50 ms 4. 2 DISK$ENG_PERL: [FDS 0. BUILD. LOG]INSTALL_FDSD 07. LOG; 2 0<100 ms 0<500 ms 0<1 s 0<3 s 0<10 s 0>10 s 2352 30 EA 5941 2350 BATCH_1266 Histo: 2352<10 ms 0<20 ms 30 EDD 3 FA 2 GJORDAN 2 0<50 ms 0. 4 DISK$JAMS: [JAMS. TEMP 2]INSTALL. RON 05 0<100 ms 2<500 ms 0<1 s 0<3 s 0<10 s 0. 2 long: 0. 4 time: 24 -SEP 14: 46: 52. 044859 0>10 s

FAD Latency + When FAD sees a new PW lock requested: – If the resource is a type FAD monitors • FAD records the event time from the LCK TRACE data • The lock request might be granted immediately or might need to wait, no distinction is made + When a DEQueur or Convert is seen: – if a start time for the same lock was recorded: • A latency is computed and saved for the resource and for the process + The latency is the total time to acquired and hold the lock

SHOW COLLECT/LATENCY/PROC=5/HIST Ave Latency -------521. 2 421. 3 281. 7 Access Count File Name -------------------------------------------45 DISK$JAMS: [000000]JAMS. DIR; 1 Histo: 8<10 ms 2<20 ms 3<50 ms 4<100 ms 13<500 ms 11<1 s 5<3 s 0<10 s 0>10 s 208 B 4617 2 BOCI_PRU__33523 1435. 5 long: 2870. 9 time: 25 -SEP 02: 10. 459854 2093 E 66 E 1 BATCH_4003946 2765. 4 long: 2765. 4 time: 25 -SEP 02: 10: 11. 007110 208 A 4 AC 8 7 BATCH_1005834 465. 1 long: 1834. 7 time: 25 -SEP 02: 10: 41. 909403 2097 E 68 C 2 TCPIP$FTPC 7 DD 79 867. 4 long: 1717. 7 time: 25 -SEP 02: 10: 51. 695059 209 CE 689 1 BATCH_1773 1146. 8 long: 1146. 8 time: 25 -SEP 02: 10: 31. 712092 100 DISK$JAMS: [JAMS]DATA. DIR; 1 Histo: 12<10 ms 3<20 ms 9<50 ms 15<100 ms 31<500 ms 18<1 s 11<3 s 0<10 s 0>10 s 208 A 4 AC 8 6 BATCH_1005834 768. 2 long: 2632. 1 time: 25 -SEP 02: 10: 35. 625684 208 FB 63 F 11 RWI_QNT_51924 665. 0 long: 2500. 4 time: 25 -SEP 02: 10: 31. 658594 20810498 16 BATCH_1556 435. 1 long: 2354. 0 time: 25 -SEP 02: 10: 31. 708491 2094168 A 8 FQLCACHE__13506 581. 3 long: 1714. 1 time: 25 -SEP 02: 10: 51. 697064 208628 E 9 4 BATCH_2805 599. 3 long: 1464. 6 time: 25 -SEP 02: 19: 47. 780517 56 DISK$JAMS: [JAMS. DATA]PARAM. DAT; 2 Histo: 19<10 ms 0<20 ms 1<50 ms 9<100 ms 12<500 ms 13<1 s 1<3 s 0<10 s 0>10 s 20810498 9 BATCH_1556 334. 5 long: 1836. 6 time: 25 -SEP 02: 10: 34. 080480 208 A 4672 6 BATCH_1853 267. 1 long: 899. 2 time: 25 -SEP 02: 10: 49. 502851 2094168 A 5 FQLCACHE__13506 515. 9 long: 828. 3 time: 25 -SEP 02: 10: 54. 069982 208 A 4 AC 8 2 BATCH_1005834 448. 3 long: 814. 9 time: 25 -SEP 02: 10: 37. 883698 20821677 7 ING_IM_QN_40236 106. 0 long: 740. 7 time: 25 -SEP 02: 10: 40. 832391

Locks Monitored by FAD + F 11 B$s<file-id> – New PW mode and converts from NL to PW are counted as an access + F 11 B$v<volume-name> – Converted from CW to PW mode for various file system operations that require synchronization + INSTALL$KNOWN FILE – Acquired during image activation an by the INSTALL utility in PR mode + INSTALL$INSLOCK – Acquired by the INSTALL utility in PW mode when doing an INSTALL ADD, REMOVE, REPLACE command

FAD SHOW COL/LAT/PROC=4 100. 3 93. 8 31. 0 5. 7 121 INSTALL$INSLOCK Histo: 0<10 ms 0<20 ms 0<50 ms 78<100 ms 42<500 ms 2094 E 934 121 BATCH_2002764 100. 3 long: 272. 1 243 DISK$BUILDARCH 2: [FDS 0. BUILD 107]I 64. DIR; 1 Histo: 1<10 ms 0<20 ms 0<50 ms 186<100 ms 56<500 ms 2094 E 934 241 BATCH_2002764 93. 8 long: 257. 1 208 A 4 AC 8 2 BATCH_1005834 92. 2 long: 97. 4 1491 INSTALL$KNOWN FILE Histo: 927<10 ms 162<20 ms 253<50 ms 208 F 4579 25 20849133 110 BATCH_1005252 2092913 E 40 BATCH_2004091 2085313 C 7 PRICE_MGR_30330 2081 F 13 B 116 BATCH_2004058 F 11 B$v. EMPLOYEES 3 Histo: 28<10 ms 1<20 ms 0<50 ms 20801012 6 CACHE_SERVER 2092261 E 4 CS_MGR_51793 20890682 5 BATCH_1006348 208 B 36 AB 2 _FTA 749: 2087 D 651 4 CLIENT_ACC 92<100 ms 47<500 ms 102. 3 long: 1912. 9 96. 3 long: 1757. 0 69. 7 long: 1231. 6 122. 0 long: 773. 4 36. 5 long: 558. 8 0<1 s 0<3 s 0<10 s 0>10 s time: 24 -SEP 15: 00: 33. 203277 0<1 s 0<3 0<10 s 0>10 s time: 24 -SEP 15: 00: 33. 188796 time: 24 -SEP 15: 00: 32. 474733 8<1 s time: time: 6<3 s 0<10 0>10 s 24 -SEP 15: 09: 49. 479848 24 -SEP 15: 00: 24. 541662 24 -SEP 15: 00: 24. 385287 24 -SEP 15: 00: 18. 446187 24 -SEP 15: 00: 33. 074686 0<1 s time: time: 0<3 s 25 -SEP 25 -SEP 31 2<100 ms 25. 9 1. 6 0. 9 0. 5 0<500 ms long: 58. 5 long: 6. 0 long: 5. 1 long: 1. 6 0<10 s 0>10 s 00: 35. 319455 00: 03. 400194 00: 27. 061828 00: 35. 780554 00: 20. 895629

What can High Access Counts tell you? + High Access Counts on a file could indicate: – A high rate of opening and closing the file – A fragmented file with a lots of window turns + High Access Counts on a directory file: – Constant checking if a file exists – Attempting to open a file that doesn’t exist

What can High Latency tell you + High Latency on a file could indicate: – A slow down on the IO Subsystem – Many processes opening and closing the same file • We often see this for SYSUAF. DAT and RIGHTSLIST. DAT – A file continually expanding especially when: • • Disks space is low Free space is fragmented The expansion size is large The expansion request if for contiguous space or contiguous best try + High Latency on a directory could indicate: – A directory with a large number of files – Many processes doing operations within the directory • On our large clusters, we found that the rate of file system operations on a directory maxes out around 700

Long latencies $ TYPE FAD_ROOT: [FDSB 14]FAD-201509250210 -FDSB 14. LAT; 1 Ave Access Latency Count -------21445. 7 12 65 A 3977 D 65 A 2 F 77 E 65 A 3 E 77 F 65 A 3 B 780 65 A 36781 302. 6 54 65 A 32782 65 A 3777 B 65 A 010 B 4 65 A 3977 D 65 A 3176 F 280. 3 272 65 A 2 F 77 E 65 A 33784 65 A 35786 65 A 3 B 76 D 65 A 33785 File Name --------------------------------------------DISK$JAMS: [000000]000000. DIR; 1 1 32711. 7 long: 32711. 7 time: 25 -SEP 02: 16: 37. 180490 1 32031. 7 long: 32031. 7 time: 25 -SEP 02: 16: 40. 672348 1 31824. 3 long: 31824. 3 time: 25 -SEP 02: 16: 43. 773263 1 28937. 5 long: 28937. 5 time: 25 -SEP 02: 16: 47. 920245 1 25976. 4 long: 25976. 4 time: 25 -SEP 02: 16: 51. 063101 DISK$JAMS: [000000]JAMS. DIR; 1 2 822. 7 long: 1320. 0 time: 25 -SEP 02: 16: 56. 731196 2 682. 8 long: 1240. 6 time: 25 -SEP 02: 16: 32. 388431 6 PSDC$DC_SERVER 401. 8 long: 1115. 0 time: 25 -SEP 02: 18: 00. 484472 2 455. 2 long: 896. 9 time: 25 -SEP 02: 16: 38. 077671 2 630. 4 long: 841. 1 time: 25 -SEP 02: 16: 11. 713801 DISK$JAMS: [JAMS]DATA. DIR; 1 11 458. 2 long: 1356. 8 time: 25 -SEP 02: 16: 45. 147082 11 383. 2 long: 1286. 8 time: 25 -SEP 02: 17: 05. 202329 11 227. 8 long: 1177. 7 time: 25 -SEP 02: 17: 13. 974230 11 240. 0 long: 1123. 7 time: 25 -SEP 02: 16: 08. 946838 11 368. 0 long: 1039. 3 time: 25 -SEP 02: 17: 09. 795229

Caveats + If a file is deleted: – FAD will not be able to lookup the name and instead will show the file id – Should the file id be re-used by the file system, FAD will attribute all of the previous files operations to the new file name + FAD process all new trace entries in the LCK TRACE buffer once a second – If locking activity is high, the default 1280 page ring buffer might wrap in under 1 second • This will result in FAD missing events so file access counts will be lower and latency computations will not be for all operations • The LCK DEBUG command will show the Max Index and Cur Index into the trace buffer – repeating the above will give you an idea on how fast the Cur index is changing + FAD displays 500 files at most – FAD does provide an 100% complete picture

Gathering Trace Entries + For FAD to display a file name, File System locking occurs – The filename lookup only happens for files that are going to be displayed – The filename lookup only happens once • Continually issuing FAD SHOW COLLECT will not result in continually looking up the same filenames + The latency will display as Na. NQ if there is no latency data – This can happen when FAD sees a new lock request but never sees a deq or convert for the lock

FAD_COLLECT + The product contains some procedures to help collect FAD data – FAD_COLLECT_MAIN. COM – FAD_COLLECT. COM – WAITSOME. COM – FAD_ROOT must be defined (see the instructions) – FAD$SDA. EXE must either be placed in SYS$COMMON: [SYSLIB] or the logical FAD$SDA points to the image + An Access and Latency report are created every 10 minutes – A cleanup routine is currently not provided – We keep about 3 weeks of FAD data around

Cluster View + Looking at a single node doesn’t always + provide the full picture PERL parses have been created to take FAD output and produce a cluster view – The parsers expect FAD data files with 10 minutes of data and a specific file name FAD_yyyymmddhhmm_nodename. * – The collect routines provided create these names with. TXT and. LAT for output with the /LATENCY qualifier + The parses can produce summaries for one 10 minute interval – *_201509281000*. txt + Or an entire day – *_20150928*. txt

FAD Parser for Access data perl fad_parser. pl [-f filter] [-e text] [-t n] [-i n] input-files – [-f filter] processes selections that contain the filter text – [-e text] exclude selctions that contain the text – [-t n] number of files to display – [-i] display top individual captures

FAD Parser Cluster Report $ PERL fad_parserl. pl FAD_ROOT: [*]FAD_201509100000*. TXT Parsing 38 files …… Top 20 summed across range: Lock. Cnt ======= 57807 55725 52502 41888 28508 26717 22271 19962 17975 17963 16373 12086 10954 10287 10072 10045 9509 8824 8771 8592 Pct. Tot ====== 3. 2% 3. 1% 2. 9% 2. 3% 1. 6% 1. 5% 1. 2% 1. 1% 1. 0% 0. 9% 0. 7% 0. 6% 0. 5% Lock. Rate ==== 101. 4 97. 8 92. 1 73. 5 50. 0 46. 9 39. 1 35. 0 31. 5 28. 7 21. 2 19. 2 18. 0 17. 7 17. 6 16. 7 15. 5 15. 4 15. 1 # Capt ====== 18 28 23 22 20 38 1 38 5 38 36 12 1 27 1 1 3 5 1 29 Filename ==== DISK$PRD_EST_1: [FDS 0. FDS_EST. V 6]LAST_RT_UPDATE. TXT DISK$DATABASE 7: [FDS 0. QUANT]PROGS. DIR DISK$ENG_PERL: [FDS 0. PERL-5_8_7_I 64_ROOT. LIB. VMS_IA 64]5_8_7. DIR DISK$ENG_PERL: [FDS 0. PERL-5_8_7_I 64_ROOT]LIB. DIR DISK$PRD_EST_1: [FDS 0. FDS_EST]V 6. DIR DISK$FDSVMS: [VMS$COMMON. SYSEXE]SYSUAF. DAT DISK$PRD_FI_2: [FDS 0. SP_MUNI]SP_MUNI_PRICE_VEND_DESC. FDB DISK$FDSVMS: [VMS$COMMON]SYSEXE. DIR DISK$CLIENTAH: [FDS 0. LAZARD_SC]DA. DIR DISK$FDSVMS: [VMS$COMMON. SYSEXE]RIGHTSLIST. DAT DISK$FACTSET: [000000]FDS 0. DIR DISK$ENG_PERL: [FDS 0. PERL-5_8_7_I 64_ROOT. LIB. PERL. VMS]XML. DIR DISK$CLIENTA 1: [FDS 0. STRONG]$$FI_RECON_REPORT. DB DISK$DATABASE 9: [FDS 0]MSCI_V 6. DIR DISK$CLIENTAZ: [FDS 0. AMER_CAP. AXYS]11226686. OFDB DISK$PRD_SCR_LOGS: [FDS 0. DB_UPDATE]FRMLA_CACHE. DIR DISK$EMPLOYEES 4: [000000]FDS 0. DIR DISK$CLIENTAL: [FDS 0. EAGLE_STRAT]DOCUMENTS. TLB DISK$CLIENTAF: [FDS 0. WESTWOOD]4483. OFDB DISK$MISC_PROD: [FDS 0]DBCFG. DIR

More FAD_PARSER. PL examples $ perl sysprogs: fad_parser. pl -t 2 -f prd_est fad_root: [*]*201509250000*. txt Parsing 38 files. . . . Top 2 summed across range: Lock. Cnt ======= 57807 28508 Files : Range : Total : Duration Window Pct. Tot ====== 3. 2% 1. 6% Lock. Rate ==== 101. 4 50. 0 # Capt ====== 18 20 Filename ==== DISK$PRD_EST_1: [FDS 0. FDS_EST. V 6]LAST_RT_UPDATE. TXT DISK$PRD_EST_1: [FDS 0. FDS_EST]V 6. DIR 38 fad_root: [bid]fad-201509250000 -bid. txt -> fad_root: [fdsa 52]fad-201509250000 -fdsa 52. txt 1796179 access counts (new locks) : 0 days 6 hours 20 minutes worth of data : 0 days 0 hours 10 minutes x 38 nodes $ perl sysprogs: fad_parser. pl -t 2 -e prd_est fad_root: [*]*201509250000*. txt Parsing 38 files. . . . Top 2 summed across range: Lock. Cnt ======= 55725 52502 Pct. Tot ====== 3. 1% 2. 9% Lock. Rate ==== 97. 8 92. 1 # Capt ====== 28 23 Filename ==== DISK$DATABASE 7: [FDS 0. QUANT]PROGS. DIR DISK$ENG_PERL: [FDS 0. PERL-5_8_7_I 64_ROOT. LIB. VMS_IA 64]5_8_7. DIR

FAD_LAT_PARSER. PL NJ A 08 S 3: [GJORDAN]$ perl sysprogs: fad_lat_parser. pl -ht 3 -p 4 fad_root: [*]*201509232330*. lat Parsing 38 files. . . . Top Latency Files for 09 -23 -2015 23: 30 to 23: 40: File name ================================================== DISK$JAMS: [000000]000000. DIR; 1 13130. 5 ms Count: fdsa 40 fdsa 11 fdsa 45 fdsa 40 33 1: <10 ms 26 A 5665 A 24 B 3 B 65 C 208 B 6657 26 A 75639 1 1 0: <20 ms 0: <100 ms BERENBDE__9212 BFE_QA_QN_33983 FRIENDS_Q_61455 BATCH_5002064 22743. 3 21547. 8 21309. 2 20235. 1 2: <500 ms long: 2: <1 s 22743. 3 21547. 8 21309. 2 20235. 1 2: <3 s time: 23 -SEP 6: <10 s 20: >10 s 23: 31: 03. 525442 23: 31: 05. 235444 23: 31: 01. 144650 23: 30: 56. 851943 DISK$CLIENTAF: [FDS 0]WESTWOOD. DIR; 1 495. 9 ms Count: fdsa 40 18 13: <10 ms 26 AAC 65 D 26 A 78329 1: <20 ms 1: <50 ms 16 BATCH_5006767 1 BATCH_1008153 528. 9 0. 1 0: <500 ms long: 0: <1 s 0: <3 s 1: <10 s 0: >10 s 7875. 4 time: 23 -SEP 23: 30: 53. 786291 0. 1 time: 23 -SEP 23: 31: 04. 372097 DISK$CLIENTAF: [FDS 0. WESTWOOD]AXYS_TEMP_26 AAC 65 D. TEMP; 1 398. 5 ms fdsa 40 Count: 22 26 AAC 65 D 26 A 94648 26 A 78329 26 A 00000 18: <10 ms 0: <20 ms 5 BATCH_5006767 13 TCPIP$FTPC 3 FBF 7 2 BATCH_1008153 2 1: <100 ms 1575. 5 7. 0 0. 2 0. 0 0: <500 ms long: 7875. 5 60. 9 0. 3 0. 0 0: <1 0: <3 s 1: <10 s 0: >10 s time: 23 -SEP 23: 30: 53. 786268 time: 23 -SEP 23: 30: 29. 124487 time: 23 -SEP 23: 31: 04. 371957 time: 6 -MAY 03: 07: 06. 085507

Summary + FAD Provides – information that is hard to extract from existing performance tools – new insights into system and application performance – Can be used ad hoc when needed or use to continually collect data

Interested in FAD? + Sources are available (under the GPL) + No implied warranty or support + An Itanium image (linked under V 8. 4) is in the kit – Should also work on V 8. 4 -1 H 1 (but have not verified) + Kit available at: – Fad. netjordan. com + Send questions or comments to fad@netjordan. com