NPP Science Data Segment SDS Data Depository and

NPP Science Data Segment (SDS) Data Depository and Distribution Element (SD 3 E) Delta Design Review for CERES August 26, 2008 NASA GSFC B 8 Auditorium Evelyn Ho Evelyn. L. Ho@nasa. gov Ryan Gerard Ryan. Gerard@nasa. gov NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 1

SD 3 E Overview (1 of 2) NESDIS NSIPS NOAA ADS (CLASS) NESDIS IDPS CERES Data Production Intermediate Products Ocean PEATE RDR, SDR EDR, IP SD 3 E SDR, EDR, IP, TDR, LUTs, Algorithms • 32 Day Data Buffer for RDR, SDR, EDR, and 5 days of Retained Intermediate Products RDR, SDR EDR, IP RDR, SDR, EDR, IP • Support EDR Assessment • Support NICSE CV Assessment Pre-/Post-Launch • Provide Data Analysis Tools to ST • Develop and Demonstrate Algorithms enhancements Pre-launch data TDR, RDR, SDR, EDR, IP Ozone PEATE • Support EDR Assessment • Provide Data Analysis tools to ST • Develop and Demonstrate NESDIS C 3 S Calibration Tasking Ozone Community OMPS Limb SDR EDR NPOESS Algorithm CCB Algorithms enhancements • Assess and Validate OMPS Radiometric and Geometric Calibration Pre- and Post-Launch • Perform OMPS Limb Cal/Val • Support OMPS Limb Ops. • Generate OMPS Limb SDR EDRs • Transition OMPS Limb Algorithm to NOAA for Operations Cal. LUT and S/W updates CERES RDRs, VIIRS Subset SDRs Algorithm Enhancements IP Gen. Request Sounder PEATE (JPL) • Support EDR Assessment • Provide Data Analysis tools to ST • Develop and Demonstrate Algorithms enhancements • Assess and Validate Cr. IS & ATMS Calibration Pre- and Post-Launch (TBD) • Assess and Validate VIIRS Radiometric and Geometric Calibration Pre- and Post-Launch • Provide Calibration update Recommendations • Coordinate PEATEs Cal Related Analysis and Anomaly Characterization IDPS S/W Packages I&TSE Enhancement • Generation of Intermediate Product, if necessary • IDPS Production Algorithms / • Software Status reports, Management direction Algorithm Enhancements IP Gen. Request Status reports, Management direction STA Report NPP SDS SD 3 E CERES Delta Design Review NPP Science Team CLASS • A Mini- Version of the IDPS • Demonstration of Algorithm EDR Assessment, Algorithm Enhancements CERES Science Users NICSE Quality Control, Land PEATE Pre-launch data Casa. NOSA IPO Data Products to Characterize Global Climate change Atmosphere PEATE (U of Wisconsin) CERES RDRs RDR • Generate Climate Quality Status reports, Management direction EDR Assessment Algorithm Enhancements PSOE • Overall management direction and science guidance to the PEATEs • Coordinates EDR Assessment, Algorithm, and Calibration recommendations for submission New due to Level 1 Requirements Changes Not all data flows shown here. SD 3 E- Page 2

SD 3 E Overview (2 of 2) • SD 3 E – SDS Data Delivery & Depository Element – Performs data request and acquisition from major data providers • Acquires data from NOAA/CLASS, NSIPS, and NESDIS/IDPS • Now will handle CERES RDRs like other RDRs – Provides ~32 days “rolling storage” for pick-up by PEATEs and the NICSE – No data processing of NPP products • No reformatting • No aggregation • No subsetting NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 3

SD 3 E Level 3 Requirements • Acquire products [Req. 3. 1. 1. 1 – 3. 1. 1. 18] – RDRs from IDPS – x. DRs, selected IPs, operational algorithms and source code, software and documentation, calibration products, selected IPs, and official ancillary/auxiliary data from ADS – Retained IPs from NSIPS • Verify data integrity [Req. 3. 1. 2. 1] • Manage data requests from PEATEs/NICSE [Req. 3. 1 – 3. 10] • Provide data access to the PEATEs/NICSE [Req. 3. 1. 4. 4] • Manage 32 -day (TBD) data store [Req. 3. 1. 4. 1 – 3. 1. 4. 3] • Respond to PSOE management direction and provide status reports [Req. 3. 1. 4. 5 – 3. 1. 4. 6] • General Requirements [Req. 3. 6. 1 - 3. 6. 2, 3. 6. 7, 3. 7. 1 - 3. 8. 3. 1] NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 4

SD 3 E Level 3 Requirements for CERES Requirement Number L 3 Requirement Covering CERES 3. 1. 1. 3 The SD 3 E shall be capable of requesting and accepting all RDRs generated by the IDPS. 3. 1. 1. 9 The SD 3 E shall be capable of requesting and ingesting x. DRs (RDRs, SDRs, EDRs, and TDRs) from the ADS. 3. 1. 4. 3 The SD 3 E shall have the capability of providing 32 days of storage to store x. DRs and 7 days of storage for all other products (e. g. , intermediate products, ancillary data, operational algorithms, calibration products). 3. 1. 2. 1 The SD 3 E shall have the capability of validating all the ingested data files based on data delivery reports, digital signature, checksum or other means, ensuring data transfer was done correctly. NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 5

Context Data Flow Diagram IDPS Data Product Requests IDPS Product Requests and Status x. DRs NPP Products File Errors Request Status Data Delivery Report NSIPS Retained IPs NSIPS Data Product Requests and Status User (PEATES, NICSE) P 1 SD 3 E Management Direction Reports Digital Signature Operational Algorithms Official Ancillary Data PSOE x. DRs, Selected IPs Calibration Products ADS Product Requests and Status ADS Calibration Coefficients Software and Documentation NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 6

Level 1 Data Flow Diagram All RDRs NESDIS IDPS Selected SDR , IPs EDRs, and TDR NOAA CLASS RIPS SD 3 E NSIPS Telemetry RDR, Science RDR, Diagnostic RDR VIIRS SDRs Channels, 5, 7, 9, 10 -16 Aerosol EDR Data Subscription LAND PEATE SDS ASDC Telemetry RDR (HDF-5) Science RDR, (HDF-5) Diagnostic RDR (HDF-5) VIIRS SDRs Channels, 5, 7, 9, 10 -16 (Subsetted HDF 4) Aerosol EDR (HDF 4) NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 7

Gap Analysis • Add CERES product types to SD 3 E product definition table – CERES science RDRs – CERES diagnostic RDRs – CERES telemetry RDRs • Additional Data Volume/Disk Space – Ingest additional 2500. 48 Ki. B/Day (2083. 74 Ki. B/Day + 20% HDF overhead) – ~78. 14 Mi. B storage for 32 days NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 8

Results of RMA and Throughput Study • SDS Level 3 Requirement – 3. 7. 3 Availability • The NPP SDS shall have an Operational Availability, Ao. , of 0. 95. • Background: The SDS and associated elements have only been “Research Grade”. • Risk: Data latency when the SD 3 E system is down. • Will the current requirement be sufficient in meeting CERES requirements? NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 9

Results of RMA and Throughput Study • Assumptions – – Five year mission: 43800 hours Operating time: 10000 hours Time to repair disks: 288 hours (2 weeks) Average time for maintenance/resolve problems: 168 hours (1 week) – Average time to bring system back up (Nominal scenario): 50 minutes • • Time for operator to notice problem: 5 minutes Time to take down the system/clean up: 25 minutes Time to bring DB back up: 10 minutes Time to bring SD 3 E back up: 10 minutes NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 10

Single Disk Metrics P (single disk fails during mission) = # of hours of mission / MTBF of disk = 43800 hr / 1000000 hr =. 0438 = 4. 4 % P(single disk is down at a given moment in time during Mission) = P(single disk fails during mission) x P(disk has not yet been replaced) =. 0438 x (288 hr / 43800) =. 000288 =. 028% assumes 2 weeks to repair drive NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 11

Tier Metrics (1 of 2) • Assumptions: – disks are placed in tiers of size 10 – 2 or more disks must fail in the same tier for downtime to occur • P( no disks are down in a given tier) = P(disk 1 never goes down) * P(disk 2 never goes down) * … * P(#10 never does down) = (. 9997)10 =. 997 NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 12

Tier Metrics (2 of 2) • P(1 disk is down in a tier) = P(disk 1 is Down) x P(disks 2… 10 are Up) x # of variations of which disk is down = (. 000288)(. 9997)10(10) =. 0029 • P(2 or more disks are down in a tier) = 1 – P(no disks are down) – P(1 disk is down) = 1 -. 997 -. 0029 =. 0001 NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 13

Whole System Disk Metrics • P(2 or more disks are down in ANY tier) = # of tiers x P(2 or more disks fail in a single tier) = (50)(. 0001) =. 005 Availability = 1 – P(2 or more disks are down in ANY tier) Ao of entire disk array = 0. 995 NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 14

Results of RMA and Throughput Study DB 1 S 1 C 1 D 1 S 3 DB 2 Block S 2 Description C 2 MTBF Failure/Hour % time it will Fail DB 1 (Primary Database server) Dell Power. Edge 2850 84400 9. 66184 E-06 0. 002 DB 2 (Secondary Database server) Dell Power. Edge 2650 103500 1. 18483 E-05 0. 002 S 1 (Primary ingest server) Dell Power. Edge 2950 102100 9. 79432 E-06 0. 002 S 2 (Failover ingest server) Dell Power. Edge 2950 102100 9. 79432 E-06 0. 002 S 3 (Secondary ingest server) Dell Power. Edge 2950 102100 9. 79432 E-06 0. 002 C 1 (Primary Controller 1) DDN S 2 A 9900 56402 1. 77299 E-05 0. 003 C 2 (Secondary Controller 2) DDN S 2 A 9900 56402 1. 77299 E-05 0. 003 D 1 (RAID System) 500 GB and 1 TB 7200 RPM SATA RAID 6 drives 1000000 per disk 0. 000001 0. 0002 NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 15

Results of RMA and Throughput Study System Reliability = [(DB 1+DB 2 -(DB 1)(DB 2)] * [S 1+S 2 -(S 1)(S 2)] * S 3 * [C 1+C 2 -(C 1)(C 2)] * D 1 = 0. 00702 = 1 - 0. 00702 = 0. 99298 Operational Availability = Where: Configuration MTBF = 11447. 8 hours Mean Time Between Maintenance (MTBM) = 10000 hours Mean Down Time (MDT) = 168 hours Average down-time events over 10000 hours = 0. 8735 Average down time = 146. 75 hours Ao = 10000/(10000+146. 75) Ao = 0. 9855 NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 16

Throughput Estimates • Demonstrated Performance: – Ingest Rate: 4. 05 -4. 4 TB/day – Tests ranged between 102660 – 146064 files – Files Ranged in Size Between 500 K-300 MB – Tested while 2 simultaneous FTP pulls (simulates the PEATEs) • Hardware – Single Disk Controller (S 2 A 8000) – 2 Dell Servers (2650/2850) NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 17

Throughput Estimates • Throughput Estimate: 8 -12 TB per day • Rationale: Our memory and CPU averaged at approximately 95% during our throughput tests. Also, our Disk Usage was near 90%. Our next hardware addresses these with: – 4 high performance servers(each with quad cores) – Dedicated database servers – 2 Disk Controllers (with 3 times the performance of our current disk controller) NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 18

Nominal Latency SD 3 E IDPS RDRs Push … … … RDRs Push In-Bound Land PEATE ASDC Out-Bound Ingest Every 5” Directory Poll Estimated every 12 hours, Land ingests data from SD 3 E Ingest PEATE Data Pull Ingest … … … Ingest Directory Poll Estimate every 12 hours, Land Ingests data from SD 3 E Ingest RDRs Push PEATE Data Pull Ingest … … … Ingest NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 19

Build 3 Plans • Add CERES RDRs products to SD 3 E product definition table • Obtain sample CERES RDR products to test transfer, ingest, verify, and store • Test interfaces between IDPS/ADS to SD 3 E to Land to CERES NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 20

Build 4 Plans • Continue testing interfaces between NESDIS IDPS to SD 3 E to Land PEATE to CERES • Continue performance and throughput testing from SD 3 E to Land PEATE to CERES NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 21

SD 3 E Capabilities by NCT 3 • Receive subscription and ad-hoc requests via Web Interface and machine-to-machine interface • Coalesce subscriptions from PEATEs/NICSE • Determine missing products • Determine products for reorder • Submit product request using DDS GUI to mini-IDPS • Ingest, verify, store x. DR, ancillary/auxiliary, IPs, and calibration products • Ingest, verify, store CEREs RDRs • Notify operator of products for manual reorder • Operator GUIs to monitor system status and resources • Report generation capability • Database replication • Ingest of RDR Packaged Filenaming (Primary Products ID first) • Work-around to ingest RDR Packaged Filenaming (sorted product IDs) NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 22

Schedule • Build 3 delivery 11/05/08 – Handle CERES RDRs – Handle RDR packaging file naming (including work around) • • • Build 3 I&T 11/06/08 - 11/21/08 Hardware Augmentation Part 1 08/01/08 Regression Testing 12/22/08 - 01/09/09 Performance Testing 11/04/08 - 11/18/08 Build 4 11/06/08 - 12/01/09 NCT 3 06/09/09 - 06/15/09 NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 23

Issues/Concerns • No Issues/Concerns NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 24

Backup NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 25

SD 3 E-Land-CERES DFD NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 26

CERES RDR Sizing Estimates CERES RDR File Size + 20% HDF 5 Overhead Science 1370. 84 Ki. B 1645 Ki. B Diagnostic 685. 45 Ki. B 822. 54 Ki. B Telemetry 27. 45 Ki. B 32. 94 Ki. B Total: 2083. 74 Ki. B 2500. 48 Ki. B NPP SDS SD 3 E CERES Delta Design Review SD 3 E- Page 27