DPM performance tuning hints for HTTPWeb DAV and

DPM performance tuning hints for HTTP/Web. DAV and Xrootd Andrea Manzi CERN On behalf of the DPM team HEPi. X Fall 2014 Workshop IT-SDC : Support for Distributed Computing 16/10/2014 1

Outline § DPM Overview § Performance Assessment § Internal Testbed § CMS AAA § Performance Tuning § Results § Summary IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 2

Disk Pool Manager (DPM) § The Disk Pool Manager (DPM) is a lightweight storage solution for grid sites. It offers a simple way to create a disk-based grid storage element and supports relevant protocols for file management and access. § It focuses on manageability (ease of installation, configuration, low effort of maintenance), while providing all required functionalities for a grid storage solution: § support for multiple disk server nodes § multiple file replicas in disk pools § multi VOs support § multi protocol: xrootd, HTTP/Web. DAV, Grid. FTP, SRM IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 3

DPM Collaboration § Started in 2013 in order to maintain, develop and support the DPM Software stack for the benefit of the user communities members of the collaboration: § CERN, CESNET, Grid. PP, Japan, France, Italy and ASGC. IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 4

Deployment statistics § DPM is the most widely deployed grid storage system in EGI and WLCG § Over 200 sites in 50 regions § Over 300 VOs § WLCG statistics: § http: //wlcgmon. cern. ch/dashboard/request. py/siteview#current. V iew=Storage+Deployment § ~50 PB (10 sites with > 1 PB) IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 5

Emphasis § DPM emphasis has moved to better support Xrootd and HTTP/Web. Dav protocols w. r. t HEP historical data access and transfers ( e. g. SRM, RFIO) § Xrootd Federations are fundamental for some LHC experiments computing models § AAA for CMS and FAX for ATLAS § HTTP/Webdav based federation for LHCb is under evaluation and a prototype available: § http: //federation. desy. de/fed/lhcb/ IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 6

New Dmlite Architecture Grid FTP SRM RFIOD DPM DPNS Legacy ADAPTER XRoot. D Web DAV MEM MYSQL DPM CACHEHEAD NS) DMLITE The goal is to reduce interactions towards legacy components ( bottleneck for performance) IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 7

Internal Performance Assessment § For ~1 year we have been improving our sets of tests headed to METADATA performance and reliability under load § Goals are: § Perf tests: Regular maximum stress exercises on our internal testbeds § quick response: evaluate day by day any change in how the testbeds perform IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 8

Internal Performance Assessment § Nightly performance tests run towards the CERN dev testbeds § Very heavy stress generators § Protocols: HTTP, Web. Dav, Xrootd § Concurrency: sweeping from 1 to 1000 § Tasks: § Peak metadata performance: repeated stat of 10 k files by 1 -1000 clients § Small reads from each file ( few bytes) § HW: 4 core, 8 GB mem, 1 GB Net IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 9

Performance Results: 1. 8. 8 release Web. Dav peak stat performance IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 10

Performance Results: 1. 8. 8 release HTTP small file GET peak performance IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 11

Performance Results: 1. 8. 8 release Xrootd stat rate IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 12

The CMS AAA tests § CMS has started a performance assessment campaign to understand if sites can sustain the expected load for LHC Run 2 § Thanks to F. Fanzago for the info and the graphs! § CMS target for tests: § File-opening test: access total rate of 100 Hz at a site § Very aggressive test, as the entire CMS infra would access a single storage element § https: //indico. cern. ch/event/324705/contribution/17/ material/slides/1. pdf IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 13

Opening Plots for some DPM sites Federica Fanzago - INFN Padova – Tests run up 100 jobs simultaneously, opening files at rate of 2 Hz each. TEST TARGET IS 100 Hz DPM performance tuning hints for HTTP/Web. DAV and Xrootd 14

Performance Tuning § The internal performance tests + interaction with CMS sites with low performance has highlighted many performance bottlenecks and configuration parameters to be tuned § The new 1. 8. 9 release is the result of this activity § Caching to avoid DPNS/DPM services interaction § WIKI for Performance Tuning valid for 1. 8. 8 too § https: //svnweb. cern. ch/trac/lcgdm/wiki/Dpm/A dmin/Tuning. Hints IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 15

Tuning Hints § Some tuning hints are related to Mysql connections and pool size § max connection=1000 § Ns. Pool. Size= 256 § Others to number of threads for the legacy daemons: § DPM-> NB_FTHREADS=60 § DPNS-> NB_THREADS=80 § Limits also are important § § hard nofile 65000 soft nproc 65000 soft nofile 65000 hard nproc 65000 IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 16

Tuning Hints: Memcache § Usage of the Memcache plugin with a local instance or external cluster of Memcached server(s). Grid FTP Mem cached ADAPTER XRoot. D Web DAV MEM MYSQL DPM CACHEHEAD NS) DMLITE § 2 GB cache with 256 connections pool size IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 17

Performance Tuning: 1. 8. 8 Federica Fanzago - INFN Padova Before tuning Strong collaboration between site manager and DPM developers Optimization of DPM configuration parameters Performance not optimal due to headnode’s hardware After tuning 18

Performance Tuning: 1. 8. 9 Web. Dav peak performance in 1. 8. 9 … a 5 X increase, no more clogging IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 19

Performance Tuning: 1. 8. 9 HTTP small file GET peak performance in 1. 8. 9, 20 X increase IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 20

Performance Tuning: 1. 8. 9 Xrootd stat rate with DPM 1. 8. 9, 4 X increase IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 21

Performance Tuning: Puppet § Configuration changes are now possible also with Puppet as it has become the official DPM configuration mechanism ( Yaim is deprecated) § We have been developing a series of modules to configure the various plugins and frontends § Available on Puppet. Forge and Git. Hub § https: //github. com/cern-it-sdc-id/ § https: //forge. puppetlabs. com/lcgdm IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 22

Summary § Good synergy between Experiments, Site Managers and DPM Developers § DPM Xrootd and HTTP/Web. Dav Federations are ready for Run 2 § DPM 1. 8. 8 might satisfy the modern perf requirements, but needs powerful hardware § DPM 1. 8. 9 should meet them with more normal machines, e. g. 4 cores, 8 G mem, good disks, average network. IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 23

DPM HTTPS: //SVNWEB. CERN. CH/TRAC/LCGDM QUESTIONS? IT-SDC DPM performance tuning hints for HTTP/Web. DAV and Xrootd 16/10/2014 24