Windows Implementationof DIRAC ON WINDOWS Client on Windows

  • Slides: 1
Download presentation
Windows Implementationof DIRAC ON WINDOWS Client on Windows Users can create and submit jobs:

Windows Implementationof DIRAC ON WINDOWS Client on Windows Users can create and submit jobs: § Full DIRAC Python API to specify: § application to be run; § input data (if required); § any precompiled libraries § C++ Application – jobs sent to Windows resources § Python based application – jobs are cross-platform. § DISET enabled client for secure job submission. § DIRAC job monitoring web page to view status of jobs. Applications developed by LHCb can be combined to form various types of jobs, ranging from production jobs (simulation, digitisation and reconstruction) to physics analysis. Software. Packages = { “Da. Vinci. v 12 r 15" from}; DIRAC. Client. Dirac import * Input. Sandbox dirac = Dirac() = job ={ Job() “Da. Vinci. opts” job. set. Application(‘Da. Vinci', 'v 12 r 15') }; job. set. Input. Sandbox(['Da. Vinci. opts’]) Input. Data = job. set. Input. Data([ { 'LFN: /lhcb/production/DC 04/v 2/00980000/DST/Presel_00980000_00001212. dst‘ ]) "LFN: /lhcb/production/DC 04/v 2/00980000/DS job. set. Output. Sandbox([‘Da. Vinci_v 12 r 15. log’]) T/Presel_00980000_00001212. dst" dirac. submit(job) }; Job. Name = “Da. Vinci_1"; Owner = "ying"; Std. Output = "std. out"; Std. Error = "std. err"; Output. Sandbox = { "std. out", "std. err", “Da. Vinci_v 12 r 15. log” }; Job. Type = "user"; JDL API DISET is the DIRAC security framework. It is used for user authentication and authorization on the Grid with X 509 certificates. DISET is based on: § Open. SSL § py. Open. SSL modified to allow for Grid proxy support and authentication. Libraries are built for Windows and provided alongside Linux libraries. DIRAC Services Based on Linux servers CASTOR storage Storage Access to the LHCb data distributed across the Grid Storage Elements is ensured by: • access to the LHCb LFC replica catalogue. • Grid. FTP. Net client developed by University of Virginia. • integrated with the DIRAC data management tools. Development of the SRM client for Windows is also foreseen. User Proxy DISET Proxy Server LFC Access On Windows, the DIRAC LFC portal is used to gain access to the LFC. • User’s DIRAC proxy is used via DISET to authenticate to the Proxy Server. • The Proxy Server is a trusted host for the LFC service, which uses its own credentials for authentication. • A user’s identity is then used to gain authorization for the LFC operations. Desktops, Laptops, Compute Clusters … Agent on Windows An Agent is launched from available Windows resource from the CLI, and communicate with DIRAC WMS and standard Grid services. The Agent will: § retrieve any matched jobs from WMS central Task Queue; § locate and transfer any required data via LFC and Grid. FTP protocol; § runs the job via the creation of a suitable. bat script on the resource backend (PC, Windows Compute Cluster); § report back to DIRAC server, giving update of job status; § perform any data transfers requested (e. g. requested outputsandbox files to DIRAC Sandbox Service, or Grid. FTP to Storage Elements). Windows Compute Cluster external communication Windows Compute Cluster Backend Head Node Job Matcher DIRAC Agent Job Management Service DIRAC API A Windows Compute Cluster backend has been implemented. The DIRAC Agent is installed on the Head node, which provides a gateway to the rest of the cluster. § Agent retrieves jobs onto the head node. § Agent checks the backend and wraps the job in the DIRAC Job Wrapper. § The Wrapper is sent to the worker node via the internal Compute Cluster Management and Scheduler Services. It is: § executed on the allocated worker node using the installed application on the head node. § able to control the job execution, reporting heart beats and other information to the DIRAC Monitoring Service. Windows Compute Cluster internal communication Sandbox Service Software Repository AC r R I D pe Job p a Wr Job Monitoring Service Watchdog Proxy Server DISET DIRAC WMS Done ü Cross-platform job submissions ü Job output retrieval (by DIRAC job ID or Castorage) ü Job processing (full analysis, stripping and reconstruction jobs) ü Tested on Windows Server 2003 and Windows XP platforms ü Deployed on Windows machines at Universities of Bristol, Cambridge and Oxford – more than 100 nodes in total ü System successfully used for physics analysis with simulated data Local SE LFC Service Future Plans • Continue Physics Analysis work using DIRAC on Windows systems • Port LHCb Simulation application to Windows, to enable full LHCb production job chain under Windows, with consequent gain in resources • Deploy system at new sites Y. Y. Li (University of Cambirdge) on behalf of the LHCb DIRAC team