Whats new in HTCondor Whats coming ISGC 2019

What’s new in HTCondor? What’s coming? ISGC 2019 Taipei March 2019 Todd Tannenbaum Center for High Throughput Computing Department of Computer Sciences University of Wisconsin-Madison

Release Series › Stable Series (bug fixes only) h. HTCondor v 8. 8. x - introduced Jan 2019 › Currently at v 8. 8. 1 Development Series (should be 'new features' series) h. HTCondor v 8. 9. x Currently at v 8. 9. 0 › Detailed Version History in the Manual http: //htcondor. org/manual/latest/Version. Historyand. Release. Notes. html 2

3

Enhancements in HTCondor v 8. 4 › Scalability and stability h Goal: 200 k slots in one pool, 10 schedds managing 400 k jobs › › › › Introduced Docker Job Universe IPv 6 support Tool improvements, esp condor_submit Encrypted Job Execute Directory Periodic application-layer checkpoint support in Vanilla Universe Submit requirements New RPM / DEB packaging Systemd / SELinux compatibility 4

Enhancements in HTCondor v 8. 6 › Enabled and configured by default: use › › single TCP port, cgroups, mixed IPv 6 + IPv 4, kernel tuning Made some common tasks easier Schedd Job Transforms Docker Universe enhancements: usage updates, volume mounts, conditionally drop capabilities Singularity Support 5

HTCondor Singularity Integration › What is Singularity? Like Docker but… h. No root owned daemon process, just a setuid h. No setuid required (post RHEL 7) h. Easy access to host resources incl GPU, network, file systems › HTCondor allows admin to define a policy (with access to job and machine attributes) to control h. Singularity image to use h. Volume (bind) mounts h. Location where HTCondor transfers files 6

Whats new in v 8. 8 and beyond 7

Docker Job Enhancements › Docker jobs get usage › › updates (i. e. network usage) reported in job classad Admin can additional volumes Conditionally drop capabilities Condor Chirp support Support for condor_ssh_to_job h. For both Docker and Singularity 8

Not just Batch Interactive Sessions › Two uses for condor_ssh_to_job h. Interactive session alongside a batch job • Debugging job, monitoring job h. Interactive session alone (no batch job) • Juptyer notebooks, schedule shell access • p. s. Jupyter Hub batchspawner supports HTCondor › Can tell the schedd to run a specified job immediately! Interactive sessions, test jobs hcondor_now <job_id_to_run> <job_id_to_kill> h. No waiting for negotiation, scheduling 9

Higher Level Python Abstractions › HTCondor Bindings (import htcondor) are steeped in the HTCondor ecosystem h. Exposed to concepts like Schedds, Collectors, Class. Ads, jobs, transactions to the Schedd, etc › Working on a new package (e. g. import htcondor_ez … ? Name suggestions? ) h. More approachable by Python addicts • No HTCondor concepts to learn, just extensions of natural and familiar Python functionality h. Targeting i. Python users as well (aka Jupyter) h. Written on top of htcondor bindings › Also working on HTMap package 10

Example: y = f(x) See https: //github. com/htcondor/htmap 11

HTCondor "Annex" › Instantiate an HTCondor Annex to dynamically additional execute slots for jobs submitted at your site h. Get status on an Annex h. Control which jobs (or users, or groups) can use an Annex › Want to launch an Annex on h. Clouds • Via cloud APIs (i. e. Kubernetes? ) h. HPC Centers / Supercomputers • Via edge services (i. e. HTCondor-CE) 12

Grid Universe › Reliable, durable submission of a job to a remote scheduler › Popular way to send pilot jobs (used by glidein. WMS), key › component of HTCondor-CE Supports many “back end” types: h HTCondor h PBS h LSF h Grid Engine h Google Compute Engine h Amazon AWS h Open. Stack h Cream h Nordu. Grid ARC h BOINC h Globus: GT 2, GT 5 h UNICORE 13

V 8. 8 Added Grid Universe support for Azure, SLURM, Cobalt, soon k 8 s? › › Speak to Microsoft Azure Speak native SLURM protocol Speak to Cobalt Scheduler Soon? Speak to Kubernetes! Jaime: Grid Jedi Also HTCondor-CE "native" package › HTCondor-CE started as an OSG package › IN 2 P 3 wanted HTCondor-CE without all the OSG › dependencies…. Now HTCondor-CE available stand-alone in HTCondor repositories 14

CPU cores! FNAL HEPCloud NOv. A Run (via Annex at NERSC) http: //hepcloud. fnal. gov/ 15

No internet access to HPC edge service? File-based Job Submission Schedd A Job. XXX request input Schedd B status. 1 status. 2 status. 3 output 16

Compute node management enhancements › Work on Noisy Neighbor Challenges h. Already use cgroups to manage CPU, Memory… what about CPU L 3 cache? Memory bus bandwidth? h. Working with CERN Open. Lab and Intel on leveraging Intel Resource Directory Technology (RDT) in HTCondor • Monitor utilization • Assign shares 18

Compute node management enhancements, cont. › GPU Devices › HTCondor can detect GPU devices and › schedule GPU jobs New in v 8. 8: h. Monitor/report job GPU processor utilization h. Monitor/report job GPU memory utilization › Future work: simultaneously run multiple jobs on one GPU device h. Volta hardware-assisted Mutli-Process Service (MPS) 20

Security: From identity certs to authorization tokens › HTCondor has long supported GSI certs › Then added Kerberos/AFS tokens for CERN, DESY › Now adding standardized token support h. Sci. Tokens (http: //scitokens. org) h. OAuth 2. 0 Workflow Box, Google Drive, Github, … 21

Security, cont. › Planning for US Federal Information Processing Standard (FIPS) Compliance h. Can do better than MD-5, 3 DES, Blowfish h. AES has hardware support in most Intel CPUs, so looking at just doing TLS all the time by default › May motivate us to drop UDP communications in HTCondor h. Almost all communication in HTCondor is now asynchronous TCP anyway h. Anyone care if UDP support disappears? 22

Data work › Job input files normally transferred to execute node over CEDAR, now can be sent over HTTP h. Enable caching (reverse and forward proxies), redirects › Job selectivlely send output files back to › submit host or a 3 rd party (web URL, Box, etc) Integrity check per file 23

Workflows › Thinking about how to add "provisioning nodes" into a DAGMan workflow h. Provision an annex, run work, shutdown annex › Working with Toil team so now Toil workflow engine can submit jobs into HTCondor 24

Scalability Enhancements › Central manager now manages queries h. Queries (ie condor_status calls) are queued; priority is given to operational queries › More performance metrics (e. g. in collector, › DAGMan) Late materialization of jobs in the schedd to enable submission of very large sets of jobs h. Submit / remove millions of jobs in < 1 sec h. More jobs materialized once number of idle jobs drops below a threshold (like DAGMan throttling) 25

Late materialization This submit file will stop adding jobs into the queue once 50 jobs are idle: executable = foo. exe arguments = -run $(Process. Id) materialize_max_idle = 50 queue 1000000 26

Thank you! 27