LargeScale Deployments With Pogo At Yahoo Mike Schilli
- Slides: 74
Large-Scale Deployments With Pogo At Yahoo! Mike Schilli YAPC: : NA 06/14/2012
What’s Pogo? • Yahoo’s deployment system – Highly scalable (thousands of targets) – Highly parallelized rollouts – In active use for several years • Open Source version available (pre-alpha)
Pogo in a CI environment
What’s “Deployment”? • Run one command on many hosts • Each host “knows” what it’s supposed to look like via a central database • One command brings host into “desired” state • Fetches package updates from edge-server repos
Run one command on many hosts • Act on all hosts immediately
Run one command on many hosts • Act on all hosts immediately
Example Server Farm
Pogo Features • • • Sequences Parallelizes with constraints Halts if errors violate constraints Runs health checks (pre/post hooks) Command agnostic (rpm, apt-get, custom, etc. )
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Server Farm Release via Pogo
Pogo Configuration
Run a Pogo job (tags)
Pogo UI – Single Job View
Pogo UI Overview
Pogo UI – Host View
Pogo UI – Start it Up $ perl -Ilib bin/pogo-api-28> Listening on 0. 0: 7657 $ perl -Ilib bin/pogo-ui -v pogo-ui-22> Listening on port 5000
Run a Pogo job (targets)
Pogo Configuration
State of Pogo • Pre-alpha Open Source version on Github • OSS Pogo will be completed in the coming months
Pogo on Github • Developed in the open: – https: //github. com/ytoolshed/pogo • Internally used with plugins to adapt to specific data sources
CI Tests with travis-ci. org
The Gory Details
Pogo – User View
Pogo Workflow
Authentication/Authorization • Web server authentication for submitting jobs • Target authentication via passwords or priv/pub keys • Credentials encrypted with worker pubkey • Credentials never stored persistently (or Zoo. Keeper)
Pogo “Hooks” • Scripts that run before or after Pogo commands • Used to prepare host, perform check health, or other functions • Installed locally on target hosts • rc. d style invocation: run in alphanumeric order
Pogo “Hooks” (cont. ) • Two types: – pre hook: run before the specified command – post hook: run after the specified command • If any hook script fails: – No further scripts or commands run – Host is marked as failed
Pre-hook Examples • Take a host out of rotation • Drain database connections • Check that host is ready to be updated
Post-hook Examples • Check whether local webserver is serving expected content • Ping required services • Put host back into rotation
Host Failures and Concurrency • If a host fails: – Command returns non-zero – Pre- or post-hook returns non-zero • The failed host counts against the acceptable number/percent of unavailable hosts • Example: – Four hosts in an app can run at a time – One host in the app fails – Only three hosts will be acted on at a time going forward
Implementation • Any. Event Framework • Asynchronous, event-based • Added Object: : Event for event pub/sub
HTTP Client in Any. Event
HTTP Client in Any. Event
HTTP Client in Any. Event
Any. Event and Object: : Event
Any. Event and Object: : Event
Any. Event and Object: : Event
Queue Processor with Any. Event and Object: : Event
Any. Event bin/pogo-dispatcher
Testing Any. Event Components
All Pogo Components
All-In-One Pogo: pogo-one
All-In-One Pogo: pogo-one
All-In-One Pogo: pogo-one
Questions?
Thanks! • Pogo on Github: – http: //github. com/ytoolshed/pogo
Lessons learned along the Way
Set up Unit Tests with Travis-ci. org
Runs immediately after Github checkins
CI Tests with travis-ci. org
Configure CI with travis-ci. org
Select Github project
Authorize travis-ci. org
Travis-ci. org API Calls
Filling in Passwords
Filling in Passwords • • Test logging into machines without sshkeys Convenient for testing Answer to sudo prompts on targets Answer to prompts for encrypted packages
Filling in Passwords • Good password prompt: mschilli@localhost's password: • Not password prompt! … user ''@'localhost' (using password: NO)
Password. Monkey on CPAN
Plack
Plack
Plack/PSGI • Plack used for server-independent web application – API – Dispatcher Control Port
Alternatives • • Capistrano Rundeck Aegir Drush
Pogo UI – Start it Up $ perl -Ilib bin/pogo-api-28> Listening on 0. 0: 7657 $ perl -Ilib bin/pogo-ui -v pogo-ui-22> Listening on port 5000
Pogo UI • All Cient Side-Javascript • Server just redirects to index. html – Except assets/* js/*
Thanks! • Pogo on Github: – http: //github. com/ytoolshed/pogo • Password. Monkey CPAN Module: – http: //search. cpan. org/dist/Password. Monkey/ • Travis CI Service: – http: //travis-ci. org
- Yahoo pogo
- Combustion instability
- "deborah" @yahoo.com
- Video.search.yahoo.com
- Yahoo.com.yw
- Yahoo
- Yahoo
- Yahoo
- Yahoo
- Yahoo
- Yahoo com tw news
- Uuuu yahoo
- Yahoo
- Yahoo mail
- Yahoo
- Yahoo 3mail
- Images.search.yahoo
- Teoria neoclassica da administração
- Animales del mar con 9 letras
- Vmoracle
- Yahoo
- Yahoo weather rss
- Expectoration mousseuse rosée
- Defenition of insulator
- Mail handling
- Yahoo en español
- Yahoo
- Yerevan opera
- Yahoo design pattern library
- ฮาร์ดดิสก์
- Http /www.yahoo.com
- Yahoo tw news
- Yahoo
- Yahoo
- Yahoo
- Yahoo
- Yahoo
- Yahoo uuuu
- Yahoo
- Cigeg
- Pacsun mission statement
- Genentech valuation
- Yahoo
- Yahoo
- Yahoo
- Qué es una paráfrasis
- 怠惰 短気 傲慢
- Yahoo
- Nnn stock yahoo
- Syrosobserver
- Yahoo mail
- Mail @ malaysia.images.search.yahoo.com
- [email protected]
- Yahoo mmai
- Fracciones molares
- Yahoo hk news
- Wallcoo
- Yahoo
- Yahoo méxico
- Jago kluruk rame kapiyarsi artinya
- Uuuu yahoo
- Yahoo
- Yahoo internship
- Douglas crockford json
- Plica triangularis in tonsil
- Nineveh to tarshish
- Yahoo boss api
- Yahoo
- Yahoo
- Yahoo xrank vs rank
- Yahoo mail
- Yahoo user interface
- Yahoo search video
- Yahoo
- Pxlw yahoo finance