Docker Hub Past present and future We are
Docker Hub Past, present and future
We are engineers working on Docker Hub Ken Cochrane @Ken. Cochrane bc Wong @bcwalrus
Docker Hub (hub. docker. com) is a cloud registry service for sharing application and automating workflows. Major features: • Public and Private Docker repositories • Official Repositories • Collaborators, Organizations and Groups • Automated builds • Webhooks 3
Docker Hub - A brief history
Hub by the Numbers • 240, 000 Users • 150, 000 Repositories • 1, 250, 000 Automated Builds • 500, 000 Pulls 5
Hub Growth (disk space) 6
Hub Growth (bandwidth) 7
Growth of a distributed application
Docker Hub: This is your life
Docker Hub: This is your life (cont. )
Docker Hub: This is your life (cont. )
Docker Hub: This is your life (cont. )
Docker Hub: This is your life (cont. )
Docker Hub: This is your life (cont. )
Docker Hub: This is your life (cont. )
Docker Hub: This is your life (cont. )
Docker Hub: This is your life (cont. )
Docker Hub Timeline 18
Docker Hub - More than just a registry
Organizations & Groups • • • 20 Best suited for businesses Collaborate on repos with members of your organization Fine-grain permissions Add users to groups Public and Private repos
Official Repositories • • 21 80 Official Repos that you can use to build your Docker applications Repos are updated and maintained by Docker
Private Repositories • • Not searchable or browsable Can add collaborators to push or pull Only you or a collaborator can access 1 free private repo with your docker hub user account Coupon for 3 free months of the micro plan dockercon 15 22
Automated Builds • • 23 Link to your Github or Bitbucket accounts Keep your source code in sync with your docker repos You write the code, we handle the build, and keep your Docker hub repos up to date. Public and Private repos
Repository Links What if your base ubuntu: 14. 04 image is updated for a security fix? • • 24 Link your automated build repository to another repository Allows you to keep your automated build up to date
Build Triggers • • 25 Trigger an automated build from an external system Useful for integrating with CI systems
Webhooks • • • 26 Get notified when a docker repo is updated Ability to chain together webhooks to create a build pipeline Combine with a continuous integration and deployment solution to easily move your application from development to server.
And much more. . . • • 27 Search Stars Comments Notifications
Upcoming Features • Show image sizes • Git. Lab support for automated builds • Dynamic tagging for automated builds 28
Docker Hub 2. 0 (beta)
Hub 2. 0 • • • 30 New, Easier to use UI Re-architected, isomorphic javascript application based on react. js, react-router and fluxible A modern, modular approach using ES 2015 enabling faster iterations
New Architecture Web UI API Gateway Micro Services Postgresql Redis Elastic Search Rabbit. MQ Data stores Kafka S 3
Live demo time: Hub 2. 0 hub-beta. docker. com Left brain: What could go wrong? Right brain: Some people never learn.
Give us feedback please github. com/docker/hub-beta-feedback 33
Docker Hub - Performance
Docker Hub Performance - push/pull 35
Architecture Upgrade - Registry v 2 80% fewer requests 60% less bandwitdth • • 36 Registry v 2 protocol is far superior to v 1 Supported by Docker >=1. 6 Content-addressable manifest More secure
Architecture Upgrade - Disentangle • Clean up inter-service dependencies: • • 37 Hub 2. 0 uses a gateway with finer grain services. Reduces cascading failures.
Infrastructure Upgrade • Aka “throw money at the problem” • The difficult parts are when (i. e. monitoring), and the execution. • E. g. database upgrade: DB response time (before vs after) 38 External ping latency (worldwide average)
Better Service Isolation 39
Automated Builds (v 1 current) Architecture • A set of long-running build servers w/o auto-scaling • One build per server at a time (docker in docker) • Thousand’s of builds per day Challenges • # of builds <= # of servers, which means pending builds • Security concerns • Build server issue == lost capacity 40
Automated Builds (v 2 upcoming) Speed and efficiency • Auto-scale on-demand builds • Per-user queuing • Better scheduling Security • Use VM for isolation between users • TLS everywhere • Ephemeral build servers 41
For those about to rock. . .
Scalability & Reliability Work: Myth vs Reality n fu l l i t s s ey, it’ H Need a team that can do the first 80% and the last 20%. Need a culture that understands quality. 43
Why is my sink not draining? From our post-mortems & bug reports: • DB capacity • ORM layer makes slow queries • Networking flakiness • Rabbit. MQ failures (and we didn’t understand its failure models) • Supervisor failed to restart appserver, but NGINX keeps accepting calls • Expired intermediate cert • Docker btrfs driver bug • . . . Know when to tighten a bolt, and when to get a new sink. 44
High Velocity Cycle (invest in tooling) Features & fixes Monitoring & logging (what’s slow) QA, scale test, perf regression test (prove that things work) 45
Future Performance Work • Continue to eliminate bottlenecks as Hub grows • Speed up push/pull outside the US • Scalable pipeline into data warehouse 46
Credits: The Hub Team Ryan Abrams Chris Biscardi Ken Cochrane John Costa Roberto Hashioka Daniel Hiltgen Nathan Hsieh Nick Kraly Adrian Kreher Toli Kuznets Dustin Lacewell 47 Nick Lang Sean Li Amy Lindburg Marcus Martins Chinmayee Nirmal Atul Phadnis Mario Ponticello Arunan Rabindran Josh South Justen Stepka bc Wong
Credits: The Hub Team
Questions? Remember to check out hub-beta, and send us feedback: github. com/docker/hub-beta-feedback
- Slides: 49