Docker Hub Past present and future We are

Docker Hub Past, present and future

We are engineers working on Docker Hub Ken Cochrane @Ken. Cochrane bc Wong @bcwalrus

Docker Hub (hub. docker. com) is a cloud registry service for sharing application and automating workflows. Major features: • Public and Private Docker repositories • Official Repositories • Collaborators, Organizations and Groups • Automated builds • Webhooks 3

Docker Hub - A brief history

Hub by the Numbers • 240, 000 Users • 150, 000 Repositories • 1, 250, 000 Automated Builds • 500, 000 Pulls 5

Hub Growth (disk space) 6

Hub Growth (bandwidth) 7

Growth of a distributed application

Docker Hub: This is your life

Docker Hub: This is your life (cont. )

Docker Hub Timeline 18

Docker Hub - More than just a registry

Organizations & Groups • • • 20 Best suited for businesses Collaborate on repos with members of your organization Fine-grain permissions Add users to groups Public and Private repos

Official Repositories • • 21 80 Official Repos that you can use to build your Docker applications Repos are updated and maintained by Docker

Private Repositories • • Not searchable or browsable Can add collaborators to push or pull Only you or a collaborator can access 1 free private repo with your docker hub user account Coupon for 3 free months of the micro plan dockercon 15 22

Automated Builds • • 23 Link to your Github or Bitbucket accounts Keep your source code in sync with your docker repos You write the code, we handle the build, and keep your Docker hub repos up to date. Public and Private repos

Repository Links What if your base ubuntu: 14. 04 image is updated for a security fix? • • 24 Link your automated build repository to another repository Allows you to keep your automated build up to date

Build Triggers • • 25 Trigger an automated build from an external system Useful for integrating with CI systems

Webhooks • • • 26 Get notified when a docker repo is updated Ability to chain together webhooks to create a build pipeline Combine with a continuous integration and deployment solution to easily move your application from development to server.

And much more. . . • • 27 Search Stars Comments Notifications

Upcoming Features • Show image sizes • Git. Lab support for automated builds • Dynamic tagging for automated builds 28

Docker Hub 2. 0 (beta)

Hub 2. 0 • • • 30 New, Easier to use UI Re-architected, isomorphic javascript application based on react. js, react-router and fluxible A modern, modular approach using ES 2015 enabling faster iterations

New Architecture Web UI API Gateway Micro Services Postgresql Redis Elastic Search Rabbit. MQ Data stores Kafka S 3

Live demo time: Hub 2. 0 hub-beta. docker. com Left brain: What could go wrong? Right brain: Some people never learn.

Give us feedback please github. com/docker/hub-beta-feedback 33

Docker Hub - Performance

Docker Hub Performance - push/pull 35

Architecture Upgrade - Registry v 2 80% fewer requests 60% less bandwitdth • • 36 Registry v 2 protocol is far superior to v 1 Supported by Docker >=1. 6 Content-addressable manifest More secure

Architecture Upgrade - Disentangle • Clean up inter-service dependencies: • • 37 Hub 2. 0 uses a gateway with finer grain services. Reduces cascading failures.

Infrastructure Upgrade • Aka “throw money at the problem” • The difficult parts are when (i. e. monitoring), and the execution. • E. g. database upgrade: DB response time (before vs after) 38 External ping latency (worldwide average)

Better Service Isolation 39

Automated Builds (v 1 current) Architecture • A set of long-running build servers w/o auto-scaling • One build per server at a time (docker in docker) • Thousand’s of builds per day Challenges • # of builds <= # of servers, which means pending builds • Security concerns • Build server issue == lost capacity 40

Automated Builds (v 2 upcoming) Speed and efficiency • Auto-scale on-demand builds • Per-user queuing • Better scheduling Security • Use VM for isolation between users • TLS everywhere • Ephemeral build servers 41

For those about to rock. . .

Scalability & Reliability Work: Myth vs Reality n fu l l i t s s ey, it’ H Need a team that can do the first 80% and the last 20%. Need a culture that understands quality. 43

Why is my sink not draining? From our post-mortems & bug reports: • DB capacity • ORM layer makes slow queries • Networking flakiness • Rabbit. MQ failures (and we didn’t understand its failure models) • Supervisor failed to restart appserver, but NGINX keeps accepting calls • Expired intermediate cert • Docker btrfs driver bug • . . . Know when to tighten a bolt, and when to get a new sink. 44

High Velocity Cycle (invest in tooling) Features & fixes Monitoring & logging (what’s slow) QA, scale test, perf regression test (prove that things work) 45

Future Performance Work • Continue to eliminate bottlenecks as Hub grows • Speed up push/pull outside the US • Scalable pipeline into data warehouse 46

Credits: The Hub Team Ryan Abrams Chris Biscardi Ken Cochrane John Costa Roberto Hashioka Daniel Hiltgen Nathan Hsieh Nick Kraly Adrian Kreher Toli Kuznets Dustin Lacewell 47 Nick Lang Sean Li Amy Lindburg Marcus Martins Chinmayee Nirmal Atul Phadnis Mario Ponticello Arunan Rabindran Josh South Justen Stepka bc Wong

Credits: The Hub Team

Questions? Remember to check out hub-beta, and send us feedback: github. com/docker/hub-beta-feedback