Vulnerability Management at Scale Alexandre Fiori Production Engineer

Overview What’s this about The evolution of the Vulnerability Management program at Facebook, the birth of the Production Engineering team to support the program, and the pragmatic approach to build a highly scalable system for an ever-growing environment.

“Facebook was built on an open source stack. We support and encourage the use and development of open source software and hardware”

Vulnerability Management Timeline • … 2015: PCI DSS • 2016: Internal scanning tools • 2017: Broader network scanning • 2018: Reboot, build up • 2019: VMaa. S

2018

New challenge • Scan all infrastructure • Merge scanning technologies • Improve vulnerability matching • Start and track remediation • Manage remediation lifecycle

Architecture

Mindset • Big data • Scalability and reliability • On-line vs off-line pipelines • Concept validation and XFN • Fast prototype and launch

ETL

Collect Process Report

Collectors hardware operating systems application s Inventory Scan asset inventories and print to standard output

Processors Normaliz e Aggregat e Scan Vulnerabilities Pre-process, scan vulnerabilities, post-process

Reporters Notify Escalate Track Validate Cleanup Reporters manage escalation lifecycle Done

Inventory Classes • Hosted software • Installed software • Running software • Network scanners • Hardware

Vulnerability Database • Public vs Proprietary • Multi-vendor system • General purpose datasets • Specialized for ecosystem • Standard format for product->vulnerability matching

Implementation

Design Principles • Command line tools do one thing • Communicate over a text interface • Core functionality shared as libraries • Composable code, tools, and pipelines • Rely on well established UNIX conventions

Industry Standard Technology • MITRE / NIST / NVD • Common Platform Enumeration • Common Vulnerabilities and Exposures • Common Weakness Enumeration • Common Vulnerability Scoring System

Infrastructure • Tools • Services • Data warehouse • Dashboards • Notifications

Internal Pipelines • Tupperware • Container images and packages • First-party vs third-party codebase • Bad Binary Hunter and Buck • Attribution from package to service

Vulnerability Database Tools • nvdsync and $vendor 2 nvd • vulndb command line tool • Uses NVD CVE JSON 1. 0 format • Manages versioned datasets backed by My. SQL • Supports vendor snapshots, custom CVEs, and snoozes • vulndb thrift service • CVE lookup and CPE matching

Decoration • CWE and CVSS • Domain and sub-domain • First seen • Backlog vs influx • Owner (on-call ID or UNIX username) • Threat Intelligence

Remediation • Starts from CVE inventory • Depends on decoration data • Supports feedback loop (e. g. snoozes) • Understands release cycles per inventory class • Manages escalation and lifecycle

Lessons Learned • Normalization, aggregation, and blackholes • Per-customer decision trees are burdensome • Handling delays, XFN work, and fine tuning • Tasks and notification updates are annoying • False positives can compromise credibility

2019: VMaa. S

Goals • Self-service system for vulnerability scan • Custom aggregation defined by collectors • Configurable providers and thresholds • Tier-based service, starting from bronze • Default dashboards and reports per tier • Common CVE inventory for all customers

Progress • XFN partnership with early adopters • Migrated “hosted software” inventory class • Total of 10+ collectors in operation • Customers fixing bogus CVEs in the database • Snoozes effectively helping fine-tune reports

Next up • Migrate other inventory classes • Improve data quality and detection speed • Tackle backlog via remediation campaigns • Tackle influx via push-blocking scans • Influence company culture outside security org

Thank you

Q&A