Network Security Effective Practices Vo IP SIP H

  • Slides: 65
Download presentation
Network Security Effective Practices - Vo. IP: SIP, H. 323 A Survey of Vo.

Network Security Effective Practices - Vo. IP: SIP, H. 323 A Survey of Vo. IP Security Practices in Higher Education H. Morrow Long Director, Information Security Yale University Educause 2007 Annual Conference Session Wednesday, October 24, 2007 11: 30 a. m. - 12: 45 p. m.

Introductions 2

Introductions 2

Overview This presentation will discuss a survey and informal poll of the current campus

Overview This presentation will discuss a survey and informal poll of the current campus network Vo. IP security practices and products in higher education on both wired and wireless networks. 3

Agenda Introduction What is Vo. IP? Vo. IP Threats Vo. IP Security Checklists Vo.

Agenda Introduction What is Vo. IP? Vo. IP Threats Vo. IP Security Checklists Vo. IP Effective Practices in Higher Ed Survey of Vo. IP Security in Academia Discussion and Questions 4

Vo. IP Security Goes Mainstream In 2006, Vo. IP Security entered the SANS Top

Vo. IP Security Goes Mainstream In 2006, Vo. IP Security entered the SANS Top 20 for the first time: http: //www. sans. org/top 20/#n 1 N 1 Vo. IP Servers and Phones 5

Vo. IP Security Flaws Go Mainstream 2006 Vo. IP Security vulnerabilities: Asterisk CVE-2006 -2898,

Vo. IP Security Flaws Go Mainstream 2006 Vo. IP Security vulnerabilities: Asterisk CVE-2006 -2898, CVE-2006 -4345, CVE-2006 -4346, CVE-2006 -5444 Cisco Call Manager CVE-2006 -0368, CVE-2006 -3594 Vo. IP Phones CVE-2005 -3717, CVE-2005 -3722, CVE-2005 -3723, CVE-2006 -0305, CVE-2006 -0374, CVE-2006 -0834, 6 CVE-2006 -5038

Vo. IP Security Flaws Go Mainstream 2007 Vo. IP Security vulnerabilities: Asterisk CVE-2007 -1306

Vo. IP Security Flaws Go Mainstream 2007 Vo. IP Security vulnerabilities: Asterisk CVE-2007 -1306 Cisco Call Manager / IOS / PIXOS CVE-2007 -0648, SA 24180/cisco-sa-20070214 -fwsm, SA 24179/cisco-sa-20070214 -pix Vo. IP Phones CVE-2007 -1072, CVE-2007 -1063 7

What is Vo. IP? Voice over IP IP Telephony Converged Data/Voice Networking Unified Messaging

What is Vo. IP? Voice over IP IP Telephony Converged Data/Voice Networking Unified Messaging 8

What is Vo. IP? 2 Major Protocols: § H. 323 § SIP / SIPS

What is Vo. IP? 2 Major Protocols: § H. 323 § SIP / SIPS Popular Internet Vo. IP § Proprietary • Skype • Vonage Other § Zfone/ZRTP (Phil Zimmerman) Internet Standards related to Vo. IP Security: § IPSEC § SSL/TLS § SRTP (RFC 3711) 9

H. 323 and SIP The 2 Major (Local and Enterprise) Vo. IP Protocols: §

H. 323 and SIP The 2 Major (Local and Enterprise) Vo. IP Protocols: § H. 323 § SIP Both protocols: Are hard (but not impossible) to firewall Were not designed for security… Use separate signaling and media (content) channels Use dynamic ports Were not designed to be NAT “friendly” (embed IP addresses inside signaling/control information) But: H. 323 is more like ISO X. protocols (uses ASN. 1/PER) and SIP is more like Internet FTP/SMTP/HTTP/NNTP. 10

H. 323 Older protocol than SIP, implemented earlier ITU Umbrella Standard - built of

H. 323 Older protocol than SIP, implemented earlier ITU Umbrella Standard - built of other H stds First Vo. IP std to use RTP Interoperates with ISDN PBX systems Used by several voice and videoconferencing systems Built into Net. Meeting, other commercial and open source programs available GNU Gatekeeper - accounting/authorization/NAT traversal/H. 323 proxy/H. 235 security 11

H. 235 Security H. 235 provides security for H. 323 Optionally nine security profiles

H. 235 Security H. 235 provides security for H. 323 Optionally nine security profiles can be used to apply one or more of six security services (authentication, nonrepudiation, integrity, confidentiality, access control, key management) to H. 225, H. 245 and RTP traffic. 12

“Skinny” - Cisco H. 323 “Skinny” is Cisco’s lightweight proprietary version of H. 3323.

“Skinny” - Cisco H. 323 “Skinny” is Cisco’s lightweight proprietary version of H. 3323. SCCP is the acronym for Skinny Client Control Protocol. It is a lower overhead control protocol between the client and Call Manager. 13

SIP - Session Initiation Protocol Overtaking H. 323 on LANS - many clients. Created

SIP - Session Initiation Protocol Overtaking H. 323 on LANS - many clients. Created 1996. SIP 2. 0 defined in RFC 2543 (1999)-refined in RFC 3261 (2002). Lightweight, text-based protocol run on top of UDP or TCP (e. g. port 5060 - mod P 2 P model. Uses HTTP “style” status codes & email addresses. Interoperates with XMPP IM (Japper) STUN & newer TURN enable SIP through NAT using public Internet servers. Uses other protocols: SDR, RTP, MGCP, RTSP. Can be stateful/less, client/server or P 2 P. 14

SIP/RTP Architecture Vo. IP User-Agent RTP SIP SRTP TLS UDP and TCP IP Data

SIP/RTP Architecture Vo. IP User-Agent RTP SIP SRTP TLS UDP and TCP IP Data Link Layer Credit: Practical Vo. IP Security, Syngress 15

SIPS - Secure SIP is a security mechanism defined by SIP RFC 3261 (2002)

SIPS - Secure SIP is a security mechanism defined by SIP RFC 3261 (2002) defines Secure SIP -- a security mechanism using TLS to send SIP messages over an (Transport Layer Security) encrypted channel. Fairly new, competes with IPSEC, VPNs, SRTP -often referred to as SIP with TLS -- used when IPSEC is overkill or SIP proxies must be used. 16

SRTP Adds message encryption, authentication, integrity and replay protection to to RTP Sister to

SRTP Adds message encryption, authentication, integrity and replay protection to to RTP Sister to SRTCP (Secure RTP Control Proto) SRTP/SRTCP encryption, authentication and integrity are independent and can be disabled (“Null” encryption). Single Cipher (AES), 2 modes (counter & feedback modes) External Key mgt (ZRTP, Mikey, …) Credit: http: //en. wikipedia. org/wiki/Secure_Real-time_Transport_Protocol 17

SRTP Interoperability Hard IP Phones § Avaya, Cisco, Ericsson (&TLS), Siemens, Linksys, Snom (&TLS)

SRTP Interoperability Hard IP Phones § Avaya, Cisco, Ericsson (&TLS), Siemens, Linksys, Snom (&TLS) Soft IP phones § Gizmo, Kphone, Snom 360 (&TLS), minisip (&TLS) Hard IP PBX - Alcatel and Ericsson Soft-IP-PBX - Asterisk (SIP & H 323) and pbxnsip SBC (Session Border Ctrlr) / SIP Firewall § Covergence (& SIP & H 323) § In. Gate (SIP aware firewall) Credit: http: //en. wikipedia. org/wiki/Secure_Real-time_Transport_Protocol 18

Zfone/ZRTP Created/driven by (Phil Zimmerman) 2 nd attempt (PGPfhone) Designed to work with current

Zfone/ZRTP Created/driven by (Phil Zimmerman) 2 nd attempt (PGPfhone) Designed to work with current SIP phone programs (via plug-ins). Zfone is the program. ZRTP is an extension to RTP (Real-time Transport Protocol) providing secure real-time transport to secure sessions (SIP, H. 323, etc. ) already established. § Keys are transmitted and managed outside the std signaling. § Protection against Mit. M (man in the Middle) attacks. 19

Skype Peer to Peer Model Supernodes route traffic for other calls Can be blocked

Skype Peer to Peer Model Supernodes route traffic for other calls Can be blocked and bandwidth managed Outlawed at some institutions Proprietary strong encryption Non-CALEA compliance? 20

More Vo. IP Terminology “Presence” (R U there? ) Convergence (Data + Voice =

More Vo. IP Terminology “Presence” (R U there? ) Convergence (Data + Voice = Synergy) Voice Messaging Unified Messaging Systems 21

More Vo. IP Acronyms ACD IVR ICE RSVP RTSP SDP STUN TLS TURN TTS

More Vo. IP Acronyms ACD IVR ICE RSVP RTSP SDP STUN TLS TURN TTS Automatic Call Distribution(Call Ctr) Interactive Voice Response Interactive Connectivity Establishment Resource Reservation Protocol Real Time Streaming Protocol Session Discovery Protocol Simple Traversal of UDP through NAT Transport Layer Security (ala SSLv 3) Traversal Using Relay NAT Text-to-speech server 22

Non-Cyber Securityrelated Vo. IP Issues 911 - where does 911 ring? E-911 - need

Non-Cyber Securityrelated Vo. IP Issues 911 - where does 911 ring? E-911 - need to provide location information? Emergency access § during network or power outages • Use Power-over-Ethernet (Po. E AKA IEEE 802. 3 af) cabling • Provide at least the minimal # of land lines per # rooms (e. g. or as required by law) 23

PBX System Components PSTN Endpoints (Phones, Faxes, Modems. ) Lines (e. g. Station lines)

PBX System Components PSTN Endpoints (Phones, Faxes, Modems. ) Lines (e. g. Station lines) Trunks Remote PBXes Adjuncts (VM, ACD, IVR, …) CDR (Call Detail Recording) Voice/PBX Firewalls 24

Vo. IP System Components Media Gateways -- e. g. to PSTN/PBXes Endpoints (User Agents):

Vo. IP System Components Media Gateways -- e. g. to PSTN/PBXes Endpoints (User Agents): softphones, IM/Video/Vo. IP/ATA (Analog Telephone Adatper) Media Servers (VM, ACD, IVR, TTS, VC) H. 323 Gatekeepers SIP Registration, Redirect Servers SIP Proxy Servers Firewalls/ALGs 25

Vo. IP Threats Vo. IP Networks have many of the same threats to security,

Vo. IP Threats Vo. IP Networks have many of the same threats to security, privacy and reliability as data networks do, but they also bring in the problems of the telephone system and have some special threats all their own. Converged networks can combine threats from the data and Vo. IP world -- making the new network less secure (in the opinion of some). Data network people are afraid Vo. IP infrastrucutre will weaken the security of their data network and the voice/telecom people feel the same about data / IP networks. 26

Other Vo. IP Architectures Skype IAX H. 248 Microsoft Live Communication Server 2005 (MLCS)

Other Vo. IP Architectures Skype IAX H. 248 Microsoft Live Communication Server 2005 (MLCS) § TLS between client and server § Mutual TLS server-to-server 27

Vo. IP vs. PSTN Remember that “POTS” telephones have little security -- ordinary phone

Vo. IP vs. PSTN Remember that “POTS” telephones have little security -- ordinary phone conversations are not encrypted and can be tapped or eavesdropped. You can actually have better security using Vo. IP IF you use strong encryption (and a good implementation). 28

Vo. IP Threats DDo. S / Do. S Attacks § ICMP Flood (eg ‘pings’)

Vo. IP Threats DDo. S / Do. S Attacks § ICMP Flood (eg ‘pings’) to Phone or Call Mgr Unauthorized Access Toll Fraud Voicemail hacking Eavesdropping (Call and/or Control) Call Hijacking Application Level Attacks Credit: Juniper Networks 29

IP Network Threats Ethernet, IP and DNS address spoofing ARP and DNS Cache Poisoning

IP Network Threats Ethernet, IP and DNS address spoofing ARP and DNS Cache Poisoning Quantity-based packet flooding Stack Do. S attacks VLAN “jumping” Qo. S / prioritization attacks 30

Organizing Vo. IP Threats Standard IP Network Threats (to the CIA triad) C Confidentiality

Organizing Vo. IP Threats Standard IP Network Threats (to the CIA triad) C Confidentiality I Integrity A Availability 31

Organizing Vo. IP Threats Advanced IP Network Issues/challenges (triple A) A Authentication A Authorization

Organizing Vo. IP Threats Advanced IP Network Issues/challenges (triple A) A Authentication A Authorization A Accounting 32

Application-Specific Vo. IP Threats “Phone” spoofing - registering a SIP client with someone else's

Application-Specific Vo. IP Threats “Phone” spoofing - registering a SIP client with someone else's identifier (no auth. ). § a successful attack would cause the similarly registered phone to ring when someone called the legitimate owner of the number. Credit: Jeremy George, Yale University 33

Threat to Confidentiality Programs exist to listen to SIP and other Vo. IP streams

Threat to Confidentiality Programs exist to listen to SIP and other Vo. IP streams (and record them). It is possible to capture packets on switched networks (by overflowing ARP tables, poisoning ARP caches, etc. ). Encryption should be used but has sideeffects: : on latency, on sound qulaity (packetization and compression chunking can lead to clipped staccato speech). 34

Application-Specific Vo. IP Threats Caller-ID / ANI “Spoofing” (faking source #) § Trivial to

Application-Specific Vo. IP Threats Caller-ID / ANI “Spoofing” (faking source #) § Trivial to do -- don’t trust Caller-ID -- OK to screen w/ Credit: Jeremy George, Yale University 35

Threat to Integrity It is possible to ‘hijack’ sessions. It is possible to modify

Threat to Integrity It is possible to ‘hijack’ sessions. It is possible to modify voice over IP streams. Once again, use encryption (or at least cryptographic integrity checks) to prevent this. 36

Application-Specific Vo. IP Threats Mit. M “spoofing” § CALEA is a ‘legit’ application of

Application-Specific Vo. IP Threats Mit. M “spoofing” § CALEA is a ‘legit’ application of this. § Do. S attacks are known immediately by communicating parties § Call content is neither overheard nor compromised. § Some proxies have logic in them that identifies a likely Do. S attack and discard those packets (ask your vendor!). § Encryption is the best proection against Mit. M spoofing. Credit: Jeremy George, Yale University 37

Threats to Availability Quality of Service (Qo. S) problems: § Latency - time for

Threats to Availability Quality of Service (Qo. S) problems: § Latency - time for traffic to go from source to destination (one-way and round-trip). 150 ms is Max RTT for PSTN. Vo. IP at 400 ms is at outer limit of tolerable range. § Jitter - variability in latency and out-of-order packet arrival times. Buffering can help here. § Packet Loss - results in gaps in communication. 38

Application-Specific Vo. IP Threats “Special Do. S (Denial of Service) attacks § § high

Application-Specific Vo. IP Threats “Special Do. S (Denial of Service) attacks § § high volume flood of SIP INVITEs high volume flood of SIP REGISTER commands Control Packet / Call Data Floods Packet Replay / Injection / Modification Credit: Jeremy George, Yale University 39

Application-Specific Vo. IP Threats “BID attacks on SIPS” § Get SIPS devices to downgrade

Application-Specific Vo. IP Threats “BID attacks on SIPS” § Get SIPS devices to downgrade to ordinary SIP Credit: Jeremy George, Yale University 40

Application-Specific Vo. IP Threats Rogue SIP Proxies § Impersonate a proxy to a User-Agent

Application-Specific Vo. IP Threats Rogue SIP Proxies § Impersonate a proxy to a User-Agent Credit: Practical Vo. IP Security, Syngress 41

Vo. IP Security Checklist Practical Vo. IP Security “high level short list”: Create, publish

Vo. IP Security Checklist Practical Vo. IP Security “high level short list”: Create, publish and enforce security policies. Practice rigorous physical security. Verify user identities. Actively monitor logs, firewalls & IDSes. Logically segregate data & voice traffic. Harden Oses. Encrypt whenever and whatever you can. 42

Vo. IP Security Checklist Juniper Best Pracices Security Measures 1. 2. 3. 4. 5.

Vo. IP Security Checklist Juniper Best Pracices Security Measures 1. 2. 3. 4. 5. 6. 7. Maintain Current Patch Levels Install a Good Anti-Virus System and Update it Regularly Apply State-of-the-Art Intrusion Detection and Prevention Systems Install Application-Layer Gateways between Trusted and Untrusted Zones. Enforce SIP security by means of Authentication, Authorization and IPSec Establish Policy-Based Security Zones to Isolate Vo. IP Segments. Run Vo. IP Traffic on VPNs to Minimize Eavesdropping Risk on Critical Segments. 8. Use VLANs to Prioritize and Protect Voice Traffic from Data Network Attacks 9. Apply Encryption Selectively 10 Protect Against UDP Flooding 11. Develop a Holistic Security Program 43

Meta Group Checklist IP Telephony-Specific Security Features The Call Control Server 1. Harden/Strip down

Meta Group Checklist IP Telephony-Specific Security Features The Call Control Server 1. Harden/Strip down OS. 2. Use secure OS. 3. Authenticate & authorize all user & device access to servers. 4. Require strong authentication for all configuration and software upgrades. 5. Should support app level signaling message auth. 6. Should support call setup info encryption. 44

Meta Group Checklist IP Telephony-Specific Security Features The Voice Gateway: 1. Require strong authentication

Meta Group Checklist IP Telephony-Specific Security Features The Voice Gateway: 1. Require strong authentication for all configuration and software upgrades. 2. Provide Do. S protection on IP inteface. 3. Should be configured to route calls only via the call control server. 4. Secure OS w/anti-virus AND host-based IDS. 5. Should support call setup info and media (voice content) encryption. 6. Should support a media (voice content) protocol authentication on a per-packet basis. 45

Meta Group Checklist IP Telephony-Specific Security Features The IP Phone: 1. Must authenticate itself

Meta Group Checklist IP Telephony-Specific Security Features The IP Phone: 1. Must authenticate itself to the call control server or a proxy server upon initial registration 2. Must support strong authentication for any remote configuration and software upgrades. 3. Should support a configurable access control list to control any incoming traffic (e. g. H. 323/SIP, RTP, HTTP, FTP, DHCP). . 4. When supporting an additional Ethernet port for PC connectivity, should have this implemented via a switching function combined with VLAN functionality. 5. Should support encryption of both call setup info and media as needed. Using encryption can additional end-to-end delay on each media packet. 46

Vo. IP Security Checklist Detailed and Specfic list: Use a separate VLAN with 802.

Vo. IP Security Checklist Detailed and Specfic list: Use a separate VLAN with 802. 1 p/q Qo. S w/priority VLAN tagging for the Vo. IP network. Use a private (RFC 1918) IP network for the Vo. IP LAN. Use NAT and/or proxies to hide internal addresses. Use a firewall (packet filtering or ALG) to protect & connect the Vo. IP network to the data IP network. Use an IDS or IPS to examine the traffic allowed through the firewall (may be built into the firewall). Use TLS to protect SIP and SRTP to protect RTP. Use NAC, 802. 1 X & RADIUS auth & SIP-aware FW. 47

Listservs & Newsgroups EDUCAUSE Security Discussion Listserv http: //www. educause. edu/Security. Discussion. Group/979 VOIPSA

Listservs & Newsgroups EDUCAUSE Security Discussion Listserv http: //www. educause. edu/Security. Discussion. Group/979 VOIPSA Best Practices Working Group http: //www. voipsa. org/Activities/bestpractices. php VOIPSA Best Practices WG List http: //voipsa. org/mailman/listinfo/bestpractices_voipsa. org NIST Publication Mailing list http: //csrc. nist. gov/compubs-mail. html 48

Vo. IP Security Effective Practices in Higher Ed One anonymous school: Uses separate VLAN,

Vo. IP Security Effective Practices in Higher Ed One anonymous school: Uses separate VLAN, L 2 switches and RFC 1918 IP addresses for Vo. IP network. Provides separate connections (and bandwidth) to each building with Vo. IP. Softphones can participate from regular campus LAN (aren’t required to use a 2 nd NIC on the Vo. IP network). 49

Vo. IP Security Effective Practices in Higher Ed A 2 nd anonymous school: Has

Vo. IP Security Effective Practices in Higher Ed A 2 nd anonymous school: Has enterprise polycom gateways (a bunch of them) that have priority in QOS on the routers. . Allows traffic via ports inbound on the above routers for this ‘legit’ traffic. Doesn’t restrict H. 323. Blocks SIP and Vonage because they don’t open the inbound ports. Packet 8 and other SIP applications which use STUN work fine (because of tunneling). Skype is a problem (paritcularly Supernodes at times). 50

Survey http: //www. surveymonkey. com/s. asp? u=8229 93567486 51

Survey http: //www. surveymonkey. com/s. asp? u=8229 93567486 51

Vo. IP Higher Ed Security Survey 52

Vo. IP Higher Ed Security Survey 52

Vo. IP Higher Ed Security Survey 53

Vo. IP Higher Ed Security Survey 53

Vo. IP Higher Ed Security Survey 54

Vo. IP Higher Ed Security Survey 54

Vo. IP Higher Ed Security Survey 55

Vo. IP Higher Ed Security Survey 55

Vo. IP Higher Ed Security Survey 56

Vo. IP Higher Ed Security Survey 56

Vo. IP Higher Ed Security Survey 57

Vo. IP Higher Ed Security Survey 57

Vo. IP Higher Ed Security Survey Which Vo. IP Security mechanisms do[n’t] you use?

Vo. IP Higher Ed Security Survey Which Vo. IP Security mechanisms do[n’t] you use? Use H. 235 for H. 323 security profiles (for H. 225, H. 245 and RTP traffic). Use SIPS (Secure SIP - RFC 3261 - SIP over TLS). Don't allow SRTP with null cipher (e. g. don't allow use of SRTP for just authentication). Use z. RTP for key management. Use Mikey for key mgt/exchange. Use SDES for key exchange. Use SRTCP for authentication. Use SRTCP for encryption. IPSEC to secure MGC (Media Gateways/Controllers) communication. Use of separate physical LAN(s) for Vo. IP for segregation from data IP network.

Vo. IP Higher Ed Security Survey Which Vo. IP Security mechanisms do[n’t] you use?

Vo. IP Higher Ed Security Survey Which Vo. IP Security mechanisms do[n’t] you use? Use of IPS between Vo. IP network and data IP network. Use of IDS between Vo. IP network and data IP network. Use NAC (network access control) such as 802. 1 X and RADIUS to authenticate hard phones. Softphones require the use of the separate Vo. IP network (physical LAN, VLAN, subnet address, etc. ) from the data IP network. Softphones are allowed with IPSEC transport mode. Softphones are allowed with IPSEC VPNs. Use NAC (network access control) such as 802. 1 X and RADIUS to authenticate hard phones. Allow NAT traversal via STUN or TURN Internet proxies. Provide separate dedicated bandwidth for Vo. IP traffic to the Internet. 59

Survey http: //www. surveymonkey. com/s. asp? u=8229 93567486 60

Survey http: //www. surveymonkey. com/s. asp? u=8229 93567486 60

Wrap-Up Question & Answer Session Evaluation & Feedback 61

Wrap-Up Question & Answer Session Evaluation & Feedback 61

Contact Info H. Morrow Long morrow. long@yale. edu Security. yale. edu 62

Contact Info H. Morrow Long morrow. long@yale. edu Security. yale. edu 62

Credits: Cisco - Configuring SIP High Availability Applications, http: //www. cisco. com/univercd/cc/td/doc/product/software/ios 123/123 cgcr/vvfax_c/callc_c/sipha_c/hachap

Credits: Cisco - Configuring SIP High Availability Applications, http: //www. cisco. com/univercd/cc/td/doc/product/software/ios 123/123 cgcr/vvfax_c/callc_c/sipha_c/hachap 2. htm Jeremy George, Yale University, “ “SIP. edu Cookbook - Security Considerations” http: //mit. edu/sip. edu/security. shtml Deb Shinder, 2006/12/1 “Make a SIP-based Vo. IP network more secure”, http: //articles. techrepublic. com/5100 -1035_11 -6145231. html? part=rss&tag=feed&subj=tr Deb Shinder, 2007/1/7 “Take a multi-layered approach to Vo. IP security”, http: //articles. techrepublic. com/5100 -1035_11 -6145231. html? part=rss&tag=feed&subj=tr Jose J. Valdes, Jr. , Colorado State University “Voice over Internet Protocol (Vo. IP) Security”, Net@Edu Conference, ICS – Wireless Group Meeting, Tempe, Arizona, February 6, 2005 63

Credits: Practical Vo. IP Security by Larry Chaffin, Jan Kanclirz, Jr. , Thomas Porter,

Credits: Practical Vo. IP Security by Larry Chaffin, Jan Kanclirz, Jr. , Thomas Porter, Choon Shim, Andy Zmolek, Syngress, March 2006 Wikipedia (pages on H. 323, SIP, SRTP, ZRTP), Zfone, etc. ) 64

This has been a chalk outline™ production.

This has been a chalk outline™ production.