The Value of Operational Testing and Evaluation of

The Value of Operational Testing and Evaluation of Security Technologies David Speaks Manager, Technology Applications Center

Intended Audience Those involved in the development of new security technologies Those involved in the purchase, implementation, and installation of security technologies

Many Types of Tests • • • Laboratory Tests Developmental Tests Experimental Tests Scenario Tests Pilot Tests • • Demonstration Tests Validation Tests Acceptance Tests Site Acceptance Tests Operational Testing is somewhat universal in its meaning- The Real World

Operational Testing conducted to evaluate a system or component in its operational environment Operational Environment § People § Processes § Environmental factors (atmospheric conditions, etc. )

Examples in the Security Operational Environment People § Employees § Level of Training § Security Personnel § Level of Motivation § General Public § Special Needs/Circumstances Processes § Access Control § Screening of Persons/Vehicles § Perimeter Security § Command/Control/Communication Environmental Factors § Day/Night § Hot/Cold § Indoor/Outdoor § Lighting Intensity/Variation § Dry/Wet § Wind

Why Do We Test ? Reasons for Testing: § § Evaluating cause/effects Learning something that is unknown Comparing performance of different alternatives Many other good reasons… For this presentation- consider this one: § To Mitigate Risk

Testing to Mitigate Risk R&D - developmental risk § Will it work? § Is it safe? § Is it marketable? Customer - implementation risk § Will it work for MY application? § What is the security benefit vs. the costs (value)? § Will it be accepted by the end users?

Concept of Risk vs. Scope of Testing More Testing

How Does this Relate to Security Technologies? §Most security technologies are highly dependent on the operational environment § The current demand for security solutions is driving a major investment in all aspects of security technology development and implementation § The rush to implement solutions can lead to a lack of emphasis on operational testing The Result = Likelihood of $ Wasted

Without Operational Testing? Decisions based upon “laboratory” results and/or vendor-supplied information § Laboratory results not always a good predictor of operational performance § Vendor information is typically skewed towards “best case” scenarios Decisions based on the concept of “something is better than nothing” § Not always true if technology is ultimately unusable or provides a false sense of security

“After spending more than $4. 5 billion on screening devices to monitor the nation’s ports, borders, airports, mail and air, the federal government is moving to replace or alter much of the antiterrorism equipment, concluding that it is ineffective, unreliable or too expensive to operate. ” “And not all the devices were tested to see how well they worked in the environments where they would be used. ” New York Times May 7, 2005

“The government has had difficulty getting independent, reliable technical assessments about the plausibility, cost, and benefits of advanced technology before Congress and agencies commit to spending. ” “…the primary technical advisors to federal officials often have been the contractors themselves. ” The Washington Post July 20, 2007

If you want to make HEADLINES – SKIP THE OPERATIONAL TESTING !

When is Operational Testing Most Valuable? In R&D- Operational Testing should be introduced into the process as early as possible to maximize its impact For those Purchasing/Implementing Technologies - Operational Testing results should be considered before and/or during the procurement process

DOD Example DOD-Defense Acquisition Guidebook Recognize the Value of T&E is a key part of the system engineering process. It is the validation step in the feedback loop for system design. Use T&E to understand risk and help determine technical issue areas…Failures in test, when discovered and acted on early in development will result in a better product at less cost - advantages you would not experience if you did not conduct the T&E. Studies have revealed that roughly 75% of life cycle costs of a program are fixed as a result of the initial design process. Obviously, the longer you wait to discover deficiencies, the more it will cost to implement changes. Spending the time and money early in a program for a rigorous test program will save time and money later.

Operational Input to TRL Assessment

Operational Input to Spiral Development Boehm’s Spiral

How Much Testing Should Be Done? More Testing

How Much Testing Should Be Done? Line A- tends to be the perspective of the funding organization, management-types, etc. (accept more risk to speed up process) Line B- tends to be the perspective of the testing organization (achieve very low uncertainty and risk) Can be a major point of disagreement- how much testing do we really need? Scope that provides greatest VALUE usually lies somewhere in between

How Much Testing Should Be Done? Most Valuable Testing More Testing

Challenges of Operational Testing § Perception of being too expensive or taking too long § May “inconvenience” users and require normal operations to be temporarily altered- pushback from operations personnel § For developers- access to the operational environment may be difficult § Not easy to find experienced testing resourcesespecially independent ones

Recent/Current Issues with Security Technology Testing § Poorly coordinated tests across various agencies and other organizations § Lack of emphasis on operational parameters in the development process § Absence of standards for operational testing § No centralized place for compiling test results § Classification of results makes distribution difficult

Operational Testing- Points to Consider Developers/Sponsors: § Build operational testing into the project schedule and budget § Introduce the project technical resources to the operational environment early on § Identify strengths/weaknesses of the security solution and choose applications accordingly § Document and share test results as much as possible

Operational Tests- Points to Consider Implementers/Purchasers: § Insure that the test is designed with the appropriate parameters known and included § Involve technology supplier to insure that the system is setup and operators trained appropriately prior to testing, however, not to the point that they influence the test plan or results § Allocate dedicated resources to manage the test activities or contract for these services

Operational Testing- Points to Consider § If meaningful, provide a baseline test to provide a “before” condition that the test results can be compared against § Account for the inherent learning curve of users in the test planning § Design the scope of testing such that it is enough to obtain statistically relevant data but not so large as to be impractical §Insure that results are qualitative and not subjective

Conclusions Operational testing and evaluation has an important role in the development and implementation of security technologies Challenges and pitfalls in Operational Testing must be considered and dealt with to avoid wasting time/money More work needs to be done to make Operational Testing more standardized and the results more accessible

THANK YOU !! David Speaks Y-12 National Security Complex Oak Ridge, TN 865 -574 -9132 speaksdm@y 12. doe. org