CSRIC WG3 Indoor Location Accuracy Test Bed Report

CSRIC WG#3 Indoor Location Accuracy Test Bed Report March 6, 2013 Stephen J. Wisely & Richard Craig Co-Chairs

Work Group 3 Areas of Focus Outdoor Accuracy Testing Indoor Accuracy Testing Impact of Commercial LBS Technologies Outdoor Report Issued Work Effort Completed Indoor Report Issued Work Effort on Test Bed completed Feb. 2013 Work Effort Completed February 2013 2

Ø The WG addressed the FCC’s questions as best we could for now Ø We developed a testbed framework and have solicited interest from suppliers of current technology on their in interest in having their product go through the TB Ø We are currently working on the finer points of the testing process and logistics associated with TB program oversight Ø We anticipate conducting testing in two stages: Ø Stage I – Characterize performance of current technologies Ø Stage II – review of future technologies Ø We will provide a read out on Stage I efforts to the CSRIC committee at the September 2012 and March 2013 meetings Ø We believe the Testbed concept could survive beyond the current CSRIC charter as an independently supported entity for evaluating future technologies 3

Participants WG 3 Indoor Location Test Bed Subgroup consisted of the following members: First Name Last Name Organization Wayne Andrew Richard Khaled Thomas Jeanna Roger Ryan Marte Sandra Mike Ballantyne Beck Craig Dessouky Dombrowsky Green Hixson Jensen Kinder Lott Loushine Motorola Mobility, Inc. Comm. Scope Verizon Wireless Techno. Com Corporation Wiley Rein LLP Sprint NENA T-Mobile Time Warner Cable Century. Link Applied Communication Sciences Barry Martin Boeing Kathy Martin Gary Ganesh Gustavo Raghavendhra Chuck Mc. Mahon Moody Parsons Pattabiranan Pavon Rao Ronshagen APCO International Metro Emergency Services Board Next. Nav LLC True Position, Inc. AT&T Cassidian Communications First Name Brett De. Wayne Norman Susan Girish John Dorothy Bill Greg Bruce Stephen Richard Deh. Min Last Name Organization Schneider Sennett Shaw Sherwood Sivaram Snapp Spears-Dean Tortoriello Turetzky Wilson Wisely Bexar Metro 9 -1 -1 Network District ATIS Polaris Wireless, Inc. Verizon Wireless Tele. Communications Systems, Inc. (TSC) Intrado, Inc. Virginia Information Technologies Agency US Cellular CSR Technology Inc. Qualcomm Inc. APCO International Wu Nokia Siemens Networks 4

Indoor Test Bed –Stage 1 • Goal: Independent comparative testing that provides the FCC with meaningful, objective results on indoor wireless location performance • Constraints: Within time frame of CSRIC III; financially feasible framework • WG 3 chose San Francisco Bay Area due to the wide range of wireless environments (morphologies) present • Invitation extended to a host of location technology vendors to participate • 7 vendors started the process and 3 eventually participated • Two existing and one new location technologies tested: – Next. Nav (wireless beacon technology) – Polaris Wireless (RF pattern matching/fingerprinting) – Qualcomm (AGPS/AFLT hybrid) • Four major wireless carriers participated: – AT&T, Sprint, T-Mobile, Verizon • Competent, experienced independent test house selected (Techno. Com) 5

Indoor Test bed Methodology • Based on sound statistical test methodology – Articulated in ATIS-0500013; received wide consensus • Multiple buildings (19) of different sizes & types across range of environments – Dense urban, suburban, and rural morphologies • Multiple test points in each building – 2 to 6 test points depending on size and complexity of selected building – Representing typical use scenarios • Statistically significant, adequately large number of independent test calls at each stationary test point – 100 test calls per technology minimum target per test point – >13, 400 test calls per technology placed at 74 test points, well over 100 avg. – Aggregated over different devices and networks used by the location technology under test at each test point • Indoor ground truth accuracy < 1 m (professional survey accuracy 2 cm) • Performance attributes tested: accuracy, yield, TTFF, uncertainty, scatter 6

Test Environments and Buildings Dense urban Downtown Financial District San Francisco (6 bldgs) Suburban Santa Clara and Sunnyvale (6 bldgs) Urban Downtown San Francisco and San Jose (5 bldgs) Rural San Benito County (2 bldgs) 7

Summary Results-- Accuracy Urban environment had the most challenged horizontal accuracy followed closely by dense urban Next. Nav technology provided vertical results 8

Summary Results: Yield, TTFF, and Uncertainty • Yield varied with the severity of the environment but was generally high – >90% for all technologies in the configurations tested with one minor exception (86% AGPS/AFLT yield in dense urban) – More yield variation for the technology actually integrated into the wireless network and has to exchange messages per the standard E 911 implementation through severe RF fading • TTFF was stable and not an issue – Small variation about design values for non-network integrated implantations • 27 sec. for Next. Nav and 24 sec. for Polaris – More variation for AGPS/AFLT, which is actually integrated into the network • 28 sec. avg. with 33 sec. 90% in urban/dense urban vs. 24 sec avg. & 26 sec. 90% suburban/rural • Uncertainty is still a useful parameter and its reliability generally reflects how well a system is performing in a given environment 9

Location Scatter Relative to Building Sizes & Density • Traditional benchmarks from outdoor performance may not best suite the indoors, especially in denser cities • Even traditionally good performance may not meet the unmitigated expectations of some--education on indoor issues is important Test points in the 5 -story building partially shadowed by tall bldg Test point 3 (T 3) at center of 50/150 m circles 10

• • Lessons and Recommendations The cooperatively funded indoor test bed is a viable framework, which yields very valuable results – It’s a model that should be repeated with some adjustments, based on the lessons learned – Key to its success are the experience & strong commitment of its participants Ø Create and maintain the CSRIC-type framework that fosters the active and balanced participation by the various stakeholders There a number of difficult challenges that need to be met repeatedly – Project setup and contractual arrangements are difficult and time consuming – Building access is the most vexing of challenges facing indoor testing Ø Allow for the long lead time of several months required to properly tackle these issues Ø Create a test management resource with contractual authority and some funding capability Ø Develop a more structured approach to dealing with building access and, as needed, use creative incentive mechanisms for buildings to participate Deployed location technologies help with indoor performance and new technologies tested have promising performance, but there is a long way to go to meet expectations for reliable indoor use of wireless location for E 911 Ø Continually test new technologies, including next generations of existing systems, technologies used for LBS (e. g. , Wi. Fi) & new technologies targeted to the latest air interfaces The Bay Area has proven to be a good place for the indoor test bed Ø Leverage it fully in future test beds 11