November 2005 doc IEEE 802 11 051194 r

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Video Testing Methodology Authors: Name Company Address Phone email Philip Corriveau Intel HF 3 -96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696 -1837 philip. j. corriveau@intel. com Audrey Younkin Intel HF 3 -96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696 -3947 audrey. c. younkin@intel. com Chris Olsen Intel HF 3 -96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696 -7548 Chris. olsen@intel. com Fernald Royce Intel HF 3 -96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696 -4318 Royce. fernald@intel. com Rik Logan Intel 5200 NE Elam Young Pkwy Hillsboro, OR 87124 (503)-712 -1675 Rik. e. logan@intel. com Uriel Lemberger Intel PO Box 1659, Matam Industrial Park, Haifa 31015 Israel +972 -4 -865 -5701 uriel. lemberger@intel. com Neeraj Sharma Intel 13290 Evening Creek Drive, San Diego, CA 92128 (858)-385 -4112 neeraj. k. sharma@intel. com Sasha Tolpin Intel PO Box 1659, Matam Industrial Park, Haifa 31015 Israel +972 -4 -865 -5430 alexander. tolpin@intel. com Notice: This document has been prepared to assist IEEE 802. 11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802. 11. Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures <http: // ieee 802. org/guides/bylaws/sb-bylaws. pdf>, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard. " Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair <stuart. kerry@philips. com> as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802. 11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at <patcom@ieee. org>. Submission 1 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Agenda • • Video Assessment GED Process Overview Purpose Objective Outputs Experimental Design Methodology Overview Summary Submission 2 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Intel Video Testing Methodology Concepts and Methods • • • Philip Corriveau & Audrey Younkin User Centered Design Media and Acoustics Perception Lab Submission • • • 3 Olsen Christopher & Fernald Royce Platform Systems Technology Wireless Integration Team Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Video Assessment ~Not a One Step Process~ • The Gross Error Detector (GED) is one of the first steps in assessing video experience. – The GED presents a simple and quick way to measure large errors, such as dropped or repeated frames or video playback at an incorrect frame rate. • The GED is designed to communicate an Experiential Value that can be directly correlated to a real end-user. • Other tools like VQM, then capture and measure video quality (artifacts). Submission 4 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Why Use GED? • The Video Gross Error Detector is a high-level video performance analysis application • Determines whether a video stream arrives intact, in sequence and at the correct frame rate • Enables automated, quantitative, repeatable measurements of video quality • Significantly less expensive and more accurate than manual video quality assessment Submission 5 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 GED Process Overview 0. Source Material 1. Marked Source Material GED Encode WMV Encode 2. Compressed and Marked 3. Capture Results 4. GED Analysis System Under Test GED Decode Submission 6 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 GED Encode Submission 7 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Purpose • Currently the Gross Error Detector (GED) measures video performance based of a set of objective metrics. • However, this does not present the entire picture for video performance and assessment. For more accurate end user analysis, there needs to be a subjective mean opinion score (MOS) coupled with the objective metrics. Submission 8 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Current Objective Outputs • • Dropped Frames Repeated Frames Delayed Frames Non-sequential Frames Future Objective Outputs • Audio Error Detection (end of Q 4) • AV Sync (end of Q 4) • Frame Hashing - detection of blur and video corruption (Q 1 ’ 06) Submission 9 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Experimental Design • Currently the GED experimental design is broken into two Phases – Phase I: Expert Subjective Assessment – Phase II: Non-Expert Subjective Assessment Submission 10 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Phase I Expert Assessment • Using a SD (720 x 486) compiled collection of 10 second clips adding up to a total of 321 seconds (~5 minutes). • The GED tool will do an average over the 10 second clips with respect to metrics. • The GED is independent of the codec, streaming server and hardware platform. • Using Windows Media Services with 1 -6 Mbps WMVs and the East Fork server with 1 -8 Mbps MPEG 2 SD files. Submission 11 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 General Methodology • To create a Subjective Score (1 -5) there needs to be an evaluation of what real life situations produce in respect to a wireless connection. • The wireless path loss and other effects will cause different amounts of frames dropped/repeated that can be captured and compared to the original GED. Submission 12 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Example of Expected Scoring Value • However, the temporal factor also needs to be put in place, not only the number of dropped frames, but where they occur in the sequence and how long each error occurs for. • Each clip is 260 frames long. Submission Expert Score 13 Number Percentag Frames e Frames Dropped 5 >4 1. 5% 4 >10 3. 8% 3 >50 19% 2 >100 38% 1 >200 77% Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Phase II Non-Expert Subjective Assessment • Subjects: 40 -70 non-expert external participants will rate video quality on a scale of 1 -5 (half male/half female). Using Market Decisions Corporation for outsourcing of participants. Submission 14 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 General Methodology • Based on the Expert Analysis, five new sequences (one for each score) will be developed based on the GED outputs and temporal factors. • One or two participants per test session. • Upon arrival to the study location, participants will be asked to read through a brief overview of the study consisting of a summary of the test procedure requirements and will be asked to sign a confidentiality agreement and consent form. Submission 15 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 How to Rate • Prior to each session, participants will be instructed to rate each video sequence based on its real-time smoothness and flow. • Do not rate experience based on content quality but rather the flow of presentation. Submission 16 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Practice Trial • A familiarization and practice trial will carried out to begin the test session. • Here, participants will be shown the best and the worst quality for frame of reference. Submission 17 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Randomization • The presentation order of the video content will be randomized using a pseudo-random number generator tool to prevent ordering effects, i. e. to mask any tendency for a participant observer to rate a clip in relation to the previous one. • Randomization is a key element of psycho-visual testing to ensure that participants do not see the material in a repeated fashion that would allow a learning effect. Submission 18 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Voting Opinion • After each video sequence, participants will be presented with a pop-up window requiring their input of their opinion of the video experience on a subjective rating scale. This scale ranged from excellent (numerical equivalent of five) to bad (numerical equivalent of one). Submission 19 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Numerical Values Value Presented to Participant Actual Numerical Value Excellent 5 Good 4 Fair 3 Poor 2 Bad 1 Submission 20 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Participants will sit at a predetermined viewing distance 20° H 5 H Submission 21 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Post Test • At the end of the testing session, participants will be asked to fill out a post test questionnaire based on their experience. Submission 22 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Methodology Overview Submission 23 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Summary • Video Assessment not a one step process. • We presented one of the tools GED, which is quantitative, repeatable measurements of video quality. • Experiments concepts were described that help to correlate subjective test results with objective measurements. • Results from the experiment described within will be presented in the next meeting in January Submission 24 Philip J. Corriveau - Intel

November 2005 doc. : IEEE 802. 11 -05/1194 r 0 Next Steps • Intel will correlate GED and Subjective data to Link (Phy and Mac) Layer statistics for 802. 11 • Develop a model for 802. 11 for Video Quality • Deliver results at January meeting Submission 25 Philip J. Corriveau - Intel