ProcessorControlled Test Updates and DFT Requirements Jeffrey A

Processor-Controlled Test Updates and DFT Requirements Jeffrey A Moore EMC Corp. © Copyright 2008 EMC Corporation. All rights reserved. 1

A Typical Storage Processor (SP) Core Storage Processor Core DO AS YOU’RE TOLD LAN 0 LAN 1 PCIe DIMMs IOH ICH X D P COMM 1 SIO COMM 2 CPU PCT Pod FLASH BIOS NVRAM POST PCT Software controls CPU through PCT POD & Adapter CPU registers in turn initialize and operate busses at normal speeds Tests progress outward from CPU one chip at a time until all chips are tested No need for DRAM, or BIOS, or Flash memory to be working or loaded with data © Copyright Libraries of chip models make init and operations easy to program 2010 EMC Corporation. All rights reserved. 2

Test Stage Replacement Replace Custom Functional Test (CFT) w/ Basic Functional Test (BFT) AOI ICT CFT CIT FC (FRU Point) AXI AOI ESS ICT BFT ESS (FRU Point) AXI • Test Suite to Contain: CMD Power Tests Boundary Scan (BScan) PCT Functional Test Code Load, Boot-up Custom Diag, POST, and Resume Write © Copyright 2010 EMC Corporation. All rights reserved. 3

Basic Functional Test (BFT) Improvements BFT now combines 6 test steps into one – – – – All steps using same test station, so no added tester costs or test stage CMD Power test Checks voltages and margin status BSCAN test checks shorts/opens and some memory tests PCT test checks functionality of entire board PCT / Embedded SAS Tests guarantee operation of 25 drive slots Code load programs the CMD, MCU, BIOS, POST ROMs Boot-up test allows board to boot and run POST to verify readiness for next stage CMD BSCAN © Copyright 2010 EMC Corporation. All rights reserved. PCT SAS LOOP CODE LOAD BOOT-UP 4

Advantages of Combining all tests CMD / BScan / PCT / Code Load merger into One Tester – – – Improved coverage significantly Provides better diagnosis software at the CM for functional test CMD provides quick debug of power problems PCT reduces debug repair costs & eases lack of ability to use scope probes PCT Diagnoses “No-Boot” problems to boost CFT, ESS, or CIT yield SAS controller and SAS/Drive loop back bus tests provide quicker debug and better coverage © Copyright 2010 EMC Corporation. All rights reserved. 5

BFT CMD Test Capabilities CMD Power Sequencer Hardware (Control & Monitoring Device) – All CPU & I/O boards have a CMD chip (DSP w/many A/D and PCM ports) – CMD sequences, monitors & adjusts all DC-DC power circuits – If a power-up fault occurs, sequencer stores values of supplies before shutting down – Communicates by TWI bus to test card’s USB-to-TWI pod CMD Sequencer Test Software Features – – – Check CMD status Change margins (normal/high/low) User configuration data (Set margin limits) Restore p-good limits to factory settings Display Power down log (Useful for boards that won’t power up) Reads the POL and Vin voltages and compare them against the PTR limit © Copyright 2010 EMC Corporation. All rights reserved. 6

Current SAS Testing at BFT SAS Testing Carried out at BFT – – – Check for access & training to the SAS Controller (read Device/Vendor ID) Individually test 4 SAS Controller PHYs that connect to the SAS Expander Test 4 SAS Controller PHYs connected to SAS Front Panel Expansion Port Test access to the SAS Controller EEPROM and Flash PHYs that connect the SAS Expander to the SAS Controller are tested (4 Direct and 4 via the SAS Cable. ) § Command sent to each Drive PHY to test device presence and speed § The SAS Address of the Attached Device is read and the Device is registered. § Buffer test is carried out on the Drive Buffer © Copyright 2010 EMC Corporation. All rights reserved. 7

Cost Savings Realized at BFT SAS Test Removed SAS Controller Flash Prom. – Saved $2 per board - $1 for Flash part & $1 for Pre-programming it. – This eliminated ability to run SAS tests in BFT – Developed a method to store Flash data in memory and then have SAS Controller download it from there over PCIe bus. – Now SAS test is running OK again. Embedded Test Techniques Save Tester Cost – – – Removed expensive 6 GB SAS drives (Qty. 25) Replaced drives with loop-back midplane Developed use of chip vendors’ internal HS serial loop back tests Protocol agnostic – vendor supports loopback test Requires some DFT in design and vendor supplied firmware Reduced tester cost by $7500 and saved headaches from having to debug drives through the fixture as they fail. – Able to show significant changes in the PMC DFE 0 and Gain values when a cap was removed © Copyright 2010 EMC Corporation. All rights reserved. 8

SP (Storage Processor) w/ Custom 25 -Drive Midplane © Copyright 2010 EMC Corporation. All rights reserved. 9

Typical BFT Hardware © Copyright 2010 EMC Corporation. All rights reserved. 10

EMC BFT Test Features Tests controlled and results displayed via custom “Block Diagram” GUI – GUI displays each chip tested and lights each one Green or Red as it is tested – Techs can use script language to flip bits, exercise buses, read back data, etc. – GUI written by Cork, Ireland Test Engineers using PCT vendor’s standard API Test Time – Total BFT test time of UUT with 2 I/O cards is 13 minutes/unit – Code Load + Boot + POST is about 47 minutes – ouch! Coverage – – Was 51% coverage on robotic fault insertion vs 35 -40% on CFT Now have 87% coverage for BFT Tests 90% diagnostic accuracy vs ~70% on CFT In actual run of boards, of 11 faults found by PCT, 10 were predicted correctly to a faulty device/device group Yields – Prototype BFT yield was 83. 5% (Boards had no ICT tests yet) – Subsequent CFT test had 92. 8% yield – PCT test was improved to catch more faults, so final CFT yield is now 97. 6% © Copyright 2010 EMC Corporation. All rights reserved. 11

DFT Needed for PCT Test XDP connector too expensive and weak Route signals through front panel or midplane connector TCK, TMS, TDI, TRST, TDO routed to Processor Main board 3. 3 V reset & 1. 1 V Reset from Processor 2 Handshake signals from Processor (PREQ & PRDY) Most are 1. 1 V levels so pay attention to signal quality Need to keep round-trip-time short from test card – cable – CPU EMC BFT runs at 10 MHz TCK PCT can run up to 20 MHz Need to use mux to switch CPU signals between XDP & External Conn Test card uses mux to switch BSCAN / PCT/ MCU signals to JTAG bus © Copyright 2010 EMC Corporation. All rights reserved. 12

Conclusions / Future Needs Conclusions – – – – PCT is now integral part of BFT test suite BFT is now integral part of our test process Achieved replacement of CFT stage on latest products PCT Improves Diagnosability of BFT tests Use of Embedded SAS tests reduced cost of tester Decreases Time To Market for new products Enables use of less experienced techs at CMs Enables development of more embedded tests and instruments Future Needs – Looking for ways to reduce Code Load time – Need to develop multiple test sites on one tester like CFT – Investigating use of many PCT pods over Ethernet to one tester © Copyright 2010 EMC Corporation. All rights reserved. 13