New Methods for Approaching OTE MOEs and MOPs

Ghana, 16 March 2001 March 4, 2004 Syzygy Technologies, Inc.

Who We Are n Syzygy Technologies, Inc. n n n Full service IT company:

Why Automated Testing? n Key Tenets for Successful Testing: Tests must be: n n

Why Automated Testing? (cont’d) n The human factor = the weak link n n

Why Automated Testing? (cont’d) n Determining if the “system is getting better” is subjective

The Correlation Problem n n Correlators are basic to an effective C 4 I

Defining the Process n n Correlation Testing Process was outlined by SPAWAR Systems Center,

Defines the Metrics n n n n n Consistent Correlation Correct Correlation Total Fragmentation

Executing the Process was Difficult ZNR UUUUU R 010001 Z APR 97 FM TABULAR

JFATE is Born n n The Joint Fleet Automated Track Evaluator automates the process

JFATE March 4, 2004 Syzygy Technologies, Inc.

MOE/MOP: Makes it Easy to Compare Software Performance March 4, 2004 Syzygy Technologies, Inc.

JFATE Advantages n n Running a test in minutes rather than days “Expertise” is

Test Tools in the Dev Cycle Development Project Acceptance Test Reqt/Design Refinement March 4,

Test Tools in the Dev Cycle Development Reqt/Design Test Tools Test Project Acceptance Test

Points to Consider n n n Evaluation is programmatic: we will only catch errors

For More Information n For Testing Services Using JFate: n n Syzygy Test Tool

Slides: 20

Download presentation

New Methods for Approaching OT&E MOEs and MOPs Automated Generation of Measures of Effectiveness/Performance for Correlation in C 4 I Systems Syzygy Technologies, Inc. Albert R. Crespo, CTO Joe Discar, CHAFO

Ghana, 16 March 2001 March 4, 2004 Syzygy Technologies, Inc.

Who We Are n Syzygy Technologies, Inc. n n n Full service IT company: T&E, Software Development, Networking, Security, Integration One of the premier C 4 I T&E companies for the United States Navy: GCCS, Gale, TBMCS Syzygy’s S/W Developers created Automated Test Tools (ATT) for use by our testers: n n n March 4, 2004 DII COE Compliancy Message Capture, Generation and Playback MOE & MOP for Correlation Syzygy Technologies, Inc.

Why Automated Testing? n Key Tenets for Successful Testing: Tests must be: n n n Definable Measurable Consistent Objective If tests lack in any of these qualities, they may not be repeatable and therefore of much less value March 4, 2004 Syzygy Technologies, Inc.

Why Automated Testing? (cont’d) n The human factor = the weak link n n Data collection = tedium & inconsistency n n ATT minimizes human error ATT automatically collects results Subject Matter Expertise is Rare = inconsistent long-term test conduct n ATT preserves domain expertise in code March 4, 2004 Syzygy Technologies, Inc.

Why Automated Testing? (cont’d) n Determining if the “system is getting better” is subjective = upset users n n ATT can generate quantitative metrics Development slips to the right; deliverable deadlines rarely slip = little time to test n ATT can save time March 4, 2004 Syzygy Technologies, Inc.

The Correlation Problem n n Correlators are basic to an effective C 4 I system What is the effectiveness of correlation algorithms and systems? Manual data collection and analysis typifies the flaws of the human factor A faulty correlator could mean faulty command decisions March 4, 2004 Syzygy Technologies, Inc.

Defining the Process n n Correlation Testing Process was outlined by SPAWAR Systems Center, San Diego Published as Technical Report 1475, Naval Tactical Command Systems Afloat (NTCSA) Correlator/Tracker Evaluation, 1991 n n Defines a number of MOE/MOPs to be used for evaluating correlation effectiveness This paper provided a solid basis for conducting correlation evaluation March 4, 2004 Syzygy Technologies, Inc.

Defines the Metrics n n n n n Consistent Correlation Correct Correlation Total Fragmentation Track Fragmentation Miscorrelation Ambiguities Constructed Track Ratio Track Purity Track Continuity Track Fidelity March 4, 2004 Syzygy Technologies, Inc.

Executing the Process was Difficult ZNR UUUUU R 010001 Z APR 97 FM TABULAR R 010001 Z APR 97 INPUT FM TABULAR OUTPUT TO GOLD BT BT UNCLASSIFIED MSGID/TABULAR/GOLD/0014/APR CTC/T 7777/UNEQUATED-UNKNOWN//////AL//HIT/00////// CTC/X 7777/UNEQUATED-UNKNOWN//////AL//HIT/00////// POS/190101 Z 1/APR/4000 N 4/08000 W 8//72. 0 T/7. 3 NM/4. 1 NM//////JM POS/190101 Z 1/APR/4000 N 4/08000 W 8//72. 0 T/7. 2 NM/4. 1 NM//////JM RADB/190101 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z POS/190102 Z 1/APR/4000 N 4/08000 W 8//72. 0 T/7. 3 NM/4. 1 NM//////JM POS/190102 Z 1/APR/4040 N 4/08000 W 8//72. 0 T/7. 2 NM/4. 1 NM//////JM RADB/190102 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z POS/190103 Z 1/APR/4000 N 4/08000 W 8//72. 0 T/7. 3 NM/4. 1 NM//////JM POS/190103 Z 1/APR/4040 N 4/08000 W 8//72. 0 T/7. 2 NM/4. 1 NM//////JM RADB/190103 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z POS/190104 Z 1/APR/4000 N 4/08000 W 8//72. 0 T/7. 3 NM/4. 1 NM//////JM POS/190105 Z 1/APR/4040 N 4/08000 W 8//72. 0 T/7. 2 NM/4. 1 NM//////JM RADB/190104 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z RADB/190105 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z POS/190105 Z 1/APR/4000 N 4/08000 W 8//72. 0 T/7. 3 NM/4. 1 NM//////JM POS/190106 Z 1/APR/4040 N 4/08000 W 8//72. 0 T/7. 2 NM/4. 1 NM//////JM RADB/190105 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z RADB/190106 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z POS/190106 Z 1/APR/4000 N 4/08000 W 8//72. 0 T/7. 3 NM/4. 1 NM//////JM POS/190107 Z 1/APR/4040 N 4/08000 W 8//72. 0 T/7. 2 NM/4. 1 NM//////JM RADB/190106 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z RADB/190107 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z POS/190107 Z 1/APR/4000 N 4/08000 W 8//72. 0 T/7. 3 NM/4. 1 NM//////JM ENDAT/OADR RADB/190107 Z 1/APR/B 999 A//4000. 000 MHZ/1000. 4060//3. 310/6. 300/Z BT ENDAT/OADR BT March 4, 2004 Syzygy Technologies, Inc.

JFATE is Born n n The Joint Fleet Automated Track Evaluator automates the process of evaluating correlation results Provides a friendly graphical interface to conduct the test and view the results Runs on multiple platforms: HP, Sun, Windows, Linux, etc. Provides consistent execution of TR 1475 correlation testing making it quick and easy March 4, 2004 Syzygy Technologies, Inc.

JFATE March 4, 2004 Syzygy Technologies, Inc.

MOE/MOP: Makes it Easy to Compare Software Performance March 4, 2004 Syzygy Technologies, Inc.

JFATE Advantages n n Running a test in minutes rather than days “Expertise” is built-in/preserved Results are objective Metrics can help the decision making process March 4, 2004 Syzygy Technologies, Inc.

Test Tools in the Dev Cycle Development Project Acceptance Test Reqt/Design Refinement March 4, 2004 Syzygy Technologies, Inc. Operational Test

Test Tools in the Dev Cycle Development Reqt/Design Test Tools Test Project Acceptance Test Refinement March 4, 2004 Syzygy Technologies, Inc. Operational Test

Points to Consider n n n Evaluation is programmatic: we will only catch errors that the tool is designed to catch. Requirements vary: the tools need to be tweaked accordingly Developing a tool is expensive: the ability to build on existing tools saves on cost – and the cost is mitigated by the savings in time/resources during the test phases of a project March 4, 2004 Syzygy Technologies, Inc.

For More Information n For Testing Services Using JFate: n n Syzygy Test Tool Development: n n Al Crespo, crespo@syzygy-tech. com, CTO Syzygy Business Development: n n Bob Higginson, SSC-SD 619 -553 -2569 Merv Cutler, mcutler@syzygy-tech. com, Marketing Other Information: n Joe Discar, discar@jfate. test. xhome. us March 4, 2004 Syzygy Technologies, Inc.