DOE in TEMPs, T&E Concepts,Test Plans and BLRIPs

DOE in TEMPs, T&E Concepts,Test Plans and BLRIPs

DOE in TEMPs, T&E Concepts, Test Plans and BLRIPs Lessons Learned from Case Studies Purpose Discuss lessons learned from past tests Illustrate how DOE thinking can be applied to TEMPs, Test Plans, and other documents Outline Overview Elements of DOE Process Examples Quantitative, Mission-Oriented Metrics Coverage of Operational Envelope Confidence and Power of Test Summary Overview Based upon DOT&E initiative: Whenever possible, our evaluation of performance must include a rigorous assessment of the confidence level of the test, the power of the test and some measure of how well the test spans the operational envelope of the system. IDA conducted analysis of select BLRIPs from last two years Noted a structured approach to testing that capture many aspects of these concepts The analysis also identified areas of potential improvement

Modify TEMPs, T&E concept papers, Test Plans, and BLRIPs to incorporate DOE concepts Elements of DOE Process Have quantitative, mission-oriented metrics: What is the question(s) we are trying to answer? e.g., Can a unit equipped with the Mobile Gun System (MGS) successfully accomplish its missions? What are the applicable metrics? Describe how well the operational envelope is covered: Identify factors that drive performance e.g., threat, terrain, environment, mission Identify levels for each factor Show how well the test covers the operational envelope For both individual test periods and the overall test program Calculate the confidence level and power of the test: Test plan: Significance, Power, Effect Size, sample size Test reports: XX% confidence intervals Confidence performance above threshold

Consider whether standard DOE techniques are applicable There is no one size fits all solution Quantitative, Mission-Oriented Metrics Mission-Oriented Metrics Case studies identified several areas for potential improvement Ensure metrics and KPPs are measureable and testable As defined, many are not, e.g., The Mobile Gun System (MGS) primary armament must defeat a standard infantry bunker and create an opening in a double reinforced concrete wall, through which infantry can pass. Frequently mission-oriented metrics do not have thresholds Consider whether they should have a threshold Is the standard at least as good as (or better than) the legacy system? Do you have quantitative data on the legacy system? Look at metrics during JCIDS process Surveys Surveys frequently have been qualitative and poorly designed There is a science behind survey design; use it Be quantitative (e.g., Likert scale) During analysis, watch for discrepancies between numerical

scores and written comments Be careful with surveys Coverage of Operational Envelope Coverage of the Operational Envelope 1st Step: Identify factors & levels of interest In case studies, factors & levels of interest were sometimes specified, other times they were added in retrospective study. 2nd Step: Determine breadth of coverage of operational envelope Tools illustrated in following examples: cross-tabular matrices, continuous plots, other graphical representations These are examples, do not restrict yourself to them Power analysis can help determine if test design is sufficient Next section of brief Mobile Gun System (MGS) Coverage of Operational Envelope 4 Factors: Mission Type, Terrain Type, Threat Level & Illumination Mission Attack Defend Stability and Support Illum OPFOR Terrain Urban Mixed Forest Desert Urban Mixed Forest Desert Urban Mixed Forest Desert Day Low 1 1 Day Med

1 1 Day High 1 1 3 Night Low 2 Night Med 2 Night High 2 1 5 3 3 3 2 Weather: as it occurred; not controlled Key - Instrumented data collected during controlled IOT at Ft. Hood; number of mission replications indicated in cell - Limited use data collected during Mission Rehearsal Exercise at Ft. Lewis; no instrumentation or control over factors - Limited use (anecdotal) data collected in theater during unit deployment to OIF, mostly on tactics and employment techniques 2 2 5 2 2

3 16 IOT test design builds on evidence from previous events Mission Rehearsal Exercise prior to unit deployment (basis for Section 231 report) Field data from unit deployment IOT scoped to focus on voids in medium and high threat levels Early deployment changed original DOE plan Mobile Gun System (MGS) Lesson Learned: DOE identified gaps in coverage, partially filled from other sources Coverage of Operational Envelope 4 Factors: Mission Type, Terrain Type, Threat Level & Illumination Mission Attack Defend Stability and Support Illum OPFOR Terrain Urban Mixed Forest Desert Urban Mixed Forest Desert Urban Mixed Forest Desert Day Low 1 1 Day Med 1 1 Day High

1 1 3 Night Low 2 Night Med 2 Night High 2 1 5 3 3 3 2 Weather: as it occurred; not controlled Key - Instrumented data collected during controlled IOT at Ft. Hood; number of mission replications indicated in cell - Limited use data collected during Mission Rehearsal Exercise at Ft. Lewis; no instrumentation or control over factors - Limited use (anecdotal) data collected in theater during unit deployment to OIF, mostly on tactics and employment techniques 2 2 5 2 2 3 16

IOT test design builds on evidence from previous events Mission Rehearsal Exercise prior to unit deployment (basis for Section 231 report) Field data from unit deployment IOT scoped to focus on voids in medium and high threat levels Early deployment changed original DOE plan USS Virginia Anti-Submarine Warfare (ASW) Search What is the operational envelope? (factors and levels) Environmental Factors Shipping Levels and Sea State (ambient noise) Sound Velocity Profiles (several types each with different sound propagation characteristics) Target types and operating modes SSN (signature, sonar capability/proficiency) SSK (signature, operating modes, sonar capability/proficiency) Test submarine configurations (two towed arrays and wide aperture array) Scenarios (area search, barrier search, cued intercept, multiple targets) Cross tabular matrix from previous example might not illustrate breadth of coverage appropriately! USS Virginia ASW Coverage of Operational Envelope Poor Acoustic Propagation;

High Ambient Noise, High Density Traffic Difficulty of Environment Virginia vs. Georgia (ASW-3) ARCI APB-03 OT 688I vs. Gotland ARCI APB-06 OT 688I vs. Todaro Favorable Acoustic Propagation; Low Traffic; Low Ambient Noise

Plot simplifies environmental and target type factors into ordinal comparisons Only tested Virginia with TB29 towed array (inadequacy noted in BLRIP) Area search considered most difficult, other scenarios not examined in IOT&E Stimulated sensors to simulate multiple target scenario No SSK testing with Virginia conducted Virginia vs. Albany (ASW-2) Snorkeling Diesel (includes most older SSK threats) Fast/Noisy SSN Slow SSN or SSBN Gotland Todaro (SSN threat equivalent) (SSK threat (Quietest SSK

equivalent) threats) Target Source Level (Decreasing) ARCI data used to provide assessment Two Virginia tests do not cover entire environmental space USS Virginia ASW Coverage of Operational Envelope Difficulty of Environment Poor Acoustic Propagation; High Ambient Noise, High Density Traffic Unknown Performance Virginia vs. Georgia

(ASW-3) Historical Data Sufficient to assess performance 688I vs. Gotland Difficult to determine response curve from two SSN tests Favorable Acoustic Propagation; Low Traffic; Low Ambient Noise ARCI APB-03 OT ARCI APB-06 OT 688I vs. Todaro Unknown

Performance Virginia vs. Albany (ASW-2) Snorkeling Diesel (includes most older SSK threats) Fast/Noisy SSN Slow SSN or SSBN Gotland Todaro (SSN threat equivalent) (SSK threat (Quietest SSK equivalent) threats) Lesson Learned: Target Source Level (Decreasing) DOE helped identify gaps Plot simplifies environmental and target type factors into ordinal comparisons

Only tested Virginia with TB29 towed array (inadequacy noted in BLRIP) Area search considered most difficult, other scenarios not examined in IOT&E Stimulated sensors to simulate multiple target scenario No SSK testing with Virginia conducted ARCI data used to provide assessment Two Virginia tests do not cover entire environmental space USS Virginia Strike Coverage of Operational Envelope Strike mission broken into phases with multiple factors/levels PMissile Placement = PEP PA PTGT PL PM Engagement Planning Alignment Targeting

Launch Systems Missile reliability Missile Type Launcher Missile Type Launcher Launcher 63 Block III 42 Block IV 11 Horizontal 29 Vertical 12 Block III 9 Block IV 28 Horizontal 42 Vertical 1 Horizontal 2 Vertical Mission Receipt

Missile Type Missile Type Missile Type 113 ESP (EHF and UHF IP) 0 INDIGO 25 Block III 15 Block IV 44 Block III 26 Block IV 2 Block III 1 Block IV Strike over Secret 17 SoS 96 non-SoS USS Virginia Strike Coverage of Operational Envelope Strike mission broken into phases with multiple factors/levels PMissile Placement = PEP PA PTGT PL PM Engagement Planning Alignment

Targeting Launch Systems Missile reliability Missile Type Launcher Missile Type Launcher Launcher 63 Block III 42 Block IV 11 Horizontal 29 Vertical 12 Block III 9 Block IV 28 Horizontal 42 Vertical 1 Horizontal 2 Vertical

Mission Receipt Missile Type Missile Type Missile Type 113 ESP (EHF and UHF IP) 0 INDIGO 25 Block III 15 Block IV 44 Block III 26 Block IV 2 Block III 1 Block IV Strike over Secret 17 SoS 96 non-SoS Limited missile firings will be discussed later Joint Chemical Agent Detector (JCAD) What is the operational envelope? (factors and levels) Agent (9 agents and 2 simulants)

Temperature, water vapor concentration, agent concentration, interferent (continuous) Environment (sand, sun, wind, rain, snow, fog) Service (Army, Air Force, Navy, Marine Corps) JCAD Mode (Monitor, Survey, TIC) Operator (Any MOS to CBRN Specialist) TTP (Monitor Mission, Survey Mission, Decon Support) Joint Chemical Agent Detector (JCAD) Coverage of Operational Joint Chemical Agent Detector (JCAD) Coverage of Operational DOE applied to full test program for breadth of coverage Response Surface Design applied to chamber tests Joint Chemical Agent Detector (JCAD) DT Chamber Test Response Surface Design Joint Chemical Agent Detector (JCAD) DT Chamber Test Response Surface Design DOE helps determine whether gaps are significant to the overall assessment Confidence and Power of Test

Confidence and Power of Test Test Planning vs. Test Reporting Test Planning What confidence level do we need? Construct power of test do we have high probability that the test will detect important differences? Test Reporting Provide confidence levels for all results. Provide confidence above threshold where required. Joint Chemical Agent Detector (JCAD) Power of Test Power Analysis for JCAD Chamber Test DT Testing Statistical Response Surface Design (I-Optimal) High power test plan Factor S:N* = 0.5 S:N = 1.0 S:N = 2.0 Temperature 32.0% 84.7% 99.9%

Water Vapor Content (WVC) 42.1% 94.1% 99.9% Concentration 46.5% 96.3% 99.9% *S:N signal-to-noise ratio, goal detectable difference as a ratio to the design standard deviation Mobile Gun System (MGS) Power of Test DOE Interrupted by Deployment Original Test Plan (Sample Size = 16) (Sample Size = 22) Factor Mission Type

Terrain Type Threat Level Illumin. S:N* = 0.5 S:N = 1.0 S:N = 2.0 7.7% 16.6% 54.1% 17.0% 51.3% 97.8% 9.4% 24.4% 75.5%

15.9% 47.9% 96.7% Factor Mission Type Terrain Type Threat Level Illumin. S:N = 0.5 S:N = 1.0 S:N = 2.0 5.7% 8.1% 18.3% 10.6% 28.0%

78.2% 6.4% 10.9% 31.2% 10.1% 26.0% 74.3% *S:N signal-to-noise ratio, goal detectable difference as a ratio to the design standard deviation Lesson Learned: smaller sample size decreases power EA-18G/EA-6B Comparison Confidence Intervals Percent Success Figure from DOT&E EA-18G BLRIP EA-18G/EA-6B Comparison Confidence Intervals Percent Success Figure from DOT&E EA-18G BLRIP

Confidence intervals make it clear performance is comparable Mobile Gun System (MGS) Confidence Intervals Even without a threshold, confidence intervals quantifies how well the metric was measured Percent Target Hit Given Engaged with 80% Confidence Bounds 100 80 60 40 20 0 100 80 60 40 20 0 Targets Hit given Engaged MH-60R/S P3I Confidence Above Threshold Metric Demonstrated

MTBOMF (Romeo) Threshold = 14.8 hours MTBOMF (Sierra) Threshold = 20.3 hours Mission Capable Rate (Romeo) Threshold = 70% Mission Capable Rate (Sierra), Threshold = 69% 49.8 hours 41.8 hours Confidence Above Threshold 99% 99% 75.2% Unknown 71.3% Unknown For both aircraft, all mission failures were due to legacy airframe issues vice P3I systems MH-60R/S P3I Confidence Above Threshold Metric Demonstrated

MTBOMF (Romeo) Threshold = 14.8 hours MTBOMF (Sierra) Threshold = 20.3 hours Mission Capable Rate (Romeo) Threshold = 70% Mission Capable Rate (Sierra), Threshold = 69% 49.8 hours 41.8 hours Confidence Above Threshold 99% 99% 75.2% Unknown 71.3% Unknown Lesson Learned: Dataaircraft, not available to calculate. For both all mission failures were due to legacy airframe

data collection and issuesWatch vice P3I systems management plan Mobile Gun System (MGS) Data Analysis DOE illustrates how performance varies across envelope Mobile Gun System Terrain Threat Mission Urban Terrain Mixed Terrain High Threat Low-Mid Threat All Attack All SASO All Defend Proportion of Successful

Missions according to Army Subject Matter Experts Infantry Carrier Vehicle Based on RTCA Data Based on RTCA Data Start Lost Start Lost Proportion of Successful Missions Based on Achieving Stated Unit Mission 80 % Confidence Interval

(# success / Total SME) Proportion of Missions where Mobile Gun System Contributed Positively to Mission as rated by Army Subject Matter Experts 63% 35%-85% 54% (22/41) 88% 24 4 32 15 75% 63% 46%-93%

35%-85% 51% (20/39) 38% (19/38) 74% 78% 24 24 8 11 32 32 9 12 75% 50% 100% 83% 46%-93% 24%-76% 32%-100% 49%-98% 59% 46% 70%

55% 84% 77% 76% 90% 24 24 6 18 1 6 0 6 32 32 8 24 12 15 1 8 (26/44) (19/41) (7/10) (16/29) Overall Mission Success Rate is 69%

Mission Success tied to unit achieving assigned objectives and unit losses USS Virginia Metrics Confidence Intervals Statistical metrics may require special techniques Metric Demonstrated Confidence Secure Search Rate versus SSN (moderately difficult environment) 9 runs against USS Georgia. Demonstrated XX nmi2/hr. Bootstrap methodology (non-parametric, but very small data set): 90% confidence Secure Search Rate is less than XX nmi2/hr Tomahawk Missile Reliability 3/3 on USS Virginia 90% confidence interval 0.37 1.0

XX/YY in testing on similar systems XX/YY yields: XX% confidence performance is above threshold of XX 90% confidence interval of XX - XX Provide supplementary details from past testing. Previous Tomahawk testing demonstrated Use factors and past data to identify limited test scenarios Summary Summary Next Steps: Modify TEMPs, T&E concept papers, Test Plans, and BLRIPs to incorporate DOE concepts Have quantitative, mission-oriented metrics: What is the question(s) we are trying to answer? What are the applicable metrics? Describe how well the operational envelope is covered: Identify factors that drive performance Identify levels for each factor Show how well the test covers the operational envelope For both individual test periods and the overall test program

Calculate the confidence level and power of the test: Test plan: Significance, Power, Effect Size, sample size Test reports: XX% confidence interval Confidence performance above threshold Consider whether standard DOE designs are applicable

Recently Viewed Presentations

  • Big Question:

    Big Question:

    Big Question: How does technology help adventurers reach new places? Monday. Tuesday. Wednesday. ... ooze - soft mud or slime, ... parts and sensing devices controlled by a computer. Vocabulary Words. sediment - material that settles to the bottom of...
  • Waves - UCLM

    Waves - UCLM

    Waves encountering barriers: Reflection, Refraction and Difraction, The Doppler Effect Superposition, Interference Standing waves INTRODUCTI0N. TRAVELING WAVES a wave is a disturbance that travels through space and time, usually accompanied by the transfer of energy.
  • (Spam vs.) Forty Years of Machine Learning for

    (Spam vs.) Forty Years of Machine Learning for

    Defeats "M A K E * M-O-N-E-Y", etc. Intended to be only one of several reps Feature Engineering Task-specific feature engineering helps a lot Construction of text & nontext features Ubiquitous in operational TC Good features are a learner's best...
  • fhsaonline.weebly.com

    fhsaonline.weebly.com

    Why is this important? In your lifetime you will be told to get vaccines through an injection. It is important to understand why we are putting these things into our body.
  • Secondary Traumatic Stress: Caring for the Caregiver

    Secondary Traumatic Stress: Caring for the Caregiver

    Secondary Traumatic Stress: Caring for the Caregiver. Mary Mueller, LMSW. Michigan Department of Health and Human Services. Objectives. Define Secondary Traumatic Stress, Its Causes and Impacts. ... Very Busy & Stressful. Busier & More Stressful. Calm & Running Smoothly.
  • Meet the Teacher Night

    Meet the Teacher Night

    VCOP is used to develop the quality of writing and success criteria is normally linked to an aspect of this. Vocabulary (Wow words) Connectives. ... Developed and reinforced through games. Vocabulary: Word of the Day. 2-3 new words per week....
  • Late Quaternary (esp. post-LGM) environmental change in the

    Late Quaternary (esp. post-LGM) environmental change in the

    The palaeoclimatic record from eastern Bolivia Late Holocene expansion of Amazonian rainforests The palaeo-climatic record from east-central Brazil The palaeo-climatic record from north-eastern Colombia Long core sites (Australia-Indonesian region) Long cores: Australia-Indonesia PowerPoint Presentation Holocene climate change in East Africa...
  • Porters Five Forces of Industry Competitiveness Substitution Threat:

    Porters Five Forces of Industry Competitiveness Substitution Threat:

    Porter's Five Forces of Industry Competitiveness Industry Rivalry Threat of New Entrants Supplier Power Buyer Power Threat of Substitution Entry Barriers: Economies of Scale Capital Requirements Access to Distribution Etc… Buyer Power: Buyer Concentration Buyer Volume Buyer Switching Costs Ability...