Analysis of GAO's Annual Weapon System Assessment

A lot of smoke but pretty small, manageable fires.

GAO published its annual weapon system assessment report. As acquisition nerds, we read it each year, even if we don’t agree with all the conclusions. This year’s assessment spans 108 high-cost acquisition programs that includes Major Defense Acquisition Programs (MDAPs) which use the Major Capability Acquisition (MCA) pathway (traditional approach), Middle Tier of Acquisition (MTA), and programs that are early in their development but mostly plan to use the MCA pathway.

GAO Leading Practices

GAO found that leading companies prioritize developing and delivering new, innovative products to customers with speed using four key principles that underpin best practices in product development. We agree with these principles and think that all acquisition programs should adopt them…although there are degrees of business case development that range from taking 4 years to plan before actually making any progress. We prefer to see the business case based on experimentation or other real-world type scenarios…less on backroom financial gymnastics.

Key GAO Findings

These are some of the key findings that GAO highlighted which, absent context and experience managing a technology development program, may initially appear that the entire system is dysfunctional. We have been vocal advocates for accelerating acquisition and acknowledge that many of these efforts should have been fielded sooner, but there are also no egregious findings that indicate malpractice or that adoption of the leading practices above would have necessarily been resolved.

• Many DOD programs continue to use a slow and linear development approach and fall short of delivering capabilities quickly and at scale.

• DOD continues to face challenges in executing modern approaches and rapidly delivering software to users.

• The average expected time for MDAPs in DOD’s portfolio to deliver even an initial capability to the warfighter is 10 years.

• The MTA programs GAO reviewed plan to take an average of 10 years after the start of the MTA effort to deliver initial capability to the warfighter.

• MCA pathway programs have experienced an average increase in cycle time of three years from their original estimate to initial capability.

• MTA programs may be pursuing overly complex, cyber-physical systems, without using iterative practices that would enable them to do so with speed.

• The average cycle time of programs assessed increased by 4% from last year.

• The 20 MTA programs GAO reviewed are using digital engineering best practices at a far greater rate than the 88 programs using the traditional MCA pathway.

MDAP Cost Changes

Of the 31 MDAP programs that GAO assessed, there are some clear outliers that drive the deltas to plan.

• The F-35, F-15EX, Air and Missile Defense Radar, and MQ-25 constitute the vast majority of the cost increases.

  • The F-15EX cost increase is due to the Air Force increasing the quantity of aircraft by 33%, thus increasing procurement costs 32%.

  • The Navy’s AMDR also increased quantities by 146% which increased the total acquisition cost by 32%.

  • The F-35 is an edge case and probably should be isolated from the broader population, but its costs grew $8.9B due to increased modernization and production costs.

• The Columbia Class Submarine, MQ-4C, KC-46, GPS III and F-15 EPAWSS constitute the vast majority of the cost decreases.

  • The $9B in Columbia Sub and KC-46A Tanker cost reductions were based on inflation assumptions.

  • GPS III cost reduction was primarily due to revised inflation indices (8% delta) and economies of scale (shifting 2 space vehicles from FY24).

  • F-15 EPAWSS cost reduction was due to 200 aircraft upgrades being removed from the schedule. This incurred a Nunn-McCurdy breach given the lower quantity drove the program acquisition unit cost (total units divided by RDT&E) up but overall lowered the total program cost.

Overall Assessment: Removing inflation calculations and quantity adjustments, the DoD (across 31 MDAP programs) has experienced:

• $9B increased costs driven by the F-35

• $3B increased costs from various other programs.

• $3B cost efficiencies by the other MDAPs reviewed.

This is not a horrendous picture given the many programs underway and the complex requirements for these major programs. While some of these costs were no doubt avoidable, they are also to be expected to some degree given the lack of management reserve that is provided to DoD program managers - that is generally afforded to those in the commercial sector.

MDAP Schedule Changes

Given that a change from the first full estimate could have occurred many years in the past, we are most focused on what schedule changes are happening most recently.

• The Improved Turbine Engine Program experienced schedule delays primarily due to parts manufacturing issues (due to new manufacturing processes being adopted). Staffing issues (common among many programs) have contributed. In the meantime, the program completed critical design review for Black Hawk integration and initiated laboratory risk reduction activities for the Apache.

• The Ship to Shore Connector Program has experienced manufacturing issues related to cracking propeller blades, premature gearbox wear and welding failures. Recent deliveries indicate that the program resolved most of the issues, but the Navy has truncated production from 4 a year to 2 a year to minimize the need to make post-production repairs.

• The Next Generation Operational Control System (OCX) Program has seen Site Acceptance Testing for OCX Blocks 1 and 2 delayed by at least 9 months driven by delays in finalizing technical order for operator training. This program has been a nightmare for the space community and DCMA anticipates over 1,900 major deficiencies to be open when OCX Blocks 1 and 2 are delivered.

• The Advanced Anti-Radiation Guided Missile—Extended Range (AARGM-ER) Program is in production but has experienced delivery delays of 5 to 8 months on its first production contract due to multiple parts delays and quality issues. IOC is still planned for next month and the program/vendor is working towards that.

Overall Assessment: While select programs have experienced issues that might have been addressed with a more robust supply chain and a higher skilled manufacturing and software development workforce, there are no significantly worrying trends that the acquisition workforce is failing to do due diligence and making grievous errors. Many of the corrective actions noted demonstrate that the workforce is being proactive in resolving the issues using different contracting approaches, integrating government and contractor technical teams and making better use of metrics.

---

Middle Tier of Acquisition

GAO selected 20 (of the roughly 106) programs using the Middle Tier of Acquisition Pathway. Their focus was on the largest MTA programs. This includes six satellite programs, three hypersonic cruise missiles, and four aircraft programs.

The 86 MTA programs not reviewed are smaller than these and thus able to deliver faster. We’re saddened when reporters or staff take conclusions from incomplete reports that the MTA pathway isn’t contributing to accelerated deliveries and learning when compared to the traditional MCA pathway. This recent article for example is often indicative of the simplistic conclusions reached.

We are also realistic and readily acknowledge there are opportunities to improve MTA program strategies and streamline department MTA processes. We also see the need to examine the additional requirements levied by DoD and Congress that run counter to the pathway’s intent - and pushing its processes closer to the MCA pathway and back to status quo acquisition.

Recommend re-reading our post on How to Properly Execute an MTA Program and also our article in Defense One on some of the successes and reasons why MTA is helping to shift the acquisition culture.

Informative Graphs on MTA

GAO assessed the below 20 MTA Programs.

GAO’s Delays to Planned Operational Demonstrations chart doesn’t easily convey the myriad drivers that contributed to the schedule delays.

• The FLRAA program was delayed a year due to Lockheed Martin’s protest over the Army’s decision to go with Textron Bell. Protest delays like this occur irrespective of acquisition pathway.

• The CPS program experienced launcher and launch sequence issues at the test range and had to take corrective actions. While unfortunate and regrettable, the program has made substantial progress, and one or two successful tests could quickly see it accelerating into fielding. Testing will also serve to validate the digital engineering models it has developed which will add to the ability to make corrections.

• The IFPC program had an unanticipated dependency issue on another program (Integrated Air and Missile Defense fire control system) that has pushed out key testing dates. While also regrettable, it does not indicate significant issues with the technical progress of the program (maybe test will reveal issues - its TBD).

Overall Assessment: Given that the most complex MTA programs were chosen, the noted schedule delays are not egregious and do not in our opinion reflect poorly on the MTA pathway. The programs experiencing a more than 12-month delay are high-end kinetic systems which are usually challenged in testing given the high level of confidence needed to field such systems. The MTA pathway was crafted to generate learning at the earliest possible point - that is clearly being done here.

Technology Readiness

GAO reviewed six MTA programs that reported their Technology Readiness Levels for last year and this year and claimed:

“Our prior work has shown that increasing even one TRL can take multiple years and becomes more challenging as the technology approaches maturity. MTA programs transitioning with immature technologies may risk additional costly and time-intensive redesign work for the overall effort.”

GAO places great emphasis on measuring Technology Readiness Levels but failed to note that the MTA pathway provides a streamlined approach for getting a prototype into an operational environment as quickly as possible - thus serving to accelerate TRL maturation. As noted in GAO’s table below, a successful developmental test can quickly push a technology from TRL 5 to TRL 8.

We personally believe that TRLs are a poor indicator of progress since cost increases and schedule delays can come from many sources. The key is risk management, which GAO does not even mention as a key acquisition practice. A limited supply chain and one backlogged vendor can delay an entire program. An inexperienced component supplier can drive critical quality issues into a production line. A vendor with a limited software development talent pool can negate all the exquisite hardware advances. OCX is a prime example here. The program continues to report its five critical technologies as mature but also continues to be a nightmare program that probably should have been canceled a decade ago.

Overall Assessment: Testing delays on select programs do not necessarily indicate technology maturity issues - just an absence of some data. The CPS program does appear to have a significant amount of testing ahead that will likely identify other issues. DARC’s radar software may still not be ready for prime time. Apart from these outlier issues, there are no major trend lines that convey an issue with the MTA pathway execution.

Software Development

GAO reviewed 45 programs using “modern software development” since 2021. Of the 45 reviewed, only one (XM-30 Mechanized Infantry Combat Vehicle) is using the Software Acquisition Pathway - which was specifically designed to integrate modern software development approaches. XM-30 is not even using the Software Pathway for the initial configuration but planned for future upgrades.

We do agree however that all DoD programs should be exploring agile approach to gain earlier user feedback and explore ways to deliver software capability more incrementally. They should also be using modular contracting approaches to encourage competition where commercial software products are unavailable. We agree with the GAO that the Software Pathway is likely being underutilized and its practices would likely be of benefit to software elements across all acquisition pathways.

53 DoD weapons programs identified the following software workforce hiring and retention challenges.

Other related factors that contributed to program software challenges include:

• Competition with the industrial base

• Location of work/cost of living

• Slow hiring process

• Lack of experience

• Time to obtain security clearances

A major GAO finding was the DoD is slow to implement the Software Cadre, per NDAA direction. This is a valid criticism and a priority need for the DoD to bring in more software talent quickly. We see programs like Gig Eagle as a prime opportunity to tap industry’s software talent via reservists and put them into key roles across DoD.

Program Assessments

We examined a few of the key programs for cost relative to contract type.

KC-46A Tanker

If you want to know why KC-46’s total acquisition cost and unit cost are held down even with a 2% quantity increase…. look no further than the FIXED PRICE contract. While Boeing suffered $7B in losses by low-bidding the contract, the Air Force’s costs are stable. The 4% decrease over the last year is due to inflation calculations.

Air and Missile Defense Radar (AMDR)

Similarly, AMDR has a firm fixed price production contract and while quantities nearly tripled, procurement costs increased only 38%, cutting unit costs in half.

GPS III Follow-On

GPS IIIF saw a 12% decline in acquisition and unit costs over the last year. I wonder what contract type they’re using, oh would you look at that… Fixed Price Award Fee.

MH-139A Grey Wolf Helicopter

14% decrease in production and unit costs. Contract Type: Firm Fixed Price

T-7A Red Hawk

5% decrease in production and unit costs. Contract type: Fixed Price Incentive.

Indirect Fire Protection Capability Increment 2

A 0.7% decrease in costs for this MTA program using a FFP via OTA.

Integrated Visual Augmentation System (IVAS)

IVAS is an outlier among major programs. After a two year MTA rapid prototyping effort, IVAS began an MTA Rapid Fielding in Dec 2020. The current assessment shows a 76% decline in quantities with a 50% procurement costs. This is a FFP contract using OTA.

MQ-25 Stingray UAS

A 6% increase in acquisition and unit costs with a Fixed Price Incentive contract.

MQ-4C Triton UAS

Cut quantities by 61% (due to JROC re-evaluation of needed assets), yielding only a 16% total cost savings and doubling unit costs. This drastic cut, based on requirements needs, during production is very difficult.

Future Operationally Resilient Ground Evolution (FORGE)

Using a cost reimbursement OT authority agreement. FORGE is implementing an iterative approach for development with MVP along with requirements and design modularity. The program saw a 9% cost reduction.

Military GPS User Equipment (MGUE) Increment 2

Leveraging CPIF/CPAF; CPFF; FFP contracts, the program saw a 27% increase. Requirements and contract deliverables continue to evolve, contributing to cost and schedule uncertainty with the three vendors having costs, schedule, or technical performance challenges. JROC approved requirements reduction and will thus reduce two vendor contract requirements.

Summary

There is plenty of information across the 261-page report to go through. However, there’s only so much one can cover on complex MDAPs in two pages. Would like to see more strategic trends, analysis, and summaries across programs. Comparing Services, Prime Contractors, Contract Types, Acquisition Phases, Acquisition Pathways, and summary tables of timelines for programs going from Idea to IOC.

Join the thousands of subscribers to get our weekly recaps and thought pieces. Paid subscribers also get budget and legislative analysis and are valued supporters of our work.