My article series to this point focused on six degrees of failure in the disastrous National Reconnaissance Office (NRO) Future Imagery Architecture (FIA) Procurement that was undertaken to revolutionize US government space reconnaissance and surveillance into the 21st century.
I want to briefly summarize-or more properly emphasize-some of the top-level points I’ve covered so far that specifically weigh-in on the scale of supporting my premise that is the basis of the story. With any series of this length, you face the challenge-and often run the risk in trying to bolster the case with examples-of wandering away from the original point of the story far enough that dear reader has no idea why we went there and where we’ve landed.
I’m as prone-if not moreso-than most in that regard, of making what I think is a cogent point that dear reader may view as a puzzling diversion. Particularly in this instance where there have been myriad examples of worst practices and lessons not learned where our government-and I mean the people trained and charged to oversee the projects-and in many cases-the leaders above them, and oversight-were in effect part of the problem. Plus, one must attend to the security concerns (“one” is me.”)
The mechanisms of government translated into governance and management-the due diligence process called “program management” that is intended to identify, ferret out and prevent or shortstop these described issues from becoming problems and eventual showstoppers in a big, sprawling, complex program like FIA-proved wanting and failed to meet obligations: in an NRO organization that has a reputation for, and prides itself on excellence in the discipline.
Many leaders (being charitable here, as I already belabored the manager versus leader issue) somewhat inherit program oversight responsibilities as they move up the Peter principle opportunity ladder in this regard, in what I have come to believe is a worst practice in government. My belief strengthened into a conviction over the years as I’ve accrued data points borne of experience on dozens of programs both large and small (from hundreds of thousands to millions to billions of dollars,)) as well as hundreds of projects that form the basis for my conviction.
I noted previously the difference between a program and a project being that one is funded in the program build, while the other is more of an agency line item that is likely rolled up in the budget of a program. But most often a program is a fairly large dollar amount spread across many years, with a concomitant mandatory oversight regimen that dictates the amount of program scrutiny and the level at which it takes place.
An Acquisition Category 1 (ACAT 1,) Major Defense Acquisition Program in DoD-the ones you read about in the paper-is of congressional interest level, managed through processes that involve the Under Secretary of Defense, Acquisition, Technology and Logistics (AT&L,) who conducts and manages the process through Integrated Product Teams. There are the boards and council processes leading to the JROC through successive levels (0-5/LTC, 0-6/COL, 0-7 FLAG/GO/SES,) with interaction from the services, agencies and combatant command representatives, and ultimately congress.
FIA went through several stages of shifting scrutiny during its history that may have contributed and played a part in allowing Boeing to continue so long on an apparently hopeless part of the program. Including the period from contract award in June and September 1999 through Gap Mitigation Study out brief to the Director of Central Intelligence (George Tenet) 18 October 2002, followed by the establishment of the Under Secretary of Defense for Security and Intelligence (USDI) in fiscal 2003-2005, and the Director of National Intelligence (DNI) standup in 2005 when the DNI assumed the Milestone Decision Authority (MDA) for the program and canceled the Boeing optical component in consultation with DoD AT&L and USDI. Subsequently awarding the optical component to Lockheed Martin Corporation (LMC.)
The mechanics of the above are likely of little to no interest to dear reader, being germane in my story context only because all of these processes throughout the years-that have worked for myriad programs-could not salvage Boeing’s optical component effort. My minor point being this was a complex program being worked all over the country employing thousands of people. Somebody new to the program-and in a case like DNI Negroponte or one of his key deputies-State Department executive Patrick Kennedy-who started showing up at these meetings with State’s commercial imagery policy lead and somewhat DNI Exec or Chief of Staff Gil Klinger-would be hard pressed to get up to speed and at a sufficient comfort level to act when and as needed.
The program and process can provide and cover and review all the documentation in the world, execute control of the decision gates, pay due diligence right up through MDA for key parts of such programs, which can be adjudicated to the USDI or remain with the DNI based on funding (National or Military Intelligence Program funds,) but if the contractor has a camel and we need a horse-no amount of oversight can make that happen.
The pace or “pacing” of the reviews comes into play as well, with a plethora of program related control sessions that review all manner of mandatory documentation, such as program level briefings that detail cost, schedule and performance data (metrics, expenditures, schedule slack or margin, insight into particularly nettlesome development line items, so-called “problem child” aspects of schedule or performance, etc.,) with typically monthly reviews within the executing organization, a minimum of quarterly at DNI or JROC-AT&L-USDI or both, and as much or as little as congress has an appetite to do in the requisite governing committees (budget, defense, etc.)
The above “pace” is for programs that are approved and carried in the budget, with all the corresponding program exhibits, documents and processes. While projects-in most cases-do not warrant this level of scrutiny and are controlled within the responsible agency. Unless they are political projects from-oh-say CIA—getting special attention from the DNI and congress to steal money from programs. This matter of programs versus projects is cut and dry and normally well understood within the business.
One more quick example: our property is zoned as agriculture. I can put up a fence, chicken coop, or barn all day long on my ranch, but mess with the power box-electric lines, septic or water system and that action crosses a threshold and requires a permit from the county: somewhat similar to the program vice project distinction.
If the above is both obvious and well-known information, the point is for those who may be unfamiliar with the distinction, or who have been following the “trail” to this point and have puzzled over what went wrong with FIA-specifically the optical component-what processes were either skipped or not done or were delegated to the contractor to execute due diligence upon-a well-known worst practice-vice the government control as would be normal.
Such a thing was unheard of later in the white space world, but with some of the engineering feats accomplished in the “early days” with somewhat minimal government involvement, a different school of management and oversight somewhat emerged. Early efforts by Clarence Leonard “Kelly” Johnson, including the P-38 and the P-80 in World War II and later efforts out of the Lockheed Skunk Works, including the U-2 in the 1950s-completed in 9 months from contract/discussion, and the SR-71 in the sixties with its titanium construct and the fuel weeping, leaking engine nacelles as an engineering trade to costly design changes (e.g., you mean the plane is leaking fuel before takeoff? As we would say in the Army, it was a seep, not a leak. How can you tell the difference? Well, the Army regs are anal in this regard, so if it is a maintenance check, you would count the drips. If it’s a mission pre-flight, it’s a seep-because you can’t take off with a leak.)
Is there any chance a plane like that would be flight certified today? Would it pass process-DAWIA muster? What was a bigger US mission priority than getting to the bottom of the “bomber gap” in the 1950s? This was a period when all manner of complex space related efforts were undertaken in the drive to catch-up and surpass the Soviet Union after Sputnik in 1957, a time when engineering requirements were much more of an imaginative, collaborative approach between the government and manufacturers focused on the art of the possible, vice the limitations of cost, schedule and performance. The cost to make the SR-71 “leak-proof” was prohibitive to the point of a significant increase in cost that in the final analysis was not implemented.
Out of such success, there later emerged a philosophy or school of thought in the early to mid-1990s that revisited this era and postulated that based on many of the previous successes, the contractor is best suited to conduct most status checks, as they involve sub-contractors indirectly or only peripherally connected to the government. Whatever expectations and best intentions were thought to be gained from this thinking, it simply did not work with the FIA optical component-nor with another half dozen or so failures that each of the big US government agencies seemed to suffer post 9-11: when money was plentiful, oversight was distracted, and billions of dollars were spent on three-legged dogs (article for another time.)
The best acquisition practice is when the government employs a strong systems integration (SI) element that is empowered to be involved with the contractor and participates in the process sufficiently to maintain the non-biased view and status of the acquisition.
My experience was that this lawyerly-or DAWIA-like point about the contractor running their own status updates was a worst practice, reminding of the man lost while traveling in a balloon, who spots somebody below and yells out “Where am I?” and gets the most useless response ever, “You’re in a balloon!” and he of course replies, “You must be DAWIA certified or a lawyer-you told me what I already know and while correct, the information was absolutely useless.”
The bottom-line is the government makes the decisions on these contractual matters-not the contractor-because the program is funded with taxpayer dollars-duh….
Just about as bad as the Peter principle elevating unqualified and untrained officials into oversight positions is the penchant to opt for-and in many cases to mandate-that an acquisition trained professional is the best lead for many of these complex, complicated and costly programs: specifically, much worse when they lack domain expertise in the issue that is the rationale for the procurement.
I can’t tell you the number of times I received a response to my tongue in cheek preface (to a slam about to emerge) like “I’m not a certified acquisition professional, but” with the predictable response along the lines of “Here I come to save the day” (random Mighty Mouse reference.)
Nine times out of ten my comment was a rhetorical device intended to take the sting out of a comment about an obvious, but unresolved issue in some aspect of the program that the person I was talking to should have fixed or taken care of at the engineering review board, the engineering coordination board, the enterprise configuration control board, the design review board, the work order review process, or any one of a dozen forums stood up to manage and approve such things-but since it was being raised by me-at the time the program MDA-something in the process had obviously failed.
But when I said-for instance-and for context this was at a final enterprise architecture design review-“explain the rationale to me of why we still have two tape robots reflected in the design and cost data-sitting right next to each other-to backup applications software, system scans and access monitoring on different parts of the system. As opposed to what I approved during the preliminary design review a few short months ago where I directed that we implement an enterprise level backup scheme with the alternative prototype that we proved works, vice each vendor producing their own solution and including separate tape robots at twice the cost?”
A tape robot at the time cost ~$10 million with about ~$2 million in ancillary cost and residuals-tapes, personnel, integration code, etc. So the above was not what was in my head, which was something more along the lines of “What the flying ^%#$ is the matter with you ^%#$@^@&-why is it every time we have a status meeting you DAWIA dweebs exhibit rampant stupidity like @$#^%@&&^$@# and I have to be the one paying attention to un^%$# things?
I did not go full maniac mansion because in the government every conversation going forward would be about the “oh my panties” use of profanity, rather than the fact that the DAWIA professional engineers had-while apparently including the prototype as directed-which was already performing within the architecture at limited scale-but then overrode the systems integration (SI) team while ignoring direction from the MDA to eliminate the 2d tape robot and revisit the need for any robot, because neither vendor wanted to budge on the issue and change the design, cost and fee basis-so “they” did not force the issue of changing the design per direction, leaving it for decision at the final review.
Details, details: never mind the fact that a risk reduction effort had been undertaken that successfully demonstrated markedly improved performance, while eliminating the need for a buffer system to stage images, while demonstrating full functionality and replacing-totally-the tape robot in favor of using a Persistent online storage device with spinning disk and hard disk drives (San disks.) The system improved performance and obviated the need for any tape storage in the architecture: the prototype was successfully demonstrated, it was already performing at scale in the enterprise, was included in the concept design and was also part of the final design.
To summarize, these geniuses agreed with and implemented the risk reduction solution within the design of the data center, but also included the tape robot-that the solution was undertaken to determine if we could eliminate entirely-and they also allowed the other vendor to include a robot to back-up “their” systems and services.
Operationalizing and scaling the prototype-which cost about $450K-would trade about ~6M in new costs between hardware, code, integration costs, and design changes, for some ~35M dollars for the tape robots, while far exceeding the design performance specifications of the enterprise requirements and providing modularity in augmenting future improvements-vice the limitations of tape only solutions.
The original storage concept that we wrote in 2000-2002 in the 2010 Concept of Operations described what was becoming a classic on-line, near-line, off-line strategy to accommodate vastly increased image file sizes, while also addressing the access/use decay that showed less than 3% access to images that were more than ~7 days old.
With the proliferation of spinning and San disk technology, new designs were turning this access construct on its head by simply adding more storage, which for some time had become the cheapest component in the architecture: but not with a tape robot-based architecture, where the tapes were getting more capable and more expensive each quarter (in this case a commercial “tape farm” where some 600 tapes were accessible within some design specification-like 20 seconds.)
The dirty little secret was that the DAWIA folks working this issue just went with the plan that was implemented on the “other side of the interface” and it was infuriating to work through that potential solution and see two tape robots therein-“side by each-” that would have been joined by our two tape systems, albeit on the other side of the interface, representing a cost of nearly ~70M dollars just to do system backup.
We resolved the problem by working past 2100 that night: we were TDY, and it was a captive audience.
I likened what our DAWIA friends had done with making sure you had a backup power system in the event of an outage by building a second house: absolute worst practice engineering.
15 December 2022
LSMBTG: Lamestream media echo chamber (LMEC-L) social media (SM) big tech tyrants (BT) and government (G)
If you enjoyed this article, then please REPOST or SHARE with others; encourage them to follow AFNN
Truth Social: https://truthsocial.com/@AFNN_USA
Facebook: https://m.facebook.com/afnnusa
Telegram: https://t.me/joinchat/2_-GAzcXmIRjODNh
Twitter: https://twitter.com/AfnnUsa
GETTR: https://gettr.com/user/AFNN_USA
Parler: https://parler.com/AFNNUSA
CloutHub: @AFNN_USA
Sir, I’ll have you know that Habus do NOT “leak”–they MARK THEIR TERRITORY.
Or so I’ve been told by more than one Sled Driver over four decades around the aerospace world…
Hahahaha-the ones I watched take off from Mildenhall marked a lot of territory…