NASA/NOAA failure report for GOES-17: The government screwed up
A joint investigation by NASA and NOAA into the failure issues on the GOES-17 weather satellite, launched in March 2018, have determined that the problem with the Advanced Baseline Imager (ABI), the satellite’s main instrument, was caused by
a blockage in the instrument’s loop heat pipes, which transfer heat from the ABI electronics to its radiator. The blockage restricted the flow of coolant in the loop heat pipes, causing the ABI to overheat and reducing the sensitivity of infrared sensors.
You can read the short full report here [pdf].
My immediate thought in reading the press release above was: So a blockage caused the problem. What caused the blockage? Was it a design failure or a construction mistake? Or what? The answer to this question is even more critical in that the same issues have been identified in GOES-16, though not as serious.
Moreover, GOES-16 and GOES-17 are the first two satellites in a planned new weather constellation of four satellites. Knowing who or what caused this blockage prior to construction and launch of the two later satellites is critical.
I immediately downloaded the report and read it, thinking it would name the contractor and the cause of the blockage.
Nope. The report is remarkably vague about these details, which the report justifies as follows:
The report is NASA sensitive, but unclassified (SBU), because it contains company proprietary information. The report also contains information restricted by the International Traffic in Arms Regulations (ITAR) and/or the Export Administration Regulations (EAR). This summary report provides an overview of publicly releasable information contained in the full report.
In other words, this report is an abridged version of the full report, which is being kept classified because it contains both commercial proprietary information and information that if released would violate ITAR regulations designed to keep U.S. technology from reaching foreign hands.
What this public report does imply in its recommendations, in a remarkable vague way, is that the problem occurred because the government had demanded changes during construction that forced significant redesigns by the contractor, none of which were then given sufficient review.
Or to put it more bluntly, NOAA and NASA, the lead agencies in the GOES project, screwed up. They forced the contractor to make changes, probably very late in the process, resulting in inadequate review of those changes.
The recommendations put forth many suggestions to institute a more detailed review process, should late changes in the construction of the next two GOES satellites be required or demanded. Such recommendations however will only further delay and increase the costs for building those satellites. Since the entire constellation went overbudget significantly (from $7 billion initially to $11 billion), and has also been very late (see this GAO report [pdf]), this means that the next two satellites will be even later and more expensive.
For NASA and NOAA this is just fine, pumping more money into each agency. For the taxpayer it is terrible.
The whole process should be dumped. Give the job of building these satellites to the private sector, entirely. Get these agencies out of the construction business. The only contribution they are presently adding is more cost and delays, while also causing satellite failures.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
A joint investigation by NASA and NOAA into the failure issues on the GOES-17 weather satellite, launched in March 2018, have determined that the problem with the Advanced Baseline Imager (ABI), the satellite’s main instrument, was caused by
a blockage in the instrument’s loop heat pipes, which transfer heat from the ABI electronics to its radiator. The blockage restricted the flow of coolant in the loop heat pipes, causing the ABI to overheat and reducing the sensitivity of infrared sensors.
You can read the short full report here [pdf].
My immediate thought in reading the press release above was: So a blockage caused the problem. What caused the blockage? Was it a design failure or a construction mistake? Or what? The answer to this question is even more critical in that the same issues have been identified in GOES-16, though not as serious.
Moreover, GOES-16 and GOES-17 are the first two satellites in a planned new weather constellation of four satellites. Knowing who or what caused this blockage prior to construction and launch of the two later satellites is critical.
I immediately downloaded the report and read it, thinking it would name the contractor and the cause of the blockage.
Nope. The report is remarkably vague about these details, which the report justifies as follows:
The report is NASA sensitive, but unclassified (SBU), because it contains company proprietary information. The report also contains information restricted by the International Traffic in Arms Regulations (ITAR) and/or the Export Administration Regulations (EAR). This summary report provides an overview of publicly releasable information contained in the full report.
In other words, this report is an abridged version of the full report, which is being kept classified because it contains both commercial proprietary information and information that if released would violate ITAR regulations designed to keep U.S. technology from reaching foreign hands.
What this public report does imply in its recommendations, in a remarkable vague way, is that the problem occurred because the government had demanded changes during construction that forced significant redesigns by the contractor, none of which were then given sufficient review.
Or to put it more bluntly, NOAA and NASA, the lead agencies in the GOES project, screwed up. They forced the contractor to make changes, probably very late in the process, resulting in inadequate review of those changes.
The recommendations put forth many suggestions to institute a more detailed review process, should late changes in the construction of the next two GOES satellites be required or demanded. Such recommendations however will only further delay and increase the costs for building those satellites. Since the entire constellation went overbudget significantly (from $7 billion initially to $11 billion), and has also been very late (see this GAO report [pdf]), this means that the next two satellites will be even later and more expensive.
For NASA and NOAA this is just fine, pumping more money into each agency. For the taxpayer it is terrible.
The whole process should be dumped. Give the job of building these satellites to the private sector, entirely. Get these agencies out of the construction business. The only contribution they are presently adding is more cost and delays, while also causing satellite failures.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
Robert wrote: “What this public report does imply in its recommendations, in a remarkable vague way, is that the problem occurred because the government had demanded changes during construction that forced significant redesigns by the contractor”
This is an advantage to small satellites that can be designed, built, and launched in a short amount of time. Not only does it give the government less time to think up new requirements, but it allows for such rapid turnaround in satellites that maybe (I may be dreaming, here) NASA might we willing to think about not insisting on changes to the current contracted set of smallsats but could be willing to wait a couple of years to put the changes on the next satellite set.
Unfortunately, wanting changes during construction is common with aerospace projects. If you watched the series “From the Earth To the Moon,” the Apollo 1 episode showed the contractors complain that there were too many change requests being made too late in the design and manufacturing process. A seemingly small change can have ramifications in other places of the spacecraft, and a change can complicate everything.
Changes can be very expensive, and the difficulty experienced in part of the primary capability of NOAA’s GOES-16 and GOES-17 satellites is part of that cost.
Changes have led to cost and schedule overruns on many military projects, too. A friend of mine worked on the Aquila recognizance drone in the early or mid 1980s. That was a fixed price contract for 300 expendable drones. As the government insisted on changes, usually additional or different recon payloads, the number of drones that could be made for the fixed price decreased. Once the number of drones dropped so low that the cost per drone went above something like $300,000 each, the Army decided to change them to recoverable drones, another change that reduced the number and increased the per-drone price.
About that same time, the Israeli Army demonstrated that if you stick to the original design, you at least get something operational and that a bird operating in the field is worth far more than all the “better” birds trapped in the manufacturing plant’s design-change cycle.
I did some work on the International Space Station, one part of which was to work on a cryocooler for snap freezing biological samples. Because NASA needed some work to assign to the Germans, when they signed onto the ISS project, NASA gave them the cryocooler that only had three or so months of design work. I moved on to another project, but I heard that the cryocooler came back, because when NASA insisted that the German company work to new — changed — requirements, the Germans told them to pound sand — er — that they would not accept the new requirements, they had signed on only to the original requirements, and that was what they were going to make for NASA. NASA is very used to making changes to what it orders and to getting its way; they gave the Germans something else to contribute as their part of the ISS.
The commercial world is different. Commercial operations are cost conscious and need their satellites up, operational, and generating revenue sooner rather than later. One communication satellite that I worked on had an engineering meeting to decide what to tell the customer who had just heard of new traveling wave tube assemblies (amplifiers) that were 1,200 watt rather than the 1,000 watt TWTAs that we already had installed on the payload panels of the satellite. (The engineer across from me at the table noted that they wanted their amplifiers to go to 11. I seemed to have been the only one to get the joke.) The change would require that the payload design and manufacture start almost from scratch, to incorporate new mount-points for the TWTAs on the panel, larger heat pipes in the panel to carry away the additional heat, minor modifications to the waveguides that required remaking many of them, additional solar array panels to power the more powerful TWTAs, and cetera. The time delay and added cost were large, and the commercial customer wisely chose not to go with any changes until the next satellite.
The difference between commercial and government contracts is that commercial customers are far less able to afford the financial and schedule costs of changes. Their customers can always go to another company for service if the first one costs too much or takes too long.
For government, well, they can always get more money from We the Taxpayers, and they can ignore us when we complain about the cost and lack of service. It’s not like we can switch to another government for services; we are a captive customer base.
What benefit do we get from changes, anyway?
https://www.youtube.com/watch?v=4xgx4k83zzc (1 minute “These go to 11”)
Edward–
Good stuff.
George Harrison –
“Got My Mind Set On You”
(Version II, 1987)
https://youtu.be/_71w4UA2Oxo
3:51