NASA/NOAA failure report for GOES-17: The government screwed up

A joint investigation by NASA and NOAA into the failure issues on the GOES-17 weather satellite, launched in March 2018, have determined that the problem with the Advanced Baseline Imager (ABI), the satellite’s main instrument, was caused by

a blockage in the instrument’s loop heat pipes, which transfer heat from the ABI electronics to its radiator. The blockage restricted the flow of coolant in the loop heat pipes, causing the ABI to overheat and reducing the sensitivity of infrared sensors.

You can read the short full report here [pdf].

My immediate thought in reading the press release above was: So a blockage caused the problem. What caused the blockage? Was it a design failure or a construction mistake? Or what? The answer to this question is even more critical in that the same issues have been identified in GOES-16, though not as serious.

Moreover, GOES-16 and GOES-17 are the first two satellites in a planned new weather constellation of four satellites. Knowing who or what caused this blockage prior to construction and launch of the two later satellites is critical.

I immediately downloaded the report and read it, thinking it would name the contractor and the cause of the blockage.

Nope. The report is remarkably vague about these details, which the report justifies as follows:

The report is NASA sensitive, but unclassified (SBU), because it contains company proprietary information. The report also contains information restricted by the International Traffic in Arms Regulations (ITAR) and/or the Export Administration Regulations (EAR). This summary report provides an overview of publicly releasable information contained in the full report.

In other words, this report is an abridged version of the full report, which is being kept classified because it contains both commercial proprietary information and information that if released would violate ITAR regulations designed to keep U.S. technology from reaching foreign hands.

What this public report does imply in its recommendations, in a remarkable vague way, is that the problem occurred because the government had demanded changes during construction that forced significant redesigns by the contractor, none of which were then given sufficient review.

Or to put it more bluntly, NOAA and NASA, the lead agencies in the GOES project, screwed up. They forced the contractor to make changes, probably very late in the process, resulting in inadequate review of those changes.

The recommendations put forth many suggestions to institute a more detailed review process, should late changes in the construction of the next two GOES satellites be required or demanded. Such recommendations however will only further delay and increase the costs for building those satellites. Since the entire constellation went overbudget significantly (from $7 billion initially to $11 billion), and has also been very late (see this GAO report [pdf]), this means that the next two satellites will be even later and more expensive.

For NASA and NOAA this is just fine, pumping more money into each agency. For the taxpayer it is terrible.

The whole process should be dumped. Give the job of building these satellites to the private sector, entirely. Get these agencies out of the construction business. The only contribution they are presently adding is more cost and delays, while also causing satellite failures.

NOAA still struggling to get GOES 17 working

NOAA is still struggling to pinpoint and correct the problem in GOES 17, launched in March, that prevents it from taking certain infrared weather images.

In a teleconference with reporters, NOAA officials said they had been able to improve the availability of infrared and near-infrared channels on the Advanced Baseline Imager (ABI) instrument on the GOES-17 satellite since the agency first reported the problem two months ago. The spacecraft, originally known as GOES-S, launched in March. “ABI is already demonstrating improved performance from what was initially observed,” said Pam Sullivan, director of the GOES-R system program. Currently, 13 of the instrument’s 16 channels are available 24 hours a day, with the other three able to operate at least 20 hours a day.

That will change, though, on a seasonal basis, depending on the amount of sunlight that shines into the instrument. By September, the hottest part of the orbit, only 10 of 16 channels will be available 24 hours a day, she said, with the other six available “most of the day.”

The satellite is second satellite launched out of a four satellite constellation that NOAA is building for $11 billion. They have now also admitted that the same problem exists on the first satellite, but does not seem to be effecting performance in the same way.

New NOAA weather satellite has serious problem

Can’t anybody here play this game? The cooling unit required to take infrared images in the new NOAA weather satellite GOES-17, launched in March, is not functioning properly.

“This is a serious problem,” Volz said Wednesday in a conference call with reporters. “This is the premier Earth-pointing instrument on the GOES platform, and 16 channels, of which 13 are infrared or near-infrared, are important elements of our observing requirements, and if they are not functioning fully, it is a loss. It is a performance issue we have to address.”

Detectors for the infrared channels must be cooled to around 60 Kelvin (minus 351 degrees Fahrenheit) to make them fully sensitive to infrared light coming from Earth’s atmosphere. For about 12 hours each day, the cooler inside the Advanced Baseline Imager, or ABI, is unable to chill the detectors to such cold temperatures, officials said.

Infrared images from weather satellites are used to monitor storms at night, when darkness renders visible imagery unavailable. The three visible channels from the ABI are not affected by the cooling problem.

“The other wavelengths, the near-infrared and infrared wavelengths — the other 13 — need to be cooled to some extent beyond the capability of the system at present,” said Tim Walsh, NOAA’s program manager for the GOES-R weather satellite series. “There’s a portion of the day centered around satellite local midnight where the data is not usable, and that’s what we’re addressing.”

GOES-17 is the second of a four satellite constellation being built by NOAA costing $11 billion.

It appears that an identical cooling system was installed on the first of this satellite constellation, GOES-16, and has been working perfectly in orbit since November 2016. Why the new unit isn’t working remains a puzzle.

The real issue here is the cost and complexity of these satellites. Because they are so complex and take so long to build, replacing them is difficult if not impossible. Wouldn’t it be better to launch many cheaper satellites to provide redundancy at a lower cost?

This is a pattern we see throughout the government aerospace industry. NASA’s Webb and WFIRST telescopes are big and take decades to build. God forbid they fail at launch. SLS and Orion are big and take decades to build. God forbid they fail at launch. The Air Force’s numerous military satellites are big and take decades to build. God forbid an enemy takes one out.

In all these cases, failure means we get nothing after spending a lot of time and money. And replacing the loss will take years and billions of dollars.

Common sense says it is time to rethink this entire operation. Unfortunately, this is the federal government. The concept of rethinking anything, or even thinking at all, is too often a completely alien concept. I do not expect anything to change, unless we elect new people in Congress and the Presidency who are willing to take a hammer to this whole insane system and smash it bluntly. Trump is kind of this type of new person, but even he isn’t willing to change that much, only some things, such as the EPA, that irk him in particular. Otherwise, he has left much of the federal bureaucracy alone — as can be seen by his administration and NASA both gearing up to fund both LOP-G and WFIRST— thus continuing this pattern of big and expensive projects that take forever to build.