Update on Boeing’s investigation into Starliner valve issue
NASA yesterday issued an update on Boeing’s investigation into Starliner valve issue, noting that progress is being made.
Boeing has demonstrated success in valve functionality using localized heating and electrical charging techniques. Troubleshooting on the pad, at the launch complex, and inside the Starliner production factory at Kennedy Space Center has resulted in movement of all but one of the original stuck valves. That valve has not been moved intentionally to preserve forensics for direct root cause analysis.
Most items on the fault tree have been dispositioned by the team including causes related to avionics, flight software and wiring. Boeing has identified a most probable cause related to oxidizer and moisture interactions, and although some verification work remains underway, our confidence is high enough that we are commencing corrective and preventive actions. Additional spacecraft and component testing will be conducted in the coming weeks to further explore contributing factors and necessary system remediation before flight.
…Boeing has identified several paths forward depending on the outcome of the testing to ultimately resolve the issue and prevent it from happening on future flights. These options could range from minor refurbishment of the current service module components to using another service module already in production. [emphasis mine]
The announcement also confirmed that the next launch attempt of the unmanned demo mission is now being targeted for “the first half of 2022, pending hardware readiness, the rocket manifest, and space station availability.”
The highlighted words raise a very serious question. How is it possible for “oxidizer and moisture interactions” to cause this problem now on Starliner, when the environmental conditions at Cape Canaveral for spacecraft have been understood for better than sixty years? Though this problem might have uncovered a previously undetected fundamental engineering issue related to valves, I am very skeptical. It seems more likely that some quality control issue occurred during this capsule’s assembly. That they are considering using a different Starliner capsule for the demo flight strongly confirms this, suggesting again that the valve issue is not systemic to all valves but is specifically linked to the assembly of this capsule.
If this speculation is correct, it suggests there are some some very disturbing quality control problems in Boeing’s Starliner design and assembly processes. First they missed about sixty software issues that forced the premature landing of the capsule in the first demo flight, issues that should have been fixed during design and construction. Now it appears they have discovered assembly problems with the capsule’s valves, and only did so mere hours before launch.
Boeing has got to get these issues fixed, or it is going to have a serious public relations problem garnering private customers outside NASA once Starliner begins commercial flights.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
NASA yesterday issued an update on Boeing’s investigation into Starliner valve issue, noting that progress is being made.
Boeing has demonstrated success in valve functionality using localized heating and electrical charging techniques. Troubleshooting on the pad, at the launch complex, and inside the Starliner production factory at Kennedy Space Center has resulted in movement of all but one of the original stuck valves. That valve has not been moved intentionally to preserve forensics for direct root cause analysis.
Most items on the fault tree have been dispositioned by the team including causes related to avionics, flight software and wiring. Boeing has identified a most probable cause related to oxidizer and moisture interactions, and although some verification work remains underway, our confidence is high enough that we are commencing corrective and preventive actions. Additional spacecraft and component testing will be conducted in the coming weeks to further explore contributing factors and necessary system remediation before flight.
…Boeing has identified several paths forward depending on the outcome of the testing to ultimately resolve the issue and prevent it from happening on future flights. These options could range from minor refurbishment of the current service module components to using another service module already in production. [emphasis mine]
The announcement also confirmed that the next launch attempt of the unmanned demo mission is now being targeted for “the first half of 2022, pending hardware readiness, the rocket manifest, and space station availability.”
The highlighted words raise a very serious question. How is it possible for “oxidizer and moisture interactions” to cause this problem now on Starliner, when the environmental conditions at Cape Canaveral for spacecraft have been understood for better than sixty years? Though this problem might have uncovered a previously undetected fundamental engineering issue related to valves, I am very skeptical. It seems more likely that some quality control issue occurred during this capsule’s assembly. That they are considering using a different Starliner capsule for the demo flight strongly confirms this, suggesting again that the valve issue is not systemic to all valves but is specifically linked to the assembly of this capsule.
If this speculation is correct, it suggests there are some some very disturbing quality control problems in Boeing’s Starliner design and assembly processes. First they missed about sixty software issues that forced the premature landing of the capsule in the first demo flight, issues that should have been fixed during design and construction. Now it appears they have discovered assembly problems with the capsule’s valves, and only did so mere hours before launch.
Boeing has got to get these issues fixed, or it is going to have a serious public relations problem garnering private customers outside NASA once Starliner begins commercial flights.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
Here is an applicable quote from 65 years ago:
“It all looked so easy when you did it on paper —
where valves never froze, gyros never drifted,
and rocket motors did not blow up in your face.”
— Milton W. Rosen,
rocket engineer
Milton led development of the Viking and Vanguard rockets, and was influential in the critical decisions early in NASA’s history that led to the definition of the Saturn rockets, which were central to the eventual success of the Apollo moon program.
If I understand this correctly, it sounds like oxidizer and moister lead to corrosion of the valve and causing it to freeze. This doesn’t sound like an assembly issue, but rather a materials problem caused by blatant disregard of the environmental conditions at the cape which, as Bob points out, has been known and extensively documented for 60 years. Probably a result of work done by a Jr. engineer who never saw conditions outside a clean room (or perhaps even a classroom) and no oversight by experienced engineers or management that what was specified in the design would indeed work within known environmental parameters.
I see such issues often in engineering endeavors, where the cleverness of a solution rather than the correctness of a solution is praised and rewarded, leading to premature and unexpected failures. I don’t know how things are at Boeing, but it’s probably not too different than industry-wide trends where older experienced (ie. expensive) engineers, who have hard-won battle scars, are pushed aside for younger inexperienced (ie. cheaper) engineers who have yet to prove they have what it takes to design something that “just works”.
mpthompson – I agree that this was probably caused by (as you said) “disregard of the environmental conditions at the cape which, as Bob points out, has been known and extensively documented for 60 years.”
That is why I posted a quote from 1956.
With these valves, Boeing has had an Abysmal Engineering Fail!
Besides me only mpthompson has commented on this. Where is the outrage from the rest of Bob’s readers with big engineering brains? As I am not one of those, I’ll relate an interesting story.
In the set of Boeing cubicles where the Starliner valve group works, behind a quality control excellence appreciation plaque from an Engineering Director that was sacked in 2019, there is an old yellowed Dilbert Cartoon. In it Dilbert is on a walk with his mother. Along the way his mother asks him quite innocently, “How was work Dilbert?” At that Dilbert goes on a mini-rant about his job saying;
“I’m like a fly stuck in a thick tar of despair.
Incompetence hangs in the air like the cold stench of death.
I’m drowning and monkeys dressed as lifeguards are throwing me anvils. As he rants on, his mother responds with. “Next time just say ‘Its fine.’”
I’m not one of the “big engineering brains” but it doesn’t take one to see Boeing has some serious issues, even before the valve failure.
At this point I’m past outrage and left with curiosity.
It’s been many weeks since the fault occurred, I would think if it were a quality control issue, something not meeting the design specs, (a fastener improperly tightened, a part machined to the wrong dimension or improperly finished, a non-specified material used etc) then the problem should have been found by now.
If it’s a design issue ( a Jr. engineer not taking something into account ), that can take longer, but why would using the other capsule, presumably built to the same design, be an option?
Reading the article possibly sheds some light.:
“This is a complex issue involving hazardous commodities and intricate areas of the spacecraft that are not easily accessed”
“Boeing completed a partial disassembly of three… of the valves last month and plans to remove three valves from the OFT-2 spacecraft in the coming weeks for further inspection.”
So in all this time they have managed a partial disassembly of three valves and it will take more weeks to remove three more valves?!!!
Maybe the root problem is the craft is designed to be so difficult to service.
Reminds me of when Orion had the problem with it’s PDUs. The time consuming issue wasn’t the PDU fault, it was the way the capsule is designed, getting to the PDUs requires major disassembly, and reassembly requires delicate precision, time consuming procedures.
I wonder how Dragon and New Shepard compare…
Mark asked: “Besides me only mpthompson has commented on this. Where is the outrage from the rest of Bob’s readers with big engineering brains?”
I have made comments on other threads relating to this problem, and at this point I am commented out, except for uninformed speculations, which I try to avoid. However, I’m not sure that outrage is an appropriate response.
Robert, Mark, and mpthompson have made good points. The update article makes it sound as though both Boeing and NASA believe that they have enough understanding of the problem to solve it. Robert’s point that it may be specific to this spacecraft and could be a manufacturing problem, but since the valves worked well a few weeks earlier, I suspect that the problem occurred due to events between test activations of these valves. This does not suggest to me that it was a design problem or a manufacturing problem, with the possible exception that the design was not adequate to handle the rough weather that they had between these test activations. Another possibility is a procedural failure to adequately protect the valves from the effects of inclement weather. To risk the speculation that I try to avoid, there is a possibility that the moisture in the lines occurred due to temperatures of those lines dropping below the dew point, and there are things that they could have done to prevent that from becoming a problem.
Meanwhile, the article shows that Boeing has been working the problem systematically, and save for any failure to verify their analysis or their solution, they could be flying in the early part of next year, only two and a half years behind where they could have been had their software team been better at quality control. When investigating a problem, care and time must be taken to avoid disturbing the evidence that is needed to find the root cause, not the proximal cause. Moisture interacting with the propellant is the proximal cause of the problem, but the root cause is the problem that allowed this to happen. Removing one valve at a time allows for future opportunities to investigate the root cause, should there be a failure during verification to their analysis or their solution.
Watching Starship development testing may have misled some people into thinking that everything can be done rapidly, but Starliner is in final verification testing with hardware that is intended to soon be operational for manned flights, not unmanned test items that were slapped together so that various systems and methods could be investigated. SpaceX is making throwaway test units at a very rapid rate, but SLS and Starliner are not throwaway items and must be treated with the care that manned flight hardware requires. When SpaceX had their valve problem on Crew Dragon, they also lost many months before they could fly, just as is now happening with Boeing’s valve problem. One difference is that Boeing has valves that can be examined.
A couple of weeks ago, I had an interesting discussion with a friend, a retired businessman and engineer, who reminded me that perfection on the initial release of any product is impossible. My brother had been working on a software problem that had been known before the product was released, but it had not been addressed until it became a problem for customers. When I mentioned my brother’s plight, my friend said that we have to do the best we can with the resources we are allotted in the timeframe that we have, and then we must be prepared to solve the problems that arise from operating in the real world. Otherwise we will be working out the bugs forever and our competitors will get the business that we should have had. The trick is to solve the important problems before product release so that the customer can get his job done while we make a more perfect product.
Boeing had two serious software problems (one caused premature termination of the test flight and the other could have caused a reentry catastrophe) and sixty to eighty minor ones. Later, they had a major hardware problem, and all three of these have delayed the customers getting their jobs done with Boeing’s hardware. Fortunately for the customers, NASA, a private company, and some individuals, the competition was able to get some, most, or all of the business that Boeing should have had, and these customers are able to get their jobs done while SpaceX operates Dragon even while making it a more perfect product. These problems have been a direct expense to Boeing, which must pay employees while Starliner is not operational, and it has caused the lost opportunity costs of two years or so of private commercial flights and perhaps one or more flights under contract with NASA. My educated guess is that the direct cost and lost income cost will come to a total of well over a billion dollars.
So, did saving a few bucks on cheaper, less experienced engineers and programmers cost the company more than it saved them? (Please consider this a rhetorical question.)
Edward, “at this point I am commented out”… five paragraphs follow! :o
All good points, I’m sure Boeing had the tools and the talent to build and operate valves on a spacecraft in the real world, but lost the lessons learned somewhere along the way. I like to blame bean counters.
This is a quick note of appreciation to all those who contribute to Behind the Black, and I acknowledge that Edward is right in stating that he (and many others) have made past comments on other threads relating to the engineering problems that Bob describes in his post.
Also, I only used the terms ‘big engineering brains’ and ‘outrage’ in my reply above as part of friendly banter, and as proof my writing had a made up story featuring Dilbert so obviously my tone had a humorous bent, and I meant no offense.
As a reader of Behind the Black, but without direct engineering work experience, when I read a post from Bob about success or difficulties at companies involved in commercial space, I have the following ideas swirling around in my head;
* Rocket & Space Vehicle Engineering is hard
* Rocket & Space Vehicle Production is hard
* But these have been successfully done in the past (hence my quoting Milton W. Rosen in my top reply), and currently today some companies are succeeding
* Management varies from poor to excellent
* Leadership varies from poor to excellent
* And lastly the probability of success in all of the above increases if the market ecosystem is based on the principles that Bob espouses through the tenets of his ‘Capitalism in Space’. policy paper
In that ‘Big Picture’ framework, let me know if I’m missing something fundamental as I’m always up for learning something new.
Mark: It should be obvious to those who read my work, I am not an engineer. I look at these stories solely from a management, social, cultural, political, and philosophical perspective.
I spent almost two decades doing production work in the movie business. To put together a feature film fast, with a crew of a 100 or more, shoot the film in a limited time frame, and then close out operations after completion of filming, requires very tight and smart management. Thus, when I see problems like those we’ve seen from Boeing, my immediate thought is not the engineering, but how the company’s managers are running the company.
My sense at Boeing (helped by its other problems outside of space) is that there are serious management problems there that must be fixed if the company is going to compete successfully in the commercial manned space market. Whether Boeing’s management has the courage or culture to do so remains a very very very unanswered question.
IMO the root of RosBoeing’s problem is a fundamentally flawed design combined with a naive engineering and development strategy. They design everything on the computer and it all fits together great, but their design appears to embed key systems in such a way that they are difficult to impossible to access after assembly thus impairing easy test and repair
If you have to disassemble the entire spacecraft in order to access and repair a key system it is not seriously designed with reuse in mind. And if such systems are unavoidable you damn well better fully integrate and test them under operational conditions prior to sealing them away inside the assembly.
You cannot assume quality, it must be proven. And when you primarily rely on computer models and piece part testing for a large complex system you make this mistake on a grand scale, and leave yourself vulnerable to experiencing highly public Ass of U and Me debacles like this.
MDN: Everything you describe is not engineering, but management. It is management that allows the engineering to be designed on a computer. It is management that does not insist that the capsule be designed so that it is easily repairable. It is management that allows such flawed and insufficiently tested designs to be built.
Engineers have to be valued. Profit motive IS the problem here. Good engineers need money…management won’t let them have it. The good engineers walk out…and know nothing kids have to start the learning curve ALL OVER AGAIN.
Musk will work you to death…but he won’t starve you. He has vision. Most businessmen don’t…..
John,
You noted: ““at this point I am commented out”… five paragraphs follow! :o”
The context that you skipped was important. ;-)
MDN wrote: “If you have to disassemble the entire spacecraft in order to access and repair a key system it is not seriously designed with reuse in mind.”
I was not under the impression that they had to disassemble the whole spacecraft, just that the valves were in a difficult to access — not impossible to access — location. This occurs often in spacecraft, which must be built small in order to save weight, thus there are parts that wind up around a corner or otherwise difficult to reach. I once had too little space to locate a large number of connectors in one of my designs, so I asked the lead technician if the techs could mate and demate them if I packed them too densely to get their fingers around. He said yes, they would work out something.
“You cannot assume quality, it must be proven.”
Very true. In addition, quality must be designed in. It cannot be inspected in. Quality must be important from the moment of project conception. It is why there are so many redundant systems in spacecraft, to avoid as many single points of failure as possible.
Mark noted: “But these have been successfully done in the past (hence my quoting Milton W. Rosen in my top reply), and currently today some companies are succeeding”
If we compare the lessons learned with the aircraft industry with those of the space industry we can see that there were far more aircraft flights per year than there are spacecraft flights. Learning the traps and where they lie is taking longer in the space industry. It is likely that different management styles give better or worse results in finding these traps without springing them. Which styles are more successful, and why?
Mitch S. pondered: “I wonder how Dragon and New Shepard compare…”
This may be more a matter of contrast than comparison. New Shepard has a much different mission, much shorter and requiring less of each consumable, such as power. Thus the PDUs are likely different, and New Shepard undoubtedly has fewer parts to need maintenance or refurbishment.
Orion is intended to be expendable, so its parts were not intended to be refurbished and some would have been seen as unlikely to need repair or maintenance, so easy access or replacement was not high on Lockheed Martin’s priority list. This decision bit them badly, almost a year ago.
Valves, too, are not a high maintenance item, and many likely would be expected to only need to be accessed during overhauls, if Boeing intended to extend Starliner’s lifespan. In addition, because they handle toxic fluids, they are located outside the pressure vessel, so would best be accessed from the exterior.
Design is always a matter of tradeoffs, and you are pretty much guaranteed to get bitten by something. The trick is to make sure the bite is an ant rather than a grizzly bear, so you want to think of everything from machining the parts to testing the assembled unit to repairing it to depressurizing it without causing condensation, or how to survive any condensation you can’t avoid. Valves bit both SpaceX and Boeing, but SpaceX was lucky in that their valve failure happened earlier in their verification process. Unfortunately for both, they were bit on hardware they had scheduled for future use. Boeing has to replace some valves, but SpaceX had to replace a Dragon. Both happened late in verification, when they could not really afford delays.
Blowing up Starliner test units during research and development is much more expected, especially when you are doing something radically new rather than the same-old same-old. The number of test units that SpaceX planned to build suggest that they learned what they wanted to know far faster than they had expected. I suspect that they thought they would be testing their ideas through most of this year and were surprised when they wound up months ahead of schedule, causing them to move forward their schedule for building their orbital launch pad and ground support equipment and facilities. (Did they get bitten by success? How often have your schedules moved to the left instead of to the right?) It is so much better to blow up your development test items than your finished product.
This is a good discussion on this post, and I hope it keeps going.
It didn’t take this long to figure out what happened.
It took this long to figure out who to blame.