Newly discovered Starliner issues delay launch again
NASA and Boeing revealed today that two newly discovered design issues involving Starliner’s parachutes and the tape used to protect the capsule’s wiring has forced it to cancel the planned June launch, with no firm new launch date scheduled.
The parachute issue involves the parachute cords, specifically the “soft link joints” that connect those lines to the capsule.
[Mark Nappi, Boeing’s Starliner VP] told reporters fabric links that join the parachutes to the lines of the spacecraft, called soft link joints, need to be replaced and possibly recertified to withstand heavier loads and stresses to ensure crew safety. “They were tested recently because of a discovery that we found during the review process where we believed that the data was recorded incorrectly,” Nappi said. “We tested (the soft links), and sure enough, they did fail at the lower limit.” [emphasis mine]
The tape — which has been found to be far more flammable than expected — is difficult to fix.
The second problem found last week is more extensive since the tape used to protect Starliner’s wiring harnesses from nicks or abrasions runs for hundreds of feet through several of the spacecraft’s internal systems. “There is a lot of tape on the wire harnesses,” Nappi said. “We’re looking at solutions that would provide for potentially another type of wrapping over the existing tape in the most vulnerable areas that reduces the risk of a fire hazard.”
That both of these issues were not fixed in development is beyond astonishing and speaks so badly of Boeing’s engineering and management that it is difficult to find words. In fact, for Boeing to use tape that could cause a fire now, more than a half century after the Apollo 1 capsule fire, suggests a level of incompetence that makes one wonder why anyone would ever fly on any of its spacecraft or airplanes. This is certainly not the company that built the 747.
Officials indicated that they might be able fix this issues fast enough that a fall launch could occur, but made no promises.
For Boeing, this new delay only worsens its bottom line. It built Starliner on a fixed-price contract, so every delay and issue must be paid for by it, not NASA. Meanwhile, the delays mean that SpaceX is getting flight contracts to ISS from NASA, contracts that Boeing would have gotten had Starliner been ready as planned. Worse, ISS is now looking at a 2028 retirement. If Boeing doesn’t get Starliner operational soon, there might not even be any contracts for it to win.
I have embedded the full press conference below for those who wish to watch NASA and Boeing officials blather about how they really haven’t done anything stupid here. Really, you have got to believe them!
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The print edition can be purchased at Amazon. from any other book seller, or direct from my ebook publisher, ebookit.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
NASA and Boeing revealed today that two newly discovered design issues involving Starliner’s parachutes and the tape used to protect the capsule’s wiring has forced it to cancel the planned June launch, with no firm new launch date scheduled.
The parachute issue involves the parachute cords, specifically the “soft link joints” that connect those lines to the capsule.
[Mark Nappi, Boeing’s Starliner VP] told reporters fabric links that join the parachutes to the lines of the spacecraft, called soft link joints, need to be replaced and possibly recertified to withstand heavier loads and stresses to ensure crew safety. “They were tested recently because of a discovery that we found during the review process where we believed that the data was recorded incorrectly,” Nappi said. “We tested (the soft links), and sure enough, they did fail at the lower limit.” [emphasis mine]
The tape — which has been found to be far more flammable than expected — is difficult to fix.
The second problem found last week is more extensive since the tape used to protect Starliner’s wiring harnesses from nicks or abrasions runs for hundreds of feet through several of the spacecraft’s internal systems. “There is a lot of tape on the wire harnesses,” Nappi said. “We’re looking at solutions that would provide for potentially another type of wrapping over the existing tape in the most vulnerable areas that reduces the risk of a fire hazard.”
That both of these issues were not fixed in development is beyond astonishing and speaks so badly of Boeing’s engineering and management that it is difficult to find words. In fact, for Boeing to use tape that could cause a fire now, more than a half century after the Apollo 1 capsule fire, suggests a level of incompetence that makes one wonder why anyone would ever fly on any of its spacecraft or airplanes. This is certainly not the company that built the 747.
Officials indicated that they might be able fix this issues fast enough that a fall launch could occur, but made no promises.
For Boeing, this new delay only worsens its bottom line. It built Starliner on a fixed-price contract, so every delay and issue must be paid for by it, not NASA. Meanwhile, the delays mean that SpaceX is getting flight contracts to ISS from NASA, contracts that Boeing would have gotten had Starliner been ready as planned. Worse, ISS is now looking at a 2028 retirement. If Boeing doesn’t get Starliner operational soon, there might not even be any contracts for it to win.
I have embedded the full press conference below for those who wish to watch NASA and Boeing officials blather about how they really haven’t done anything stupid here. Really, you have got to believe them!
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The print edition can be purchased at Amazon. from any other book seller, or direct from my ebook publisher, ebookit. The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
Similar article over at Ars Technica
The one things that sticks out is Boeing is contracted for six flights. What will happen after the sixth flight? The Starliners get mothballed? Boeing is losing money on this one.
It does seem that they might be running the clock out to avoid possible flight problems.
OTOH, how many other vehicles use the same tape? Is it a real or a panic issue? I honestly can’t tell.
Boeing charged NASA 4.2 billion for 3 capsules.
SpaceX charged NASA 2.6 billion for 3, and is building a 4th, not sure for how much.
Ray Van Dune: SpaceX already has a fleet of four capsules, Endeavour, Resilance, Endurance, and Freedom. It has used all four for NASA flights, but I suspect three would have sufficed. Having a fourth has allowed SpaceX the ability to sell tickets for additional flights.
Back when it still looked like a race for which capsule was going to fly first, I remember lots of people going on about how sure, Boeing was charging more, and their development process was longer, but we could be sure they would deliver the superior product, unlike the upstarts at SpaceX who were surely cutting corners and just missing things because they weren’t putting enough attention into the design reviews, etc.
I need to go look through a ton of history just so I can get names and ask them how they want their crow seasoned.
That quote by Nappi indicates Boeing is still looking for a comparatively cheap and quick way out of its own-goal with the flammable tape. I suspect the only thorough solution will be to build completely new wiring harnesses, using a suitable tape this time, then doing what will probably wind up being an almost complete disassembly and reassembly of the vehicle to swap the old harnesses out and the new ones in. If NASA settles for Boeing improvising some sort of condom-like “solution” to the tape problem, that will be just one more indication of how the much-vaunted NASA “safety culture” always buckles when it would inconvenience one of the legacy contractor cadre. One hopes, at least, that Boeing now intends to use a better tape on the wiring of the EUS – which NASA intends to use to launch a full crew on its first-ever flight operation as part of Artemis 4.
1. SpaceX actually has 7 Crew Dragons in operation: 4 for crew (Endeavour, Resilience, Endurance, and Freedom) and 3 cargo-variant Dragons. It is building a fifth Crew Dragon for crew use, name and completion date yet to be determined. Obviously these variants operate under different contracts with NASA, but being able to use variants for both surely reduced SpaceX’s costs.
2. I think Starliner has been as much a debacle as anyone else here, but to be fair, it does seem that Boeing overhauled a lot of the team on the project over the last few years, and the new group, as Steve Stich suggested yesterday, seems much more on the ball. The problem is, they’ve had to fix or redesign so many problematic systems developed by the original Starliner team that it’s hard to perceive actual forward progress from the outside. They keep finding new problems buried by their predecessors, and then they have to atke time to stop and fix ’em.
At some point the clock runs out on this – the ISS won’t last forever – but since it’s all being done on Boeing’s dime now (until they actually fulfill remaining contract milestones), there’s not much skin to be lost by NASA in letting them try to work these problems. But NASA really do need to up their game in putting independent assessments in the loop to be sure they really have confidence in the final product before they put human beings on this thing. Something which I assume will not happen now until some time next year at the earliest.
Hello D.E.,
I suspect the only thorough solution will be to build completely new wiring harnesses, using a suitable tape this time, then doing what will probably wind up being an almost complete disassembly and reassembly of the vehicle to swap the old harnesses out and the new ones in.
Of course we can’t say for certain without a lot more insight into what they find, but this seems *likely* to be the more prudent way to resolve this.
NASA ain’t trying to race the clock (and the Soviets) to achieve something with this vehicle. NASA already has a ride to orbit. So, if this problem is as serious as it is beginning to sound (and it must be fairly serious if it got unanimous consent from all Boeing executives to halt flight preparations), take your time, and *really* fix it.
Richard M: Assuming Boeing has replaced its entire early Starliner team (a team that had repeatedly shown itself to be utterly incompetent), has it fired those individuals, or simply moved them to a different place in the company?
When Musk learns a team of his is doing poorly, he not only removes them, he makes it somewhat public, so that everyone knows that he has done his due diligence to clean house. He did this when he fired the original Starlink team he had hired from Microsoft. He has done the same at Twitter. His goal isn’t to slander the individuals, but to make it clear that his company doesn’t tolerate weak employees.
The tradition however in the big space contractors that have worked for NASA for decades (Boeing, Lockheed Martin, Northrop Grumman, etc) has been to simply move people around. Firing people was considered too cruel, and the extra payroll increased the bottom line since most of the contracts were cost-plus.
Which as Boeing done in this case? If the former, some clarity on this point, no matter how hurtful to the fired employees, would go a long way in increasing confidence in Boeing going forward. And if the latter, it will tell us that nothing at Boeing has really changed, and that as a company quality work is not its first priority.
This tape issue is an odd one. I’m really interested in how it turns out.
Wire bundle abrasion protection isn’t an issue new to Starliner. It happens all over the aerospace industry and in all sectors. The typical solution is to wrap the bundles with kapton or fiberglass tape in areas deemed vulnerable. The tapes used aren’t something you’d get at Home Depot. They’re aerospace grade materials built to industry specs.
So what happened? Did engineering call out the wrong tape? Did manufacturing make an improper substitution? Did a design change modify the use of an existing wire bundle? Is there a manufacturing defect in a certain batch of tape? Is the issue only a problem during an obscure failure mode that only recently came to light? Is this the start of an industry-wide change due to an unrelated issue?
This could make Boeing look really bad. Or it could make Boeing look really good, catching a problem during review similar to the problem SpaceX found by blowing up their capsule after it returned from the ISS. We don’t know yet, but I’m really curious.
Hard to imagine how electrical tape for this sort of application could have been reformulated and then improperly tested and certified in the last 56 years (do the math.)
Richard M,
It isn’t just that NASA isn’t in a race or is not so desperate that it will take the same kinds of risks that they did during Apollo, Boeing has to prove to its future potential customers that Starliner is safe.
mkent,
Those are excellent questions. SpaceX was able to recover from its infamous exploding capsule, because it happened only once, and the explanation made sense.
Boeing’s problem is more insidious. Safety problem after safety problem are continually being found. It is a drip, drip, drip type of problem. It wasn’t a one-time problem and everything is OK now, but a recurring problem lasting years. For the duration of the Starliner lifetime, passengers will constantly be wondering what life-threatening problem will crop up during their flight. I have previously read opinions here on BTB from people who already don’t have enough confidence in Starliner to ever fly her. This continuation of the dripping problem fiasco can only further reduce Starliner’s potential customer base.
What is the root cause of these problems? As someone noted: probably management early on. The move from Washington State was a complete disaster for Boeing, and even farther back, keeping the McDonnell Douglas management was poor judgement. Those guys have now destroyed two great U.S. aerospace companies.
I do not thin k that tape is the real problem.
They order a tape to specs. The supplier even tests the tape before sending it.
Then the customer Boeing double checks it. ‘
‘
To not blame the supplier is suspicious.
i’m amazed they are using tape, regardless of material, for wire harness covering. When i first started designing wire harness for construction equipment and looking at different suppliers, we were offered aircraft quality harness which featured blown-on neoprene covering with neoprene overmold at junctions and connectors. However badly i wanted them, the bean counters said “No.” OTOH, on our machines weight was not an issue.
Edward,
Boeing has to prove to its future potential customers that Starliner is safe.
Oh, I agree 100%.
And Boeing dug this hole all by itself.
A depressing (but not surprising) revelation from Eric Berger in his new article today: “Nickle-and-dimed.” Now they’re paying for it.
Although [Starliner program manager Keith] Reiley said in 2009 the company was making a “substantial investment” in the development of Starliner, then known as CST-100, multiple sources told Ars that was not the case. Instead, Boeing for a long time “nickel-and-dimed” the time engineers spent working on Starliner. This was partly due to congressional underfunding of the commercial crew program but also because Boeing did not want to put skin in the game. This has been a poor decision in retrospect because, due to the fixed-price nature of its contract with NASA, Boeing is largely responsible for cost overruns and losses due to ongoing delays.
https://arstechnica.com/space/2023/06/to-keep-starliner-flying-boeing-must-make-some-hard-choices/
Richard M: Berger is now simply stating what was obvious from the very beginning. Boeing committed none of its own funds to development. It also did practically no real development work, other some pr stuff such as showing off a fake capsule interior, until it actually won the contract from NASA. Then it appeared throughout development that the company committed as little of its own funding as necessary in order to meet milestones and get payments from NASA.
In other words, from day one Boeing management never understood this was not a cost-plus contract. They treated it as such, and because NASA was not going to fork over more funds when the company faced difficulties, management instead allowed the project to suffer.
I commented on this repeatedly throughout the development phase.
Let me provide some actual data, this 2019 post:
Inspector general slams NASA’s management for bonus payments to Boeing
The IG report noted that Boeing threatened to delay Starliner for 18 months if it didn’t get a bonus payment above its fixed price contract.
Very clearly the company did not understand what a fixed price contract was, and managed the project badly from the start, thinking NASA would pick up the slack. Eventually, NASA did so in this case, but this bonus was hardly enough to fix the situation, especially as the company still did not understand it wasn’t going to get more.
The result was that management did not invest the capital required to develop Starliner properly, and the result has been a disaster for the company.
See also this 2016 post. I wrote then:
Unfortunately, Boeing’s culture never changed, at all.
Hello Bob,
In other words, from day one Boeing management never understood this was not a cost-plus contract. They treated it as such, and because NASA was not going to fork over more funds when the company faced difficulties, management instead allowed the project to suffer.
Oh, I agree. You are correct, of course (as you so often are!).
And yet, as Berger notes, Boeing made a great show of saying, from the outset, that they WERE going to invest their own resources into Starliner: “Boeing has a lot to offer NASA in this new field of commercial crew transportation services,” Keith Reiley, then the Boeing program manager for the project, said at the time. “To show our commitment, we are willing to make a substantial investment in research and development.”
In short, they simply lied. Publicly. Repeatedly.
But then, as we know from the disaster that is called the 737 MAX, we know that this is, unfortunately, not a new phenomenon for the Boeing of the 21st century. This is not, alas, the Boeing that built the B-29, the 707, or the Saturn V.
NASA cannot be fully absolved here, either. For too long, senior NASA management trusted to the Boeing legacy – trusted that they could get the job done, and *would* get the job done. And forked over more money (as you note) when Boeing came hat in hand to demand more than their firm fixed cost contract allowed. But the warning signs had been there all along, as the NASA engineers who actually had to work with Boeing on Starliner development were seeing on a daily basis. Karen Bernstein, who was a mid-level manager for NASA’s Commercial Crew division, had a whole Twitter thread rant last year on her frustrations working with Boeing. “Boeing did not allow collaboration in the same way [as SpaceX]. I was the spacecraft struc mech system manager on the nasa side. These 2 companies had very different relationships with my team. …Boeing believed it had all the answers, did not appear to understand the nature of a fixed price contract, and treated my team like people they hardly needed. SpaceX welcomed wisdom and guidance and fully partnered with our engineers. To be clear, by “Boeing” I mean mgmt. the engineers we worked with were just as frustrated as we were.”
https://twitter.com/KarenSBernstein/status/1512457903696982025
We see the results of how Boeing management ran this program today.
I’ve been wondering over the last day whether NASA would allow Boeing to just spin the whole program off and sell it to, say, Blue Origin, who could actually have a vested interest in using Starliner in conjunction with Orbital Reef and Blue Moon, and making a success of it. Unlikely, but . . . that may well be the only way to salvage this thing, I fear.
judd
The problem I have found with that spray on coating your talking about it the single wire replacement job. You would have to throw the whole of the wiring harness out if just one wire failed.
And in the field the mechanic can replace a single wire pretty quick if not just add one and kill the bad one.
On a one use space craft good tape is enough as long as heat problems are dealt with also.
P.S..
Very clearly the company did not understand what a fixed price contract was
Oh, I think they understood what it was quite well! But they clearly thought they could get away with acting as if it was not a fixed price contract. because they were BOEING.
At first, they lobbied hard to get the CCtCap contract as sole provider (and almost succeeded, as Lori Garver tells it), after which they would eventually force NASA to restructure it into a more traditional cost-plus contract. They’d have NASA over a barrel, after all. And they had friends in Congress.
After SpaceX got the other CCtCap contract, they seem to have thought SpaceX would stumble and fail, and they could force NASA to restructure it into a more traditional cost-plus contract. They’d have NASA over a barrel, after all. And they had friends in Congress.
When SpaceX did not stumble, and they could see they were running over budget, they simply tried to pry more money out of NASA anyway. Because it had always worked before. They no longer had NASA over a barrel, but they had friends in Congress.
But since 2020, they’ve run out of leverage. And now they’re spending the hundreds of millions fixing problems that they could have avoided had they spent the money up front, at the outset. And as a result, they’ve lost out on hundreds of millions in additional Commercial Crew missions that have all gone to SpaceX instead!
Richard M: You are correct in all points. The lying by Boeing management was also self-evident, as was the willingness of NASA officials to look the other way. I think this changed radically when Kathy Lueders took over the NASA supervisory role. She very clearly accepted no prevarications from Boeing, which I think explains why both NASA and Boeing began to finally find and fix problems.
I think her influence at NASA, at least for awhile, will be beneficial, even if she is no longer there.
The parachute harness should be simple to fix.
The wiring harness may involve a tear down of the capsule to fix.
Nobody mentioned another stuck valve that has never been fixed. I believe they just replaced them with new valves. If it takes 1 year to fix the wiring harness, those already unreliable valves will be getting older, not better.
At this point it might be more cost effective for Boeing to sell the 6 Atlases to ULA or Amazon and contract w/ SpaceX for the 6 crew missions if this would be permitted contractually. :-)
pzatchok wrote: “To not blame the supplier is suspicious.”
This assumes that the drawing did not call out the wrong tape. If the designer does not pay enough attention to the materials used, then it is not the vendor’s fault that he shipped the tape that the customer ordered.
Material selection is one of the important design-engineering functions.
Richard M wrote: “And Boeing dug this hole all by itself.”
Apparently with a backhoe.
Robert Zimmerman wrote: “… which I think explains why both NASA and Boeing began to finally find and fix problems.”
It is too bad that Boeing did such a poor job of design and execution. They should be far more embarrassed than they seem. They were supposed to be the grownups showing the kids how to do it right, but they got cocky and the kids now look like the experts who should be leading the future.
It also looks like that is how it will all shake out. The heritage companies are looking like the buggy whip makers, unable to adapt to the new conditions.
The kids already seem to be the ones running the show, as they are the team leaders for lunar landers, space stations, satellite constellations, space manufacturing, and other important future space industries.
Mark wrote: “At this point it might be more cost effective for Boeing to sell the 6 Atlases to ULA or Amazon and contract w/ SpaceX for the 6 crew missions if this would be permitted contractually. :-)”
Well, it certainly violates the spirit of the contract. ;-)
Rather than having the two different spacecraft from two different companies, we would only have one and no backup. But then, that is what we already have, so maybe that is the solution after all. What a good thing it is that we had SpaceX to back up Boeing on the Commercial Crew Program. Otherwise we would have been in the sewer creek without a canoe to paddle.
Perplexing how the last American company that sells commercial airliners—presumably via fixed-price contracts—is utterly incapable of delivering the same way on the spacecraft side of the company. I suspect it has a lot to do with the size of the production run. It must just come down to a lack of experience producing spacecraft, but a total unwillingness to admit that along with pride and arrogance from ancient accomplishments of its legacy companies (MacDac, North American). SpaceX obviously had no experience either, but were open to all the mentoring they could get from NASA.
Concerned: See this interview of Elon Musk by Tim Dodd:
Musk on Starship, engineering, and management
See especially point #4. To quote Musk again:
Edward
If Boeing was using a single engineer to approve the tape/wiring harness specs then they are idiots. On a project this size you would think it would at least be a two or even three man team who finally approves all material purchases.
You have a thousand engineers working for you and you only assign one to the approval job?
NASA and the legacy contractors are chock full of self indulgent engineers who pretty much spend their whole career thinking up new personal projects then finding the contract they can squeeze them into.
This is like Germany’s aircraft industry during WWII. They had thousands of engineers and workers working on new special projects that led to nothing that helped end the war in Germany’s favor.
I think at one point they had 100,000 people working for a year at least on the jet aircraft project and it pretty much led to nothing for Germany.
They even went through the motions of launching their “best and newest” U-boat in front of Hitler and then had to bring it back into dry dock to finish it and in fact it never got finished.
That sounds familiar somehow.
Work like this only gets better in North Korea.
The Soviets had a few(?) idiot moves also.
https://oldmachinepress.com/2019/11/05/alexeyev-km-ekranoplan-caspian-sea-monster/
The intent of this thing was to sneak into American water at high speed with a thousand troops and equipment at a time. They even wanted to ship tanks wit it. And it could have worked if not for the extremely narrow flight parameters. It could not fly in high seas nor take quick turns without dipping a wingtip into the water.
Some people are thinking of bringing them back as smaller passenger liners but they all have the same problem. They are extremely loud and use a lot of fuel. Plus they still have the turn radius problems and would require a set sea lane to stay clear for them. The height off of the water is directly proportional to the size of the craft. The smaller the craft the closer to the water it rides. A large hydro foil would be better.
Here is a short list of a few other aircraft failures.
And if you notice the older ones are all private constructions and new ones are ALL government contracts. Half or more by NASA.
As a nation we were lucky we could afford so many failures, not like the Soviets who tried to keep up but failed because they had no economy to keep up with the spending.
https://www.pilotmall.com/blogs/news/20-of-the-weirdest-planes-ever-to-take-flight
Sorry about that.
Here is what gets me Pzatchok.
The Soviet Vostok and Dragon are actually closely alike in that no one ever “flew” any of them.
Starliner is supposed to be what pilots like—all analog—-the way God and Chuck Yeager intended in THE RIGHT STUFF.