Boeing to return Starliner to factory
Capitalism in space: According to a Wall Street Journal story today, Boeing and NASA have decided to remove the Starliner capsule from the Atlas-5 rocket and return it to Boeing’s factory in order to do a more thorough inverstigation into the capsule’s failing valves.
This decision means that the launch of the second unmanned demo test flight of Starliner will not occur in August, and will likely be delayed several more months. NASA and Boeing just held a press conference in which they made this decision official. During that conference they said they think the moist environment at Kennedy might have caused corrosion in the valves, which caused them to stick.
I once again wonder if Boeing has any quality control systems at all. For such a serious problem — the failure of 13 valves out of 24 — to suddenly pop up just hours before launch, when they have been developing this capsule for years, and even had an extra year and a half to check the capsule out after the failures during the first unmanned demo flight in December 2019, is somewhat astonishing, and very disturbing.
Others will argue that problems like this can always appear unexpectedly in space hardware. I say hogwash. Boeing is not inventing something new with Starliner. This is a capsule, using heritage engineering first invented in the late 1950s. It should not be so hard to get this right.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
Capitalism in space: According to a Wall Street Journal story today, Boeing and NASA have decided to remove the Starliner capsule from the Atlas-5 rocket and return it to Boeing’s factory in order to do a more thorough inverstigation into the capsule’s failing valves.
This decision means that the launch of the second unmanned demo test flight of Starliner will not occur in August, and will likely be delayed several more months. NASA and Boeing just held a press conference in which they made this decision official. During that conference they said they think the moist environment at Kennedy might have caused corrosion in the valves, which caused them to stick.
I once again wonder if Boeing has any quality control systems at all. For such a serious problem — the failure of 13 valves out of 24 — to suddenly pop up just hours before launch, when they have been developing this capsule for years, and even had an extra year and a half to check the capsule out after the failures during the first unmanned demo flight in December 2019, is somewhat astonishing, and very disturbing.
Others will argue that problems like this can always appear unexpectedly in space hardware. I say hogwash. Boeing is not inventing something new with Starliner. This is a capsule, using heritage engineering first invented in the late 1950s. It should not be so hard to get this right.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
Good thing there are lots of low humidity environments for launching. Guess someone should have read those specs before sending it to Florida. ;)
This is something new…for them. Say what you will about Mike Griffin….he got a rump Ares IX flying-good bad or indifferent. That’s the power of arsenal method. It’s what got Explorer and Redstone off the ground-in spite of for profit dullards. Outside of Dragon….and Orion…we are having to rebuild everything. Now, I think Saturn IB and Apollo should have stayed in production-shuttle or no-and upgraded. NASA centers should in fact be designing rockets and spacecraft to keep it in house-to preserve tribal knowledge and morale-and to keep contractors on the short leash so if the company goes under the tech doesn’t.
Now I know you aren’t going to like me saying this, but it was Boeing’s lust for profits that put them in this fix. SpaceX is de facto arsenal method with Musk a de facto Soviet Chief Designer with absolute authority to bark orders. Engineers at Boeing were bullied by corporate….walked out…and were replaced by kids who are earnestly trying their best, but are out of their depth. I blame libertarianism. Imagine Boortz shuts down weather radars, then calls the NWS fools when folks die by the score from lack of warning. Outside of SpaceX, ALL manned work needs to be relocated to MSFC and kept just as far from corporations as possible.
Jeff Wright wrote, “Outside of SpaceX, ALL manned work needs to be relocated to MSFC and kept just as far from corporations as possible.”
The totalitarian government worker speaks! All must obey.
Mr. Wright fails to mention that since it built the Saturn-5, the Marshall Space Flight Center (Mr. Wright won’t like this either) has built practically nothing and spent billions doing it. From 1985 through 2010 Marshall was named the lead center on 22 different shuttle replacement projects costing a total of just under $25 billion.
None ever flew. None in fact were ever completed.
Though the fault was not entirely Marshall’s, as corrupt politicians in Washington interfered repeatedly, the center’s managers and engineers over that quarter century must share some of the blame. They simply couldn’t get anything built at a reasonable cost.
Yup, let’s mandate by government fiat that all our rocket industry but SpaceX be absorbed by this relatively useless NASA center whose funding was protected for decades by porkmeister Senator Richard Shelby (first a D then a R-Alabama). Isn’t that what limited government and freedom is all about?
Note that my data was very carefully documented by a NASA manager at a different agency, using publicly available information, and sent to me in a detailed spreadsheet shortly after my 2016 Capitalism in Space policy paper was published. He thought I would find the data useful.
Robert wrote: “During that conference they said they think the moist environment at Kennedy might have caused corrosion in the valves, which caused them to stick.”
This is exactly why flight hardware is so carefully stored in environmentally controlled cleanrooms. This is exactly why so many space engineers are so appalled that SpaceX stares so many flight parts outdoors just a mile from an ocean beach.
Environmentally controlled cleanrooms are supposed to prevent moisture condensation on flight parts, preventing corrosion, as is suspected, and preventing other problems. Supposedly, Starliner has been kept in clean environments until it was moved to the pad.
This reinforces my question in another thread: what was different between the last Starliner launch and this one?
The description of the problem is that they see permeation of the oxidizer through some of the seals in the valve. An interaction with moisture on the “dry side” of the valve created some nitric acid which resulted in some corrosion which resulted in stiction of the valves. They are still looking at other items on the fault tree, but the report was that the permeation of the oxidizer is their leading candidate for the cause of the issues with the valves.
The valves, as on the first launch, had been tested at the time of propellant loading, five weeks before rollout to the pad. This helps point to the permeation of the oxidizer as a cause of the problem, but it was also reported that this kind of permeation is not unexpected for the valve design, which points to the moisture as closer to the root cause.
“Others will argue that problems like this can always appear unexpectedly in space hardware. I say hogwash. Boeing is not inventing something new with Starliner. This is a capsule, using heritage engineering first invented in the late 1950s. It should not be so hard to get this right.”
Robert has a point, that we have been doing these kinds of things for spaceflight since WWII. We learned a lot about the kinds of problems that can arise, and we have learned a lot from aircraft development and operations. However, we do not yet have complete knowledge of these things. The fact that this problem occurred on this second flight but not on the first flight shows that our understanding of our spaceflight hardware is not as complete as we need it to be.
It is because of this lack of understanding that we expect to lose crews in spaceflight. We just cannot learn all that we need to know without being brave and exploring where no one has explored before with new hardware and new designs that we think will work properly.
As with the halt to this launch, we do not do this lightly. We really do try to make it as safe as possible, but we just do not yet know where all the traps are. Because we didn’t know it was there, and because we had been this way before, this is a trap that we sprang unexpectedly. And this is why we fly unmanned the first time.
Boeing is getting more scrutiny than it may have received otherwise, because it has had noteworthy problems. Is this scrutiny unfair? Not really. Having a series of problems can suggest some form of systemic problem within the company. It is similar to Russia’s string of major problems over the past two decades or so. We have doubts about Russia’s quality control, and we are forming similar doubts for Boeing. The good news on this particular problem is that it was difficult to anticipate due to the success of this part of the spacecraft on the previous flight. Having caught it before launch rather than afterward also gives Boeing’s quality control program credit for being vigilant. However, they are having a series of problems, and other recently designed commercial spacecraft have not had such problems, even on their early flights to ISS. We have cause for concern, but not yet cause for alarm.
I have worked on several spacecraft and instruments that had problems during development or during integration and test, but so far all the spacecraft worked fine, and the only instruments that didn’t get to perform their functions were lost when the spacecraft was lost (one during the Ariane 501 launch, and another when the spacecraft lost power due to solar array issues just after launch — incidentally, I only worked on the bid-and-proposals for those two instruments).
Problems can be worked out and the spacecraft can do yeoman’s work. I am not worried about Starliner.
“Outside of SpaceX, ALL manned work needs to be relocated to MSFC and kept just as far from corporations as possible. ”
I disagree. Some corporations may still be seeing NASA and government space as a source of cost-plus income, but Boeing has seemed to try to do Starliner better and faster. Until real problems cropped up, they were on schedule. Space companies may still need to take lessons from the rest of the free-market world of commerce, similar to rocket Lab’s and SpaceX’s approach, but I think that they will get there or perish to the new space companies, the ones that get their products to a free market that needs those products.
NASA and government will not always be the monopsony that it has been in the past, and to keep it as the virtual monopoly that it has been until recently is a move in the wrong direction. New customers for space products are emerging every month, making for new markets for our new space companies to move into. Government space has not been willing to support these markets in the past and I see no indication that government space would support them in the future. But our new space companies are eager and willing to do all they can to support these new markets and niches.
Boeing’s profit motive isn’t the issue, it’s their business plan for getting the profits. Since NASA will continue to pay them while they are “developing” Starliner, they have no real incentive (beyond PR) to actually put anything in space. So they can take a year to replace a power system, or months to check the valves and be sure they won’t lose any money on the deal.
Marshall’s calling was big rocketry, Robert…Venture Star/SLI the only stain. I am thankful for Shelby ’cause the rest of NASA wants us dead. We’re too Red State for the Yankees by way of Johnson. Did I ever hate Abbey…
Now Darleen Druyen shows the bad about deadwood that survives multiple administrations…but an indepedant NASA that survives different Presidencies wouldn’t have this overturn.
Keeping things in house works. Area 5I proves that:-)
I am curious about the need for environmentally controlled cleanrooms to build reusable space ships. What other transportation system is so fragile, not aircraft, ships, submarines? Can we set up clean rooms on the dusty moon and on Mars to get the space ship back home.
Obviously this is an unnecessary and impossible standard. Spacex is building a durable rocket to use and reuse. In rain, dust, micrometeors and other environmental hazards.
Jeff Wright,
A proper reply would violate Roberts’ comment rules. Did you read and comprehend anything he wrote???
Damn, just commented on this before I saw this post!
William, exactly. No clean rooms on the Moon or Mars.
John, I always read Robert’s posts. Huntsville has a great history-but too many enemies.
A thought struck me: we have an Atlas-Centaur out there going to waste. Are their any payloads out there languishing for a ride that isn’t too particular on timing? Boeing needs damage control pronto…and a freebe might stop the bleeding. Maybe a suggest a payload contest :-P
Jeff,
I read, no more Commie RD-180’s by the end of this year.
What’s the Atlas gonna do?
The root cause is using “other people’s money” and cost-plus contracts. Boeing Space is a corporate-welfare recipient performing exactly as you would expect a welfare recipient to perform.
I do understand NASA isn’t paying Boeing for this OFT-2 flight and probably demanded Boeing remove their Starliner garbage from NASA’s launchpad. That’s the only way to improve Boeing’s performance. Stop paying for incompetence.
Considering the incestuous relationship between Boeing and D.C. I’m surprised this “tough love” approach is happening, but it’s well overdue.
I think they have a stockpile of them as per NSF…and the blueprints. Oh, the scuttlebutt is that Marshall almost got Energiyas channel-wall RD-0120 hydrolox equivalent of RS-25/SSME. RS-68 has much fewer parts. Some want SLS called Jupiter IV….lets see…a recent issue of the Journal of Spacecraft and Rockets had some good entries…and the Secret Projects Forum had an entry on the huge German NEPTUN.
The rd-180 stockpile is VERY small. As in a handful of flights.
And is we had blue prints for them wouldn’t we already be making our own?
Its against US law to steal blue prints and make copies of stuff other nations hold the patent on. Especially nations and companies we want to stay friendly with. Like Russia.
Ask yourself this.
Why is not a single company in America trying to buy the manufacturing license for those engines? Not Blue Origin not Boeing no one.
Ok I will tell you why.
They would make far far far more money designing and making their own under cost plus contracts.
Posted in error to above thread:
Just listened to a recording of a NASA / Boeing news conference on the latest Starliner delay. Bottom line, the oxidizer valve (to the thruster systems) seals leak enough to allow creation of Nitric acid if there is any moisture on the “dry” side of the valve, which in turn creates corrosion which can impede movement of the valve. Even though there was a rainstorm before the day of intended launch, and thruster covers leaked water, they don’t think this is where the moisture came from! So where? The atmosphere! Remind me how long spacecraft have been launching from Florida? And I believe it sometimes rains there, no?
And the Boeing chief engineer repeatedly said they would have to take the craft “back to the factory” to take it apart! Finally a correspondent rescued them by asking “Where is the factory?” Turns out he meant the old Shuttle processing facility at the cape! These people are so isolated from reality they can’t communicate with normal human beings.
Bottom line: this thing won’t fly this year. I wonder if the former CEO of Boeing is still taking bets that “The first humans on Mars will arrive on a Boeing rocket!”?
Can someone reassure me, that the weapons Boeing manufactures for our military, will actually work, when the War starts?
I am sure the B-52 will work. They have had 60-odd years to work out the bugs. These things take time.
Say what you will about Mike Griffin….he got a rump Ares IX flying-good bad or indifferent.
I have a lot to say about Mike Griffin, but . . . that’s for another day.
But it’s instructive to think about that Ares IX test flight. That test cost about $400 million – I mean, just the cost to do the test. It cost a lot more than that for the development costs to date at that point (2009). Meanwhile, at around the same time, NASA paid $396 million to SpaceX for the initial Commercial Cargo award. But that money was not earmarked just for the rocket; it also paid for development of the Cargo Dragon spacecraft and a launch pad in Florida, too.
So, for the same amount of money it cost to test launch that Ares IX rocket, NASA got development of an entire medium class launch vehicle, an orbital cargo vehicle, and rebuild of a launc pad for all of it, out of SpaceX.
And that test launch was indeed (as you say) a true “rump.” It didn’t have a fifth segment for the first stage SRB, and the entire second stage was a dummy.
“The rocket that thundered aloft from NASA’s Launch Pad 39B sure looked like an Ares I,” Apollo 11 astronaut Buzz Aldrin said at the time. “But that’s where the resemblance stops. Turns out the solid booster was—literally—bought from the Space Shuttle program, since a five-segment booster being designed for Ares wasn’t ready. So they put a fake can on top of the four-segmented motor to look like the real thing.”
William,
You wrote: “I am curious about the need for environmentally controlled cleanrooms to build reusable space ships.”
Well, we just learned that a little moisture resulted in corrosion, the scrubbing of a mission, the removal and disassembly of the vehicle, and who-knows-how-much direct cost for this effort as well as reduced revenue to the company due to the lost opportunity from delayed operations. Since some payments are made when missions occur, there is delayed revenue. Oh, and a reputation that needs repair, too. If this and other problems can be avoided by use of cleanrooms, then they are very useful indeed.
On the other hand, how reusable are the test units that are being built at SpaceX’s virtually-outdoor facility?
Do you remember when a Falcon 9 launch had to be scrubbed because some of the engines had a little bit of contamination? Have you taken a tour of Boeing’s airplane factory in Washington state (yes, they are available)? Most of that building also has a nicely controlled environment. The paint room has a large door to the outside, so the environment is not so controlled for a few hours after a plane rolls out. Once outdoors, however, the plane is protected from most corrosive influences and from other problems of the real world. Generally, this is true for manned spacecraft as well (unmanned spacecraft often remain inside somewhat environmentally controlled fairings until reaching space).
So, why are airplanes and spacecraft so fragile? Because we have lightened them up as much as we can so that they can get into the air or into space, so a little corrosion or damage is far, far more critical than a little corrosion on a ship or a submarine. I cannot tell you how many times I’ve had to have holes fixed in a satellite’s thin composite intercostal panels due to someone being careless with diagonal cutters while tying down cable harnesses (or some other activity). These panels are strong and can handle quite a load, but they are made of amazingly thin and fragile — and lightweight — sheets.
Ships and submarines do not have nearly the same amount of weight sensitivity. Battleships had literally tons of extra armor added around their waterlines in order to protect against torpedos.
If you listen to the press conference, somewhere around the 20 minute mark someone explains that the storm had caused problems with the thruster covers. Thrusters are also susceptible to problems with corrosion and contamination, so they tend to be covered before launch.
Getting these things to work on Mars and the Moon is a challenge. It is why it is rocket science, not some sort of mass production line.
It took a long time to figure out aircraft, and their jet engines are still susceptible to problems from dust. The airframes and even tires are easily damaged from foreign objects and debris (FOD), as we all should have learned from the Concord crash a couple of decades ago.
I wouldn’t worry about cleanrooms on Mars or the Moon. There were few on Earth until we built them. I’m pretty sure the technology would be similar on those two worlds, too.
Oh, speaking of contamination, I have had problems with flight connectors and test connectors due to a little contamination between the pins. We inspect for this kind of contamination and clean it out when we find it, but my technicians have sometimes had difficulty seeing some of the contamination. One place I worked tried to reduce this problem by having a quality assurance inspector also inspect the connectors prior to mating. Where the hell did that contamination come from? We’re in a freaking cleanroom for heaven’s sake! And we inspected that same connector when we demated it after the last test, and we covered it with a clean, protective cover. Where the hell did that stuff come from?
I swear, Murphy was a flaming optimist. Even if it cannot possibly go wrong, it will.
Robert says hogwash to suggestions “that problems like this can always appear unexpectedly in space hardware.” It turns out that the darnedest things happen despite all the experience we have mating connectors. These problems bleed into the areas that we are less familiar with, too.
But to expand on the concept Robert brings up, this isn’t just space hardware, it is a development project. It is in verification mode, right now, but it is still development. That is why we aren’t flying astronauts onboard.
From the article Robert linked to in his post, Update on SLS: launch prep continues, launch in 2021 remains doubtful:
https://www.nasaspaceflight.com/2021/08/artemis-1-power-up/2/
If we were confident in our new space designs, then we wouldn’t be flying them unmanned on their first missions. Instead, we realize the limitations that we have in our knowledge of space hardware. Unlike aircraft, in which there were literally thousands of different designs over the course of the first seventy-five years of flight, manned spacecraft have been limited to a dozen or so different designs that have flown only a few hundred times. That isn’t a whole lot of variety or experience in which to discover what works best.
My last post was a response to Jeffrey Wright – that may not have been clear because I just learned the hard way that the combox won’t take boldface tags.
Chris Lopes:
Since NASA will continue to pay them while they are “developing” Starliner, they have no real incentive (beyond PR) to actually put anything in space.
Actually, they do: The CCtCap contracts are fixed cost, and payments are only made when milestones and flights are made. Boeing won’t get some of that contract money until it successfully flies OFT, CFT, and the initial operational flights.
Meanwhile, Boeing is having to pay for this re-do of the OFT flight, and all extra costs entailed with fixing these valves in the coming months. This is not a happy development for Boeing or its bottom line.
Richard M: That Ares test launch was even more fake than you describe. NASA touted it as testing the heat shield for Orion, but when this test was launched NASA had already decided that it was not going to use that heat shield on Orion.
In other words, they weren’t even testing anything they’d eventually use.
Hello Dave,
I do understand NASA isn’t paying Boeing for this OFT-2 flight and probably demanded Boeing remove their Starliner garbage from NASA’s launchpad. That’s the only way to improve Boeing’s performance. Stop paying for incompetence.
ULA needs the SLC-41 launch pad and vertical integration tower for NASA’s Lucy mission, launching in October. So ULA has no choice but to unstack the Atlas 422 that Starliner is sitting on . . . and stow away the stages for the time being while it sets up Lucy’s rocket.
Hi Bob,
In other words, they weren’t even testing anything they’d eventually use.
Indeed! The entire test was really just a PR stunt, to give the illusion of development progress on Ares I. What technical value did NASA really get out of it?
Buzz understood. Not many in Congress or the media did.
Robert,
You have confused two different launches. The Ares test Richard M referenced was for the Constellation project in 2009, after Obama announced Constellation would be cancelled, and it didn’t go to orbit. The bogus heat shield test was for what we now call Artemis.
Edward: Hm. Maybe. The launch I am referring to was not for Artemis, however. It occurred in 2014, and was launched during the battles between Congress and Obama over the cancellation of Constellation.
Robert,
That is the one. It was for the Orion heat shield. It was about the time that Obama wanted to send Orion to an asteroid, which mutated into a mission to bring an asteroid to lunar orbit or an Earth orbit about the same distance as the Moon. Then it morphed into bringing a rock from an asteroid back, all to give Orion and SLS something to do. Gateway was being conceived around the same time, too.
It was about the time that Paul Spudis wrote in his book, The Value of the Moon, “Regrettably, strategic confusion currently abounds in the American civil space program.” Eeric Burger noted that NASA had been set adrift:
http://www.houstonchronicle.com/nasa/adrift/1/
Obama and Congress turned NASA into a cluster.
I find it interesting that Elon has essentially bought himself a lot of mass margin for dealing with ruggedization issues by going big, just as the BDB (Big Dumb Booster) proponents have suggested for decades. However, by crossing the BDB stream with the fully-reusable stream, he’s opened up the potential to have his cake and eat it too. Zany ideas like turning every launch tower into a giant robot just so that he can omit the landing gear from the (Earth-only) booster, or deleting the mechanism to extend and retract the grid fins and just letting them drag on ascent, have padded that margin. And when you have margin, you can afford to spend some of it on things like dealing with the wet, salty air near a beach.
Somebody else needs to start thinking big-and-fully-reusable. Elon needs competition, needs other competent and creative thinkers to try other ideas, or he’s going to wind up with a de facto monopoly, and all of the temptations that come with one. His focus on “unprofitable” Martian colonization might kick that can down the road for a time, but eventually it would come back to bite us as his successors turn their focus more towards maximizing short-term profits.
Edward
Thank you for the thoughtful response to my question. I still don’t understand how Falcon 9 first stage can fly 10 times (or 100 times) Do they replace every valve and widget? The Boeing commercial fleet has flown for decades in rain, snow, and dust (not volcanic ash) all without catastrophic failure. Military planes and helicopters are also flown in extreme adverse environments. Take the A10 warthog, an extremely tough and durable airplane able to fly after being struck by enemy munitions.
Maybe rocket engineering needs to step up and get rid of the parts that are made like fine crystal stemware?
One thing I’m curious about…
Say the Nauka launch and docking had been flawless. Starliner was ready to go at the time, weather okay, so a launch was probable. Then the valves get stuck *in orbit* and the capsule is lost.
What happens then? Boeing has to do a third qualification flight? Or Starliner is cancelled altogether? Or, more probably, Boeing cries to Congress (“space is hard!”) and gets another cash envelope?
V-man, my understanding is that the valves would have been tested prior to launch, and failed or not. Where did the moisture come from that caused them to corrode? Nobody knows yet, but I think there is a lack of moisture in space, so if it passed the ground test it should have been okay, assuming the test was conducted close to launch. I know that is a bit circular, but that seems to be what Boeing is saying.
Willam: Y’know, there are valves I’m sure in the fuel lines on every airplane ever flown, as well as in all cars. They don’t leak or get corroded easily.
What can damage them is non-use, which might be what happened with Starliner. Starliner sat for years waiting for this flight. Sitting without much exercise the nitric acide they theorized caused the corrosion had time to work its evil. Using the valves regularly would wash it out.
SpaceX solves this problem by treating its stages like airplanes. They only make money on them when they are in the air, so they want to keep them flying.
The company also appears to have a very robust quality control system. If something fails, they fix it, but they fix it so the problem never reappears again.
Think back to early 2020. SpaceX had a string of launch aborts. Elon Musk called a meeting demanding that this be fixed. Whatever they did, they have been launching practically weekly since with no abort.
To me this whole excuse sounds like bolder dash.
Those valves are INSIDE the rocket. They are protected from outside air changes by that first.
Second the rocket is inside a climate controlled assembly building. This is to help keep moister from forming from rapid temperature changes.
Third the gaskets are INSIDE the valves. Which is an sealed and protected system.
Now was there oxidizer loaded into the system at anytime since the system was installed. If so why was the system not flushed and cleaned after that test filling?
I bet the gasket just naturally broke down after sitting for YEARS on the shelf and or installed. I never trust a rubber product after 10 years.
Especially a valve gasket. After a few years of sitting in one place they tend to take a set shape and when finally used and moved do not reshape to make a good seal. By the way O-rings are the worst.
And if this was a solid bronze valve with no gasket material needed you only have to nitride it to stop oxidation.
Several of my guns are nitrided to stop the steel the oxidizing. It works great.
Robert
I completely understand your point. I believe spacex is building the equivalent of the A10 for reliability in a frequent duty cycle under adverse conditions.
I would definitely fly in a crew rated spacex ship. If only I wasn’t too old by the time they are ready. I do remember the first moon landings televised to my first grade class. Back then I thought I would go to the moon after college. Maybe my granddaughter will go.
Willam: Ditto as for my expectations for the future, c1969.
The mistake our generation made was buying into the Soviet-style government “space program” approach. There are signs that this is now finally being abandoned, and thus we have real achievement in space.
No more “programs!” Let’s have a chaotic, free, space industry competing to provide space services to anyone who wants it, however they want it.
It is worth remembering that Spacex had a valve problem of its own. The first Dragon that docked unmanned with the ISS subsequently exploded during a ground test! IIRC the problem was a leak between two lines that contained the two components of hypergolic thruster fuel. Catastrophic failure! I have never read a thorough root cause analysis and resolution of that one!
Robert
I love your discussions with John Batchelor. I hope you highlight the cleanroom paradigm as an example of why old space is so slow and so expensive.
This all sounds like something a few silica gel packs could have prevented ;-)
Gee, I take a couple of days off to get something done and come back to find this thread. Although, truth be told, it’s not nearly as bad as some out there on the internet. Surprisingly friendly, actually.
I once again wonder if Boeing has any quality control systems at all. For such a serious problem — the failure of 13 valves out of 24 — to suddenly pop up just hours before launch, when they have been developing this capsule for years, and even had an extra year and a half to check the capsule out after the failures during the first unmanned demo flight in December 2019, is somewhat astonishing, and very disturbing.
My understanding is that the valves worked during spacecraft fueling a few weeks before the launch attempt. Whatever caused them to fail happened since then.
Others will argue that problems like this can always appear unexpectedly in space hardware. I say hogwash. Boeing is not inventing something new with Starliner. This is a capsule, using heritage engineering first invented in the late 1950s. It should not be so hard to get this right.
You’re right. Boeing is not inventing new valve technology for the Starliner. They’re using the same type of valves used in every other manned spacecraft out there, including Dragon and Orion. And like Dragon and Orion, Starliner has environmental seals protecting the propulsion system from humidity before launch. It is currently thought that those seals were damaged by a thunderstorm the day before the launch attempt, though the jury is still out on that. I expect more data will be forthcoming in the coming weeks.
Boeing’s profit motive isn’t the issue, it’s their business plan for getting the profits. Since NASA will continue to pay them while they are “developing” Starliner, they have no real incentive (beyond PR) to actually put anything in space. So they can take a year to replace a power system, or months to check the valves and be sure they won’t lose any money on the deal.
1) What power system are you talking about? 2) They’ve already lost a great deal of money on the deal — $410 million — because NASA is not continuing to pay them. The Commercial Crew contract is firm, fixed-price.
The Boeing commercial fleet has flown for decades in rain, snow, and dust (not volcanic ash) all without catastrophic failure. Military planes and helicopters are also flown in extreme adverse environments.
1) Starliner has not had a catastrophic failure. 2) The Boeing commercial airplanes, military planes, and helicopters don’t use nitrogen tetroxide as oxidizer.
I am curious about the need for environmentally controlled cleanrooms to build reusable space ships. What other transportation system is so fragile, not aircraft, ships, submarines?
Aircraft are built in similar environmentally controlled facilities as Starliner.
The root cause is using “other people’s money” and cost-plus contracts. Boeing Space is a corporate-welfare recipient performing exactly as you would expect a welfare recipient to perform.
Most of Boeing’s major government contracts are firm, fixed-price. They are the only major prime contractor that signs significant firm, fixed-price development contracts.
The rd-180 stockpile is VERY small. As in a handful of flights.
If by “handful” you mean 30, you’re right. If you mean less than 30, you’re wrong, because there are 30 Atlas V flights left on the manifest.
And is we had blue prints for them wouldn’t we already be making our own?
Pratt & Whitney already has the blueprints for the RD-180 and a license to manufacture them through 2022.
Why is not a single company in America trying to buy the manufacturing license for those engines?
What would be the point? They can’t be used for national security launches, so every launch provider who could use them would need an alternate engine anyway. Much easier to just use that alternate engine for everything.
And the Boeing chief engineer repeatedly said they would have to take the craft “back to the factory” to take it apart! Finally a correspondent rescued them by asking “Where is the factory?” Turns out he meant the old Shuttle processing facility at the cape! These people are so isolated from reality they can’t communicate with normal human beings.
Huh? What’s your point? The former Orbiter Processing Facility *is* the Starliner factory.
Can someone reassure me, that the weapons Boeing manufactures for our military, will actually work, when the War starts?
Considering that the F-15 Eagle (with a combat kill ratio of 100 to 0), F/A-18 Hornet, AV-8B Harrier, AH-64 Apache, C-17 Globemaster, CH-47 Chinook, A-10 Warthog, AGM-84 Harpoon, SLAM, JDAM, SDB, and FLM are all combat proven and considered the best in their class worldwide, whether the weapons work is not what you should worry about. You should be worried about whether the missions they’re used on are appropriate and, if so, what the rules of engagement are.
Say the Nauka launch and docking had been flawless. Starliner was ready to go at the time, weather okay, so a launch was probable. Then the valves get stuck *in orbit* and the capsule is lost.
Highly unlikely. Having the valves open is one of the launch commit criteria.
I bet the gasket just naturally broke down after sitting for YEARS on the shelf and or installed. I never trust a rubber product after 10 years.
I don’t think rubber would react well to nitrogen tetroxide. My understanding is that the seal material is teflon.
OK, that’s enough for one comment.
Ares I might have been a dog…but it flew more than New Glenn or Vulcan. OmegA might not be dead yet.
This has all been looked at as Old Space vs New Space…yet both together have been matched by Russia/China…who you might say have together moved into “deep parity.” I don’t know that old or new space can have a victory over the other without it being pyrrhic. I think many here and elsewhere see Musk beating the world, and old space dying off. Or vice versa in Gary Church’s case.
But if I were a chi-com, what would I want to see?
Reagan redistributed more wealth to the third world than any liberal who wanted a modest but honest increase in foreign aid…all by watering down the border and allowing jobs to go overseas…and daring to question others patriotism.
So I want old space and new space to bloody each other…and I’ll let Musk take down my competitor Boeing for me…maybe with a plant of my own to damage valves and make them look even worse. Then…then, when old aerospace jobs are dead I will target Elon. Texas is going purple, and a second coming of Proxmire by way of Austin means “mostly peaceful protests” are headed to Boca Chica. Oh, not now…just wait. I want to see Elon die of a heart attack on the Tesla floor…after we sell enough masks to Americans to make up for the Trump shortfalls…he the only President we didn’t own.
People who want American Old and New Space to work together? They must be silenced! Thankfully the libertarians dominating space websites are doing that for us. They are our best assets and don’t even know it. Sun Rand did more damage to the USA that our little red book ever could. Only dirty hippies who used every four letter word but “work” and “soap” read that. But the Randians get off putting people out of work even more than our Green plants do. We must have the doctors come up with another disease for us. Talk radio didn’t push back as early and as hard on medical restrictions as on environmental ones. Maybe another “Patriot Act” with Fox pushing it after another “Muslim” attack. That always works.
———-signed Xi Jinping
Mkent,
I really am confused about aerospace construction and storage of aircraft and space ships.
1. Do you believe Spacex cannot fly Starship if it is built in the open?
2. Since Starliner was completed and only waiting for launch did it need to get put back in an air-conditioned room to prevent this failure? How long outside is too long?
3. Commercial and military aircraft are not stored in air-conditioned hangars between flights I don’t think?
4. My impression is that aircraft systems tolerate humidity, cold, heat and dust. Why not space craft? Especially if we expect to bring them to the moon and Mars, and reuse them
Ray Van Dune:
“In a written statement, SpaceX said the failure of the check valve — made of titanium — in a high-pressure NTO (nitrogen tetroxide) environment was “sufficient to cause ignition of the check valve and led to an explosion.”
Jul 15, 2019 https://spaceflightnow.com › spacex…
All check valves of this type were made of titanium and the explosion was a shock to NASA. I believe they are now made with stainless steel. And NASA is requiring all to use it now thanks to SpaceX.
mkent said:
>Highly unlikely. Having the valves open is one of the launch commit criteria.
Absolutely, and they did work at the time. They failed a couple of days (weeks?) later. Now, that subsequent failure could have occurred in orbit (even worse, docked at the ISS), hence my question.
(Which no one has answered because we don’t need to – everyone knows that Boeing would have gotten another check. “We need two options! — and never mind that Dreamchaser over there…”)
Ray van Dune, IIRC the Dragon problem was a materials issue which had never been seen before even though use of those materials was standard industry practice. The whole industry learned from that.
Teflon gaskets take a set even worse then rubber.
And the same two questions about those gaskets.
If they knew nitric oxide would form when they were exposed to oxidizer why use them? And exactly how does water get into a sealed closed system inside the rocket and inside a building?
I can’t believe I’m saying this, but…mkent makes some good points.
1) Again, the CCtCap Commercial Crew contract Boeing has with NASA is a fixed cost contract, and it only pays out as milestones and deliverables are met. A fair chunk of that $5.1 billion remains unpaid until Boeing successfully does its flight tests, and then the operational missions stipulated. So: They really do have plenty of incentive to get this thing operational.
2) The more we learn about the valve problem, the more perplexing it becomes – and, probably, in ways that are not as disastrous as the valve problem SpaceX had with Dragon in April 2019. There were no issues with the valves in the failed December 2019 test flight; and a test was done in final prep last month without incident. And yet, somehow, when another check was done the day before launch, 13 of 24 NTO valves would not open. Why? Moisture penetration, says, Boeing; but not from the storm. Odd.
John Bolmer, Boeing VP and Starliner Program Manager said this in the press conference:
“There’s 24 of the oxidizer valves, there’s 24 fuel valves, and then there are 16 helium valves that comprise the system. They are all of the exact same design. The only valves that we saw issues with were with the oxidizer. And that’s due to the phenomena that I described with the permeation of the NTO through that. So we had no issues with any of the fuel valves — the less toxic, corrosive substance. And of course the helium is more inert and we didn’t have any issues with any of the helium valves. Once again, all of the same design. It’s a factor that goes with the oxidizer, and it’s limited to just those oxidizer valves.”
It’s hard to think that some kind of *design* fix isn’t going to be needed to rectify it (or mitigate it sufficiently), once they complete the root cause analysis. Which may be just as well since it’s not apparent it can’t get a launch window until November anyway.
It’s a bad look (and a bad hit to the bottom line, as this rollback and de-stacking will cost the company, not NASA) for Boeing, but perhaps more because they’re *already* so far behind the 8-ball. Were it not for the valve leak problem in the 2018 testing round and the software integration failures that busted the 2019 OFT-1 mission, Starliner would have been flying crew for two years now. This issue may be less culpable when all is said and done, but clearly people are running out of patience with Boeing; even Kathy Lueders was unable to fully hide her frustration.
Richard M: NASA had ran out of patience with Boeing back in the spring of 2020, when they chose SpaceX to provide cargo to Lunar Gateway. In assessing the four bids, Boeing ranked last, and was given this horrible assessment by NASA:
The Starliner contract had been awarded several years earlier. I suspect if that contract was offered now, Boeing would not get it.
Problems can always appear unexpectedly in space hardware, even SpaceX had problems. Why are Boeing’s problems so much more prolific? Is it luck or is there a skill that some companies have that others do not. If it is a skill, why doesn’t Boeing have it after a century of existence and experience but Orbital does have it with three decades of experience, and SpaceX does with only two decades?
How does it go wrong, and how does it go this wrong?
How does it go wrong with any company? There once was a time when, the saying went, no one got fired for buying IBM (either the stock or their products), but when was the last time you heard of IBM outside of historical context? How did they go from being the business machine company to being an unknown (are they still in business)? Did anyone else notice that they disappeared from the Earth just as soon as they introduced their version of the personal computer — the version that the whole world was waiting for so that they could buy it and be guaranteed to not get fired?
Something happened at Boeing, because the company that we thought was so careful is having such a terrible time with quality, and the company that builds its test flight units outdoors in an all-out hurry, thus is not careful at all, has so many successes doing things that no one has ever done before. It is the experienced company that is meticulous vs. the new company that innovates in a rush.
When it came to Commercail Crew Program, the tortoise lost to the hare.
In the super heavy launch arena, the same thing is happening, despite the tortoise having a six year head start. Congress has no urgency to launch SLS, despite the 2024 lunar landing goal, but SpaceX has urgency to get Starship to Mars. The hare has a reason to keep going, but the tortoise took a few years not making much progress, having had year-for-year schedule slips. It was during this time of slipping schedules that Starship was first announced to the world. The hare got into the race while the tortoise slept.
William,
You asked: “I still don’t understand how Falcon 9 first stage can fly 10 times (or 100 times) Do they replace every valve and widget? The Boeing commercial fleet has flown for decades in rain, snow, and dust (not volcanic ash) all without catastrophic failure.”
Robert is correct. Blue Origin, Rocket Lab, and SpaceX are working toward building rockets more like aircraft than the expendable items that rockets have been for seventy-five years. This means that instead of being robust for a few tests and a launch, parts have to be robust for thousands of cycles and possibly thousands of hours of flight time.
This may mean that parts that had been made as light as possible now have to be made somewhat heavier, or other changes may be necessary to reduce the effects of wear. The transition from expendable to reusable could be tricky as everyone learns what works in space for that many cycles. We cannot always use aircraft parts for spacecraft. Some parts are designed to work in air, some require gravity (flying a barrel roll with a 707 was proved possible, but some aircraft don’t do well upside down). The Athena had a failure on first flight partly because they used helicopter gyros for upper stage guidance, but in the thin air the gyros suffered from corona discharge.
Elon Musk may be very sensitive to quality control issues. Early in the Tesla production of its first car, there had been a quality problem for the entire fleet. Fortunately, the fleet was around 200 cars, but the recall was relatively expensive for a company that was just starting up.
pzatchok,
Please keep in mind my tale about the connectors becoming contaminated despite being protected with covers inside a cleanroom. The darnedest things happen. Also, the valves use Teflon gaskets, not the usual rubber-substitutes. Teflon is a tricky substance when pressed, because the stuff “mushes” in ways that rubbers don’t. My experience was to not use Teflon as a washer under a screw, for that reason, as the force on the screw will relax and it won’t be as secure in the threads as it should. Boeing said that they expected a small amount of leakage, and that they had believed this would be acceptable.
V-Man wrote: “everyone knows that Boeing would have gotten another check.”
They may not have. Their high price for the contract should anticipate expensive trouble such as this. Because these kinds of things can happen in development, the US government long ago started allowing “cost plus” contracts, which paid even the unexpected costs of development plus a profit. The problem was that this became standard, and it often resulted in incentives to not try hard to complete the development program. Fixed price is riskier for the vendor (Boeing, Northrup Grumman, Sierra Nevada, and SpaceX), but there is incentive for things to work correctly the first time.
As people have noted, it does not always work correctly the first time.
Jeff Wright wrote: “Marshall’s calling was big rocketry, Robert… Keeping things in house works. Area 5I proves that”
SLS proves that this does not work at Marshall, and several other rocket projects in the past half century proves that it doesn’t work anywhere at NASA, anymore. Congress turned NASA from a can-do organization to something else. Congress squanders the knowledge, skills, and talents of its rocket scientists, engineers, and technicians. We did super heavy launch with SLS, and it was designed by Congress but not to do what we need done. The Space Shuttle and ISS turned into disappointments, but SLS was designed to disappoint.
Even if NASA achieves its stretch goal of one launch per year, that is not enough. NASA will have to choose whether to use that launch to take people to a moon base, send a probe to deep space, or put a major piece of hardware into orbit (e.g. a Mars mission module). How long will it take NASA to send men to Mars if it can only construct the transit spacecraft when it doesn’t send men to the moon base?
Why didn’t Marshall develop reusable booster stages? Why didn’t Marshall develop Starship? Why did Orion, which isn’t that much more capable than Dragon or Starliner, take so long and cost so much to develop? The answer is that Congress is not eager to improve its space hardware, and it isn’t eager to get new hardware in operation quickly.
So, no. In house NASA does not work, because Congress won’t let it.
We had expected so much to happen as a result of the Space Shuttle. Space travel would become commonplace, alternate vehicles would be made, civilians would become astronauts, and commercial manufacturing would bring us benefits directly from space, but government chose to go a different way. With the ISS we still don’t have commercial manufacturing. We aren’t getting most of the benefits from space that we had expected and thought our tax dollars were paying for, and with government in charge we still won’t get these benefits twenty years from now. Once Starliner is operational, we will get these benefits twice as fast as we would with Crew Dragon alone. Eventually we will have manned Dream Chaser, and we will really be flying.
This is why so many people are eager for commercial space competition. The competitors will provide what we want to buy, and they will work to do it for less cost. Government has failed to do either of these things. When we let government be in charge, all we get is what government wants. When we are in charge, we get what we want.
Edward asked. “How does it go wrong, and how does it go this wrong?”
I was working for a Boeing contractor in the late ’90’s, and this article pretty well explains what happened:
“The 1997 merger that paved the way for the Boeing 737 Max crisis” on qz.com.
Then corporate moved to Chicago in 2001, putting management 1500 miles from the factory floor. The doom-and-gloom predictions of the time have fairly come true.
When we let government be in charge, all we get is what government wants. When we are in charge, we get what we want.
It’s a question of incentives.
During the 1960’s, government was driven by its legitimate foreign-policy and defense concerns, so NASA was resourced and enabled by our government to successfully address those concerns with respect to our presence in space. And the personnel within NASA, who shared those same Cold War concerns and were old enough to have a memory of the greater challenge of WWII, responded successfully to the government’s direction. All involved had a legitimate objective: out-do the significant, totalitarian, foreign competition to prevent their dominance … much the same objective Reagan had in the 1980’s regarding both economic and defense policy.
Once we beat the Russians to the moon and met that objective, NASA gradually became just another government agency and conduit for Congressional pork. It lost the incentive to sustain excellence, IMO, just like almost every other government entity does over time.
I wonder how many engineers are between the actual assembly workers and the real decision makers?
When a problem crops up how many people have to get involved before a corrective action is taken?
Add in accountants if they have closed the budget and paid for that subsection/assembly.
William,
You asked questions such as: “My impression is that aircraft systems tolerate humidity, cold, heat and dust. Why not space craft? Especially if we expect to bring them to the moon and Mars, and reuse them”
These are good questions to ask, and they are questions that rocket engineers have asked. How does New Shepard prevent its thruster injectors from being clogged with dust on landing, or Dragon prevent anything from being corroded by sea water on splashdown? The Russians don’t have to worry about the dust kicked up on Soyuz landings, because they don’t reuse their spacecraft. The Americans didn’t have to worry about sea water on Mercury, Gemini, or Apollo for the same reason.
(Airplanes don’t fly into dust, because that may end the service life of the engine(s). They usually don’t land in the water, because that ends the life of most planes, and can end the life of a seaplane when the waves are too rough.)
But now our spacecraft have to survive contact with the enemy: the outside environment outside of rocket fairings. This requires changes to designs and operations. For example, operational changes to Curiosity have dramatically reduced the wear on its wheels.
Fortunately, there was some experience with the Space Shuttle, but they didn’t have as much dust or sea water to deal with after landing. We did discover, however, that the Shuttle didn’t do well with the stuff that was kicked up from the runway on landing, no matter how much they FOD walked or swept the runway. (A foreign objects and debris (FOD) walk is a line of people walking along a runway or cleanroom looking for the tiniest debris, such as pebbles or dropped screws, zip ties, and cetera. Runway FOD walks were invented long before the Shuttle needed them.) A bad experience with the Space Shuttle also showed us one way in which it can be tricky to design for reusability.
Aircraft are susceptible to humidity, cold, heat, dust, and other factors, which is one of the reasons pilots do a walk around before entering the cockpit. A century of bad experiences has taught us much about the problems that the environment can cause. It isn’t just the environment that can cause problems with reusable flight hardware, it can be wear, methods, distracted pilots, poor maintenance (such as poorly done or not done often enough), ergonomic design, etc. There is a nonfiction book called Fate Is The Hunter and a fictitious movie by the same name that present examples of “learning experiences.”
When you have a problem while driving your car, you pull right over and call for a tow. If you have a problem while flying an aircraft, you have only minutes in which to find a place to safely land (which is trickier at night). If you have a problem while in a spacecraft, you may have to wait hours (or days or months, on a trip to Mars) before you can get to safety. The risks increase but so do the consequences. Unbelievably, we have made flying safer than driving. Making spaceflight safe in reusable spacecraft could take some doing, but we are willing to do what it takes, and our experience in making flying safe will be critical to our success.
Why build indoors? If you see unpainted airplanes, they are usually a color such as green. This is a coating which protects the otherwise bare aluminum from corrosion. Starship development test units are being built “outdoors” because they are made of Stainless Steel, or more generically and less brand-name: corrosion resistant steel. Although they have to worry about bird droppings, they don’t have to worry as much about rust and other corrosions. Due to the low number of times they expect any test unit to fly, SpaceX probably isn’t terribly worried about storing certain pieces of hardware in the open air. There may not be much time for spots of corrosion to spread into real problems.
I would expect them to build their operational flight units indoors, however.
pzatchok asked: “I wonder how many engineers are between the actual assembly workers and the real decision makers? When a problem crops up how many people have to get involved before a corrective action is taken?”
It depends upon how much of a problem it is. One of my technicians once draped a wire over a small thruster on a communication satellite, and he didn’t realize he had done so until after he tugged on the wire. I wrote a non-compliance report to make sure it was inspected — so that a quality engineer would get involved. The propulsion group sent a couple of engineers to inspect the thruster for damage and alignment, but it was OK. The tech had not tugged hard before realizing the error. Including the propulsion group and the people who track open non-compliance reports, there were probably about a dozen people involved, including the supervisors, such as mine, the quality engineer who handled the report, and the propulsion engineers who inspected the thruster. A tiny error probably cost a couple thousand dollars and added a record to the spacecraft file.
Something like Boeing’s two Starliner problems involves a large number of people at many levels of authority and responsibility, both engineering and management, looking at more than just the problem that occurred but also at related items in order to assure all levels that this was not a systemic problem but is localized to the oxidizer valves only. The fault tree that they talked about in the press conference is where they are looking at the item plus related items, and they are looking at a large range of possible causes or contributing factors to the cause of the problem, meaning that a large number of people are investigating these items and the possible contributing factors. There would also be people who had to keep the customer updated on findings and eventual corrective actions. There would be several internal meetings and some customer meetings, and there may even be a customer representative at the internal meetings. These meetings would discuss the possible contributing factors and explain why they are or are not involved in the problem. Some of these factors may include human factors (were errors made), procedures (were they followed, are they correct), design and manufacturing, performance during previous testing, the propellant loading activities are likely being closely scrutinized, the possibility of effects from the storm, assuring that the ground support equipment truly sent the proper commands, the corrosion found during inspection, and other things I haven’t thought of.
Edward … also, are the fuels and oxidizers being used in these thrusters inherently more corrosive than the jet fuel and hydraulic fluid used in aircraft? I remember that some of the hypergolic fuel-oxidizer combinations historically used in spacecraft were pretty nasty in this regard. That could be a contributing factor.
Jester Naybor
The hypergolic fuels are all very corrosive. Far Far more than standard aircraft fuels which range from little more than specially formulated gasoline to kerosene and diesel fuel.
Early German tests with hypergolic fuels.
https://www.todayifoundout.com/index.php/2020/10/the-german-rocket-fighter-that-dissolved-its-pilots-alive/