More detail on pad abort test parachute issue
At a press telecon yesterday Boeing outlined in more detail the cause of the failure of one main parachute to deploy during its November 4 Starliner pad abort test.
In a call with reporters, John Mulholland, vice president and program manager for commercial crew at Boeing, said an investigation after the Nov. 4 test at White Sands Missile Range in New Mexico led the company to conclude that a “lack of secure connection” between a pilot parachute and the main parachute prevented that main parachute, one of three, from deploying.
The pilot parachute is designed to deploy first, and pull out the main parachute. However, Mulholland said that hardware inspections and photographs taken during “closeout” of the vehicle prior to the test showed that a pin that links the pilot and main parachutes was not inserted properly.
“It’s very difficult, when you’re connecting that, to verify visually that it’s secured properly,” he said, in part because that portion of the parachute system is enclosed in a “protective sheath” intended to limit abrasion but which also makes it difficult to visually confirm the pin is in place. “In this particular case that pin wasn’t through the loop, but it wasn’t discovered in initial visual inspections because of that protective sheath.”
Mulholland said Boeing is modifying assembly procedures through what he called “fairly easy steps,” such as pull tests, to ensure those pins are properly installed. Technicians have already confirmed that the same parachute linkages are properly installed on the three parachutes on the Starliner that will launch in December on an orbital flight test to the International Space Station. [emphasis mine]
That a hardware inspection and photos taken before launch revealed this issue and resulted in nothing being done should rise serious questions at Boeing about its quality control processes. Based on the press telecon, however, it does not appear that Boeing is asking those questions. From a different report:
[John Mulholland, Boeing’s Starliner program manager] praised the rigging team, saying “even before we got eyes on the hardware, that team on their own initiative (was) reviewing the close-out photos and the processes, and they identified the potential issue that was subsequently validated by hardware inspection.”
“Most importantly, they raised their hand and and let us know what they believe the problem was,” he said. “It’s really a testament to the transparency of that team. The speak-up culture that we have, that is what we need on this program.”
While it is good that the rigging team was willing to speak up afterward, it is very bad that their procedures allowed the launch to go forward. The company says it has now changed its rigging procedures, but I don’t sense any effort on Boeing’s part to find out why its so-called “speak-up culture” failed to have these engineers speak-up, before launch.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
At a press telecon yesterday Boeing outlined in more detail the cause of the failure of one main parachute to deploy during its November 4 Starliner pad abort test.
In a call with reporters, John Mulholland, vice president and program manager for commercial crew at Boeing, said an investigation after the Nov. 4 test at White Sands Missile Range in New Mexico led the company to conclude that a “lack of secure connection” between a pilot parachute and the main parachute prevented that main parachute, one of three, from deploying.
The pilot parachute is designed to deploy first, and pull out the main parachute. However, Mulholland said that hardware inspections and photographs taken during “closeout” of the vehicle prior to the test showed that a pin that links the pilot and main parachutes was not inserted properly.
“It’s very difficult, when you’re connecting that, to verify visually that it’s secured properly,” he said, in part because that portion of the parachute system is enclosed in a “protective sheath” intended to limit abrasion but which also makes it difficult to visually confirm the pin is in place. “In this particular case that pin wasn’t through the loop, but it wasn’t discovered in initial visual inspections because of that protective sheath.”
Mulholland said Boeing is modifying assembly procedures through what he called “fairly easy steps,” such as pull tests, to ensure those pins are properly installed. Technicians have already confirmed that the same parachute linkages are properly installed on the three parachutes on the Starliner that will launch in December on an orbital flight test to the International Space Station. [emphasis mine]
That a hardware inspection and photos taken before launch revealed this issue and resulted in nothing being done should rise serious questions at Boeing about its quality control processes. Based on the press telecon, however, it does not appear that Boeing is asking those questions. From a different report:
[John Mulholland, Boeing’s Starliner program manager] praised the rigging team, saying “even before we got eyes on the hardware, that team on their own initiative (was) reviewing the close-out photos and the processes, and they identified the potential issue that was subsequently validated by hardware inspection.”
“Most importantly, they raised their hand and and let us know what they believe the problem was,” he said. “It’s really a testament to the transparency of that team. The speak-up culture that we have, that is what we need on this program.”
While it is good that the rigging team was willing to speak up afterward, it is very bad that their procedures allowed the launch to go forward. The company says it has now changed its rigging procedures, but I don’t sense any effort on Boeing’s part to find out why its so-called “speak-up culture” failed to have these engineers speak-up, before launch.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
Why does a PIN connect the drouge to the main? I’d want something clamped/locked/holding on for dear life. Full disclosure: I’m not an engineer ?
Steve,
When an engineer uses the word “pin” he’s not usually talking about something the size of what mom kept in her sewing basket to hem clothes. Pins that are part of some mechanism can be quite large and weigh many pounds each. Here are some pin pictures for you.
Mr. Z.,
You pose the same – and obvious – question that immediately occurred to me. What is the point of looking at closeout photos only after something bad happens? Seems as though they’d prove a lot more useful if looked at in advance.
At Boeing, at least, closeout photos seem to be like my usual experience with medical records – what we in computing used to refer to in jest as “Write-Only Memory” where stuff is recorded assiduously and in great volume, then never subsequently referenced.
Dick–
Good stuff!
Referencing the ‘write-only-memory’ medical-records, (my gosh, that is so darn true, it’s frightening.)–I spent 30 years in mental health, the only way I could ever make any progress in some situations, was if I went into the paper-archives and read every Case Note from day 1.
Dick Eagleson wrote: “Mr. Z., You pose the same – and obvious – question that immediately occurred to me. What is the point of looking at closeout photos only after something bad happens? Seems as though they’d prove a lot more useful if looked at in advance.”
Closeout photographs are of things that people had been looking at before closeout. The idea of the photos is to document that it was built right in the first place, but they are also used after a problem in order to discover what people saw but missed when they were viewing it all in person.
Almost everything is made correctly the first time. The inspection finds almost all the problems that were missed during that first manufacture or assembly. Closeout photos are the last resort to discovering what went wrong, because everyone thinks that it was all done right before closing up a box, a panel, or whatever. Otherwise they wouldn’t have closed it up. To repeat myself, the photographs document that it was built right.
The inspection before the closeout is generally more detailed than the photographs, because the person reviewing the setup has far more opportunity to do close inspections and can generally see more than is in a series of photographs. If something seems unusual or out of place, the in-person inspector can take a closer look, but a photograph is all that is available afterward, should something go wrong during test or on-orbit operations. Usually the photographs show that the problem was not due to the assembly or setup, but occasionally they can document that something was not done right, such as leaving a lens-cap in place when it should have been removed. (Hint: all remove-before-flight items, such as lens-caps and landing gear pins, should be red or have an obvious red tag that is an easily seen indicator that something still must be removed before closeout or flight.)
I have taken my share of pre-test photographs. In one case, they couldn’t show that a test cable had not been properly torqued before the test, which had to be investigated after the test. In another case they were also able to show that a reflective baffle had been taped in place before the test, but after the test it was seen that the baffle had sagged into the field of view of a cooling fin, causing the fin to perform poorly. In the second case, the thermal engineer had peeled off the tape to look at something behind the baffle and reused the tape after his inspection. Again, this reuse of the tape was not shown in the closeout photos. Lesson: do not reuse tape; it may be expensive stuff, but it is too likely to give way during a thermal vacuum test, causing far more problems and expense than the cost of the relatively small amount of tape.
Boeing’s lesson with their pins: do a pull test to assure that they are properly installed, because inspection is not sufficient.
When inspection is not enough, other means of verification must be employed.
Steve wrote: “I’d want something clamped/locked/holding on for dear life.”
Parachutes are outside of my expertise, but Apollo did not seem to have its drogue parachutes attached to the mains during descent. Why the drogues are released rather than kept attached is a mystery to me, but I am sure there is some reasoning to it. Hopefully, the main parachutes’ attachment points are more easily inspected.
NASA likes to have redundancy and backups, just in case something fails. Parachutes are important enough to require redundancy. Thus, when one parachute did not fully deploy on Apollo 15, the astronauts were unharmed by the splashdown. Two parachutes were sufficient.
https://www.youtube.com/watch?v=E-Vd75Ptg9I (90 seconds)
What happens if two parachutes fail to fully deploy, and how do we make sure that it does not happen? Empirical evidence showed that around one in forty Apollo parachutes failed. That suggests that around one in 1,600 flights should end with two parachutes failing to fully deploy. It also suggests that one in 64,000 flights would have had all three parachutes fail to fully deploy. Yikes! Don’t book me on that 64,000th Apollo flight.
Close out photos are just “who to blame” photos if you don’t used them before launch.