Soyuz failed because it was programmed for Baikonur, not Vostochny
It now appears that the Soyuz rocket failure this week occurred because the Fregat upper stage had not been programmed for a launch from Vostochny.
Although the information is still preliminary, it is increasingly clear that all the hardware aboard the Fregat upper stage performed as planned. But, almost unbelievably, the flight control system on the Fregat did not have the correct settings for the mission originating from the new launch site in Vostochny, as apposed to routine launches from Baikonur and Plesetsk. As a result, as soon as Fregat and its cargo separated from the third stage of the launch vehicle, its flight control system began commanding a change of orientation of the stack to compensate for what the computer had perceived as a deviation from the correct attitude, which was considerable. As a result, when the Fregat began its first preprogrammed main engine firing, the vehicle was apparently still changing its attitude, which led to a maneuvering in a wrong direction. [emphasis mine]
This reminds me of the NASA’s epic failure with Mars Climate Orbiter in 1998, where some programming used the metric system and other programming used the English system, and no one noticed.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
It now appears that the Soyuz rocket failure this week occurred because the Fregat upper stage had not been programmed for a launch from Vostochny.
Although the information is still preliminary, it is increasingly clear that all the hardware aboard the Fregat upper stage performed as planned. But, almost unbelievably, the flight control system on the Fregat did not have the correct settings for the mission originating from the new launch site in Vostochny, as apposed to routine launches from Baikonur and Plesetsk. As a result, as soon as Fregat and its cargo separated from the third stage of the launch vehicle, its flight control system began commanding a change of orientation of the stack to compensate for what the computer had perceived as a deviation from the correct attitude, which was considerable. As a result, when the Fregat began its first preprogrammed main engine firing, the vehicle was apparently still changing its attitude, which led to a maneuvering in a wrong direction. [emphasis mine]
This reminds me of the NASA’s epic failure with Mars Climate Orbiter in 1998, where some programming used the metric system and other programming used the English system, and no one noticed.
On Christmas Eve 1968 three Americans became the first humans to visit another world. What they did to celebrate was unexpected and profound, and will be remembered throughout all human history. Genesis: the Story of Apollo 8, Robert Zimmerman's classic history of humanity's first journey to another world, tells that story, and it is now available as both an ebook and an audiobook, both with a foreword by Valerie Anders and a new introduction by Robert Zimmerman.
The ebook is available everywhere for $5.99 (before discount) at amazon, or direct from my ebook publisher, ebookit. If you buy it from ebookit you don't support the big tech companies and the author gets a bigger cut much sooner.
The audiobook is also available at all these vendors, and is also free with a 30-day trial membership to Audible.
"Not simply about one mission, [Genesis] is also the history of America's quest for the moon... Zimmerman has done a masterful job of tying disparate events together into a solid account of one of America's greatest human triumphs."--San Antonio Express-News
Halt and Catch Fire, Episode 1,
“Computer’s aren’t the thing…”
https://youtu.be/YQLbi4VXYcA
(1:00)
Regarding Mars Climate Orbiter, yes a “simple” units discrepancy was the root of the failure. But really it’s a management problem, and a team culture problem.
Just like this Soyuz failure, and just like Challenger, and Columbia, and Hubble, and…
More details about Mars Climate Orbiter here https://spectrum.ieee.org/aerospace/robotic-exploration/why-the-mars-probe-went-off-course
Read the whole thing. If we can’t learn from our failures, we are doomed to repeat them.
Steve Golson: In every case you cite, of which I have written a great deal, including several books, the root cause always comes back to government management and the government politics that go with it. Private enterprise is certainly not perfect or immune to failure or human error, but because private companies can only exist if customers voluntarily give money to them, they tend to work to a higher standard. Otherwise, they go out of business.
Government operations have different motives. They get their money through tax dollars, which are obtained by coercion and are doled out based on political deals, not because the government was searching for the best product to get the job done. The result too often is bad management (Challenger, Columbia, Russia’s rockets), budget overruns (Webb, WFIRST, SLS), long delays (SLS, Orion, Webb, WFIRST), and badly built products (Mars Climate Orbiter, Russian rockets, Hubble’s mirror).
I could list more examples. They are endless, and are not confined just to the space program. Even the most naive person today recognizes how badly our government functions. Yet, as a society too many people still want to give that government more power and more responsibilities. The contradiction baffles me.
Private enterprise is certainly not perfect or immune to failure or human error, but because private companies can only exist if customers voluntarily give money to them, they tend to work to a higher standard
I sometimes hear, with apparent pride, “our salespeople do not work on commission”….as if that’s necessarly a good thing.
I once found myself selling stereo equipment (as a commissioned salesman) alongside a left-wing Democrat. He was mean, surly, sour, and a pessimistic nihilist.
Yet when a customer walked through the door, he was beaming, friendly and genuinely helpful. Why? Because it was in his best interest!
And I thought, “What a wonderful thing this Invisible Hand. It even creates nice people”. (if only temporarily)
Steve Golson
November 30, 2017 at 1:27 pm
Regarding Mars Climate Orbiter, yes a “simple” units discrepancy was the root of the failure. But really it’s a management problem, and a team culture problem.
Ding ding ding.
An easy enough problem to fix.
Steve Golson,
Thank you for the link. It has a lot of detail.
From Steve’s linked article: “There were no navigation experts on the team at this point (they were in fact not added until two months before launch, and had no significant knowledge of the spacecraft’s peculiarities even then).”
This shows the need for concurrent engineering. Yes, it is rocket science, and that means that the whole system must be considered throughout the process (planning, design, operations, etc.). I don’t know about other industries, but systems engineers are important in the space industry. More than one has explained to me why my initial designs conflicted with an aspect of the rest of the mission.
From Steve’s linked article: “But the rushed and inadequate preparations left no time to do it right.”
This is one of the reasons that I had low expectations that Falcon Heavy would fly in the announced late December time frame. Over the course of six weeks, something is bound to come up that needs more time to do it right, and apparently something has. Rather than rush it, do it right. This is a reason why Richard Branson is so often wrong about his own time frames, because he is trying to do it right — although he has been much farther wrong than most schedule slips.
From Steve’s linked article: “Although the navigators continued to express concern about the spacecraft trajectory, NASA’s Stephenson explained why there had been no management response. ‘They did not use the existing formal process for such concerns,’ he stated.”
And there is one of the bad management problems that Robert mentioned. A bureaucratic SNAFU if ever there was one. ‘Your concern cannot be valid, because you did not go through proper channels.’
Robert,
It seems that Russia also has a quality control problem with its software. This should have been an easily discovered problem. It is important to learn from other peoples’ errors, such as the software error that doomed Ariane V’s first flight, although that was a more difficult problem to find after the fact.
As my first mentor said, “you can’t inspect quality into a design.” Meaning that quality must be designed in, even planned into the design before pencil is put to vellum. My first mentor was a long time ago, when we still used pencil on vellum.
This is a problem that the Russians are probably going to have for quite some time. They lost one mission because the upper stage was used in a way that the designers had not intended; the designers allowed for a possible fuel line routing along an area where the fuel could freeze in the line during a long pause between engine firings. That upper stage design worked for a while, because the other, alternate, routing just happened to be used on the previous upper stages used in that fashion. How many more similar surprises are in store for the Russians as they continue to use ancient designs in unintended ways? Come to think of it, the Baikonur settings were used in an unintended way: a launch from Vostochny.
Part of the checklist for launch should be to enter the launch location and the software should abort with an explanation. A good programmer would not have missed this (always check assumptions even if just in debug code.) Documentation of assumptions (right in the code) should be the least thing they do.
Units of measure is a bit trickier since uom are not included in raw calculations. Perhaps a language that type-checks uom would be a good idea?
People excel in some things and computers in others. We don’t distribute that workload very well. Microsoft even killed the most popular and productive language they ever had and nobody has been able to replace it in over a decade.