One person's error, missed backups, full reboots, organizational chaos, privacy and self-regulation
Key details are beginning to emerge regarding the failure of the European Galileo satellite navigation system that occurred this summer - and they turn out to be quite unsightly.
One of the project managers tried to blame one person for the failure of the entire system, but sources inside the project warn that organizational chaos, excessive secrecy and unusual practices of self-regulation are also to blame.
Along with these problems, the battle of European organizations for control over the satellite system and the delay in independent reporting on the July plug indicate serious problems for the European competitor to the American GPS. Perhaps the system has a long overdue shake.
In mid-July, the agency responsible for the operation of the network of 26 satellites, the European agency of global satellite navigation systems EGSA, warned of "deterioration of the service", but promised that all problems would be quickly resolved.
However, they could not be solved, and even six days later the system did not just not work - its readings became less and less accurate , and the location of the satellites they reported was radically different from the real one. This was a serious problem for a system whose sole purpose was to provide positioning data with an error of up to 20 cm.
Billions of organizations, individuals, telephones, applications, and other things around the globe simply stopped using Galileo. It’s hard to imagine a bigger mess, except if these satellites hit Earth.
But, despite the denial of service and heavy criticism of those responsible for the inability to explain what was happening, the agencies and organizations involved in this project practically did not give any information.
Investigation
In September it was announced that an independent investigation into the causes of the incident was planned, mainly due to a lack of information. The "preliminary recommendations" of the investigation promised to be given by October. And still no information.
Then, in early November, the person in charge of the entire system, Deputy Director Pierre Delso, broke the silence at a meeting of European space policy officials in Washington.
During the Q&A session after the presentations, Delso was asked about the reasons for the lack of information, opacity, and the lack of a backup system. He exploded, claiming that the problem was caused by one person who made a mistake and failed to correctly perform actions to eliminate it. He said that this mistake was "unacceptable" and stated that it would "never happen again."
Delso said nothing about the backups or the lack of information and transparency. But other European officials also did not respond to these questions, indicating that the problem was discussed at a recent conference - it was after this that some space lovers buried in information from the website of the Navigation Institute, from the page of the conference in Miami, held in September.
One of these researchers was Bert Hubert, a DNS expert who became interested in the mess that had been going on with Galileo this summer and decided to create his own independent resource to monitor the status of the system. In early November, Hubert published a report on what he had discovered since the beginning of his investigation, listing some of the organizational and political problems that Galileo had.
No numbers
Among the most interesting details related to the Galileo, there is one such thing: out of 26 satellites in space, only 21 work; however, to achieve the planned accuracy, the system requires at least 24 satellites.
It is planned to launch additional satellites next year, but the likelihood of their introduction into orbit is constantly decreasing, especially in view of the feud between the European Space Agency and the European Union.
ESA has created the Galileo system and is working on an updated version of the system for which it is necessary to launch new satellites. But due to EU policies that have worsened due to Brexit , the EU wants to achieve stricter control over this project.
The EU is going to create a new European space agency, EUSA, which, in essence, will be a simple renaming of the existing agency of global satellite navigation systems. And another space agency, GSA, will become the EU space program agency, and the EU will soon appoint a new director for the “defense industry and space”. In short, there is a lot of political turmoil, which leads to a variety of problems.
Brief summary
And in the process of all this, the Galileo’s flagship European satellite navigation system is completely abandoned, and no one can explain in plain language what happened and why. Here's what we know based on a report based on the September Miami conference and additional details excavated by Hubert and others.
- The foggy reports of the Galileo team, which claimed that everything was fine and nothing to worry about, related to the fact that physically all the satellites were in working condition (except for those that did not work), and at the expected positions. In other words, everything was normal with iron in orbit; the satellites did not collide with anything and did not fly anywhere.
- The real problem is almost certainly related to software that solves the complex task of synchronizing the entire system. It is quite difficult to observe the nanosecond accuracy of atomic clocks that satellites flying in different orbits are equipped with. During the update, a certain anomaly occurred in the system related to the reference time - this was the operator’s error - because of which the whole system went to pieces.
- For reasons that remained unclear, the backup system was unavailable, so you could not just roll back to the previous version. As a result, everything began to work less and less accurately.
- In addition, apparently, at the time of the failure, the system was not configured correctly, so it was very difficult for the engineers to figure out how to make it work again.
- In the end, it was decided that if it takes so long to find the answer to the question of what went wrong, it will be easier to reboot the entire system. What the engineers did. But since the system is extremely complex, it took several days to reboot.
This is the most complete incident information. However, there is some small but valuable information on how and why everything went wrong, and why the system did not have an adequate recovery method.
Complexity
It is becoming increasingly likely that an important part of the problem was the complex interweaving of organizations responsible for maintaining and developing the various parts of this system. Immediately after everything broke down, it became clear that within the framework of the Galileo project there was no effective scheme for exchanging information, and also that everyone immediately began to blame each other for problems, which only worsened the situation.
Incomplete Galileo Project Management Scheme by Bert Hubert
In addition, there is the question of whether one particular organization, GMV, has additional responsibility for all this confusion. After all, she controls no less than three different parts of the Galileo structure.
What is particularly noteworthy, she manages the Orbital Synchronization Processing (OSPF) department, which is responsible for issuing the data that went into the separation in this case - the ephemeris . In addition to the ephemeris, GMV is subordinate to the data integrity processing department, which independently monitors and tracks the same data.
Was one of the reasons for the collapse that one company checks itself?
As for public relations, not a single organization from the constellation surrounding Galileo considered itself entitled to talk about what was happening, and left this task to officials - none of whom knew what was happening. In other words, it was a classic failure in the exchange of information.
We still do not exactly imagine what happened, but we hope that as a result of an independent investigation a detailed report will be released soon. It should be completed by the end of the year.
Meanwhile, the dangerous volume of political maneuvering is forcing engineers not to stick out. It’s a pity, because, apparently, they spend large volumes of work complicated by organizational chaos.
In short, Galileo is a classic European enterprise: a great idea in the implementation of talented people, turned into a bureaucratic mess in which no one wants to be responsible for problems arising from unjustified organizational difficulties.