Tiger Team solves mystery of Euclid’s galaxy speed camera
When tested in space-like conditions, the instrument on ESA’s Euclid space telescope set to measure the velocities of fast-receding galaxies abruptly stopped operating. Blindsided, the mission team feared it might take weeks or even months of extra time to uncover the reason why. Then a newly-convened ESA Tiger Team came to their assistance, pinpointing the cause and keeping Euclid on its set launch schedule.
Euclid is a space telescope set to explore the dark Universe. Over the course of its six year mission, operating 1.5 million km from Earth, Euclid’s 1.2 m diameter telescope will map the 3D distribution of up to two billion galaxies up to 10 billion light years away – around a third of the observable Universe. By revealing the Universe’s large-scale structure, and its pattern of expansion, the mission will cast light on the mysterious dark energy and dark matter making up the vast majority of the cosmos.
To achieve this Euclid will employ a pair of instruments, designed by the nearly 1 000 scientists from 100 institutes from 13 European countries making up the Euclid consortium (plus NASA contributions). A visible light camera to image distant galaxies works alongside the Near Infrared Spectrometer and Photometer. NISP splits infrared light coming from these galaxies to derive key data, including their speed of outward expansion – measuring their ‘red shift’, on the same principle as a police radar gun.
The problem? All of a sudden – while being tested with the rest of Euclid’s payload module in a cold vacuum state – NISP refused to go on working.
A camera that doesn’t respond
Paolo Strada, ESA’s Senior Instrument Engineer for Euclid, explains: “The anomaly emerged with the instrument’s flight model, once it had already been fully tested, formally accepted then integrated into the spacecraft’s payload module – along with all other elements including the telescope, camera and optic bench.
In late May the payload module had commenced ‘thermal vacuum’ testing inside a Centre Spatial de Liège chamber in Belgium when NISP began failing to initiate a second acquisition once its first acquisition was complete.
“Imagine having a nice digital camera. You take a picture, the image is good and you want to take another. But as you try to do that, the camera control is no longer responding: you cannot set the exposure, or release the shutter. The only way to take another picture is to restart, but after taking another picture the problem reoccurs.”
Regular restarts would scarcely be an option if Euclid is to complete its mammoth observing task; Euclid is designed to stream 850 gigabits of data daily to Earth, adding up to many petabytes of raw imagery over the mission’s lifetime.
Checking clues from telemetry
“This was something we had never experienced before,” adds Paolo. “The NISP instrument had performed wonderfully through hundreds of hours of flawless operation, transferring hundreds of terabytes with no problem. The stopping did seem to depend on the cryogenic temperature experienced inside the chamber – it disappeared in ambient conditions – but our hands were tied when it came to investigating: NISP and the rest of the payload module stayed inside the thermal vacuum chamber, as the nominal test campaign continued in the meantime. It was almost like the mission was already in space; for the time being we were limited to the information we could obtain from telemetry.”
Consortium members, industry partners and the ESA mission team were collectively baffled. Suspicion centred on the cables that transferred data from NISP’s focal plane sensors to its computer. Perhaps there was signal noise, electromagnetic interference from elsewhere, or cabling being improperly grounded?
The range of possibilities was so wide however, that the Euclid team were considering a second, dedicated thermal vacuum campaign to get to the root of the problem – which would add weeks or months to their schedule and put their planned launch date beyond reach.
Enter the Tiger Team
Before such drastic action was taken, the mission put together a ‘Tiger Team’ of ESA and industry experts to try and solve the mystery.
Tiger Team member Felix Siegle, of ESA’s On-Board Computers and Data Handling Section, discovered the halting was rooted in a software bug, rather than any physical issue: “I read through the documentation, talked to the engineers, applied my trade, then began to feel a bit suspicious. If this really was purely physical in nature, then I’d expect the failure to be intermittent, but instead the freezing was occurring every single time.”
Felix homed in on areas of software code where data transmission might overlap with instrument control, telling the Euclid team to take look in such regions.
What they found was an error flag that in normal conditions should have been no problem was being wrongly interpreted by the software to signal that the telemetry and telecommand interface interface to the instrument sensors had become faulty, shutting it down.
Flagging an error
“The error flag had not appeared before because the cable configuration had shifted very slightly from previous tests, changing the signal propagation time from the sensors to the computer interface,” adds Felix.
“The cryogenic temperatures also played a role, creating slightly longer latency, hitting the window needed from a shift of just a few billionths of a second. The resulting flag was actually informative in nature, rather than something the software should have acted upon, but once identified, this unwanted behaviour could be corrected with a simple software patch.
“In terms of lessons learned, in dealing with such complex software, it is good to look at the areas of interface between hardware and software, the work of differing companies or engineers, because misunderstandings or mistakes can occur easily in such in-between regions.”
Paolo and the rest of the Euclid team were delighted with the rapid solution: “Felix had called me and said he thought he’d found something within a half an hour. It shows the importance of a fresh look at a problem.”
Euclid remains on its nominal schedule, with its payload module due to be transported to Thales Alenia Space in Turin, Italy to be integrated with its platform module.
The next step will be spacecraft-level testing ahead of a launch by a baseline Soyuz vehicle from Kourou in French Guiana, scheduled for a launch window opening from the end of 2022.
Tiger Team members
The Euclid NISP Tiger Team was made up of the following experts: Demis Boschetti (Thales Alenia Space); Emiliano Artina (Thales Alenia Space); Felix Siegle (On Board Computer & Data Handling Section in ESA's Directorate of Technology, Engineering and Quality); Johannes Wolf (EMC & Harness Section in ESA's Directorate of Technology, Engineering and Quality); Giorgio Magistrati (Studies and Technologies (ExPeRT) Team in ESA's Directorate of Human & Robotic Exploration); Paolo Strada (Mission & Payload Section, Euclid Project in the Projects Department of ESA's Directorate of Science); Veronique Ferlet Cavrois (Power Systems, EMC & Space Environment Division in ESA's Directorate of Technology, Engineering and Quality); Xavier Roy (Airbus Defence & Space), replacing his colleague Frederic Petit. The Tiger Team was chaired by Ferdinando Tonicello (Electrical Department,in ESA's Directorate of Technology, Engineering and Quality).