DOCUMENTATION: A Diary of the Recovery
At 23:16 GMT on 24 June 1998, after comleting its nominal two-year mission in April 1998, ground controllers lost contact with the SOHO spacecraft during routine maintenance operations, and the satellite went into Emergency Sun Reacquisition (ESR) mode. The ESR mode is activated when an anomaly occurs and the spacecraft loses its orientation towards the Sun. When this happens, the spacecraft automatically tries to point itself towards the Sun again by firing its attitude control thrusters under the guidance of an onboard Sun sensor. Efforts to re-establish nominal operations did not succeed and telemetry was lost, not to be reestablished for several weeks. On 15 July the SOHO Mission Interruption Joint ESA/NASA Investigation Board released its Preliminary Status and Background Report in which it concentrated on three errors that appear to have led to the interruption of communications. It determined that the first two errors were contained in preprogrammed command sequences executed on ground system computers, while the last error was a decision to send a command to the spacecraft in response to unexpected telemetry readings. The first error was in a preprogrammed command sequence that lacked a command to enable an on-board software function designed to activate a gyro needed for control in ESR mode. The second error, which was in a different preprogrammed command sequence, resulted in incorrect readings from one of the spacecraft's three gyroscopes, which in turn triggered an ESR. In an attempt to recover SOHO as soon as possible, the Flight Operations Team continued uplinking commands to the spacecraft via NASA's Deep Space Network, for at least 12 hours per day (normal pass) plus all supplementary time given by DSN. The ESA ground stations in Perth, Vilspa and Redu supported the search for a downlink signal. Special equipment was set up at the ground stations to search for spikes in the downlink spectrum and view it in real time at the SOHO operations facilities at Goddard Space Flight Center. Analysis by attitude experts led to the conclusion that SOHO went into a spin around an axis such that the solar panels were faced nearly edge-on towards the Sun, and thus did not generate any power. Since the spin axis is fixed in space, as the spacecraft progressed in its orbit around the Sun, the orientation of the panels with respect to the Sun gradually changed, resulting in increased solar illumination of the spacecraft solar arrays as time progressed. On July 23 researchers at the National Astronomy and Ionosphere Center (NAIC) in Arecibo, Puerto Rico, used the facility's 305-meter diameter radio telescope to transmit a signal toward SOHO while the 70-meter dish of NASA's Deep Space Network in Goldstone (USA) acted as a receiver, locating the spacecraft's echo and tracking it using radar techniques for more than an hour. SOHO was found to be slowly rotating near its expected position in space. On 3 August contact was re-established with SOHO following six weeks of silence. Signals sent through the DSN station at Canberra, Australia, were answered at 22:51 GMT in the form of bursts of carrier signal lasting from 2 to 10 seconds. These signals were recorded both by the NASA DSN station and the ESA Perth station. 37 different command procedures had been tried before this first detection of a carrier spike. Command sequences were uplinked to divert the available solar array power into a partial charging of one of the on-board batteries. After 10 hours of battery charging, the telemetry was commanded on and seven full sets of telemetry frames giving the spacecraft's on-board status were received on 8 August, at 23:15 hrs GMT, six days after receiving the first signal. Further details on the on-board conditions were obtained the following day (Sunday 9 August) in two subsequent telemetry acquisitions lasting four and five minutes respectively. Data gathered included information on the temperature of the payload instruments. After both batteries were fully charged thawing of the hydrazine fuel in the tank was started on 12 August at 22:39 UT. It was interrupted several times during the week in order to recharge the batteries, necessary because the power data revealed a slightly negative power balance. Thawing of the hydrazine in the tank was completed on 28 August at 23 UT after 275 hours of tank heating. After 36 hours of recharging the batteries, heating of the first of four fuel pipe sections, which connect the tank to the thrusters, commenced at 12:30 UT on 30 August. On August 31, the SOHO Mission Interruption Joint ESA/NASA Investigation Board released its final report (http://sohowww.estec.esa.nl/whatsnew/SOHO_final_report.html). It concluded that the chain of events leading to the interruption of contact with the SOHO spacecraft, which was described in its preliminary report, was correct, and it recommended, in order to prevent similar mishaps in the future, that ESA and NASA review and correct the S/C ground procedures, the procedure implementation, the management structure and process, and the ground systems. No fault on the spacecraft contributed to the mishap. As the fuel pipes were slowly thawing, attitude recovery was planned from early September on. Due to the precarious power balance it took until 10 September to thaw one of the two redundant branches of the fuel pipes. After this the batteries were recharged and the propulsion system temperature was maintained in preparation for the attitude recovery maneuver. The verification of the procedures for attitude recovery was completed on 14 September and a rehearsal of the attitude recovery maneuver was carried out on the next day. Finally, on 16 September, the first but important step in the SOHO recovery was successfully completed. Sun pointing (without roll control) was achieved at 18:30 UT, after a gradual despin of the spacecraft followed by a (planned this time ...) Emergency Sun Reacquisition. All operations went according to plan. The experiment substitution heaters were switched on 42 minutes after the ESR was triggered. After a busy week of recommissioning activities of the various spacecraft subsystems and an orbit correction maneuver, SOHO was finally brought back to normal mode on 25 September at 19:52:58 UT. Instrument switch on started on 5 October 1998 with the SUMER instrument, followed by VIRGO on 6 October, GOLF on 7 October, COSTEP and ERNE on 9 October, UVCS on 10 October, MDI on 12 October, and LASCO and EIT on 13 October. At the time of writing (14 October), no signs of damage due to thermal stress during the deep freeze have been detected. VIRGO, GOLF, COSTEP, and ERNE are already fully functional, the other instruments are still undergoing careful recommissioning activities. CDS switch on is planned for 17 October, SWAN switch on for 18 October, and CELIAS switch on for 23 October. The latest EIT images can be found on the web at http://umbra.nascom.nasa.gov/eit/eit_full_res.html MDI images under the MDI home page at http://soi.stanford.edu. Details of the recommissioning status of UVCS are available at http://cfa-www.harvard.edu/uvcs/recommission.html, the LASCO recommissioning page can be found at http://lasco-www.nrl.navy.mil/recommiss/recom_sum.html.