SOHO - A Diary of the Recovery


At 23:16 GMT on 24 June 1998, after comleting its nominal two-year
mission in April 1998, ground controllers lost contact with the SOHO
spacecraft during routine maintenance operations, and the satellite
went into Emergency Sun Reacquisition (ESR) mode. The ESR mode is
activated when an anomaly occurs and the spacecraft loses its
orientation towards the Sun. When this happens, the spacecraft
automatically tries to point itself towards the Sun again by firing its
attitude control thrusters under the guidance of an onboard Sun
sensor.  Efforts to re-establish nominal operations did not succeed and
telemetry was lost, not to be reestablished for several weeks.

On 15 July the SOHO Mission Interruption Joint ESA/NASA Investigation
Board released its Preliminary Status and Background Report in which it
concentrated on three errors that appear to have led to the
interruption of communications. It determined  that the first two
errors were contained in preprogrammed command sequences executed on
ground system computers, while the last error was a decision to send a
command to the spacecraft in response to unexpected telemetry
readings.

The first error was in a preprogrammed command sequence that lacked a
command to enable an on-board software function designed to activate a
gyro needed for control in ESR mode. The second error, which was in a
different preprogrammed command sequence, resulted in incorrect
readings from one of the spacecraft's three gyroscopes, which in turn
triggered an ESR.

In an attempt to recover SOHO as soon as possible, the Flight
Operations Team continued uplinking commands to the spacecraft via
NASA's Deep Space Network, for at least 12 hours per day (normal pass)
plus all supplementary time given by DSN.  The ESA ground stations in
Perth, Vilspa and Redu supported the search for a downlink signal.
Special equipment was set up at the ground stations to search for
spikes in the downlink spectrum and view it in real time at the SOHO
operations facilities at Goddard Space Flight Center.

Analysis by attitude experts led to the conclusion that SOHO went into
a spin around an axis such that the solar panels were faced nearly
edge-on towards the Sun, and thus did not generate any power. Since the
spin axis is fixed in space, as the spacecraft progressed in its orbit
around the Sun, the orientation of the panels with respect to the Sun
gradually changed, resulting in increased solar illumination of the
spacecraft solar arrays as time progressed.

On July 23 researchers at the National Astronomy and Ionosphere Center
(NAIC) in Arecibo, Puerto Rico, used the facility's 305-meter diameter
radio telescope to transmit a signal toward SOHO while the 70-meter
dish of NASA's Deep Space Network in Goldstone (USA) acted as a
receiver, locating the spacecraft's echo and tracking it using radar
techniques for more than an hour. SOHO was found to be slowly rotating
near its expected position in space.

On 3 August contact was re-established with SOHO following six weeks of
silence.  Signals sent through the DSN station at Canberra, Australia,
were answered at 22:51 GMT in the form of bursts of carrier signal
lasting from 2 to 10 seconds. These signals were recorded both by the
NASA DSN station and the ESA Perth station. 37 different command
procedures had been tried before this first detection of a carrier
spike.

Command sequences were uplinked to divert the available solar array
power into a partial charging of one of the on-board batteries.  After
10 hours of battery charging, the telemetry was commanded on and seven
full sets of telemetry frames giving the spacecraft's on-board status were
received on 8 August, at 23:15 hrs GMT, six days after receiving the
first signal. Further details on the on-board conditions were obtained
the following day (Sunday 9 August) in two subsequent telemetry
acquisitions lasting four and five minutes respectively. Data gathered
included information on the temperature of the payload instruments.

After both batteries were fully charged thawing of the hydrazine fuel
in the tank was started on 12 August at 22:39 UT. It was interrupted
several times during the week in order to recharge the batteries,
necessary because the power data revealed a slightly negative power
balance.

Thawing of the hydrazine in the tank was completed on 28 August at 23
UT after 275 hours of tank heating. After 36 hours of recharging the
batteries, heating of the first of four fuel pipe sections, which
connect the tank to the thrusters, commenced at 12:30 UT on 30 August.

On August 31, the SOHO Mission Interruption Joint ESA/NASA
Investigation Board released its final report
(http://sohowww.estec.esa.nl/whatsnew/SOHO_final_report.html).  It
concluded that the chain of events leading to the interruption of
contact with the SOHO spacecraft, which was described in its
preliminary report, was correct, and it recommended, in order to
prevent similar mishaps in the future, that ESA and NASA review and
correct the S/C ground procedures, the procedure implementation,  the
management structure and process, and the ground systems. No fault on
the spacecraft contributed to the mishap.

As the fuel pipes were slowly thawing, attitude recovery was planned
from early September on.  Due to the precarious power balance it took
until 10 September to thaw one of the two redundant branches of the
fuel pipes. After this the batteries were recharged and the propulsion
system temperature was maintained in preparation for the attitude
recovery maneuver.

The verification of the procedures for attitude recovery was completed
on 14 September and a rehearsal of the attitude recovery maneuver was
carried out on the next day.

Finally, on 16 September, the first but important step in the SOHO
recovery was successfully completed. Sun pointing (without roll
control) was achieved at 18:30 UT, after a gradual despin of the
spacecraft followed by a (planned this time ...) Emergency Sun
Reacquisition.  All operations went according to plan.  The experiment
substitution heaters were switched on 42 minutes after the ESR was
triggered.

After a busy week of recommissioning activities of the various 
spacecraft subsystems and an orbit correction maneuver, SOHO 
was finally brought back to normal mode on 25 September at 19:52:58 UT.

Instrument switch on started on 5 October 1998 with the SUMER
instrument, followed by VIRGO on 6 October, GOLF on 7 October, COSTEP
and ERNE on 9 October, UVCS on 10 October, MDI on 12 October, and LASCO
and EIT on 13 October. At the time of writing (14 October), no signs of
damage due to thermal stress during the deep freeze have been detected.
VIRGO, GOLF, COSTEP, and ERNE are already fully functional, the other
instruments are still undergoing careful recommissioning activities.
CDS switch on is planned for 17 October, SWAN switch on for 18 October,
and CELIAS switch on for 23 October.

The latest EIT images can be found on the web at
http://umbra.nascom.nasa.gov/eit/eit_full_res.html 
MDI images under the MDI home page at 
http://soi.stanford.edu. 
Details of the recommissioning status of UVCS are available at
http://cfa-www.harvard.edu/uvcs/recommission.html, 
the LASCO recommissioning page can be found at
http://lasco-www.nrl.navy.mil/recommiss/recom_sum.html.  

Last modification: June 04, 2002
Feedback and comments about this site, please direct to the
SOHO Webmaster.
Questions about our mission, the spacecraft and science, please check: Dr. SOHO.

Click here to escape from frames