/u/ecs/soc/instrument_emergency_list.txt Note: This file is not complete. 16 May 2008 EBZ 30 Jul 2008 EBZ 20 Oct 2008 EBZ - read events table, no changes to list 28 Oct 2008 EBZ 29 Oct 2008 EBZ - added CDS stuff 17 Dec 2008 EBZ - added SUMER stuff 18 Dec 2008 EBZ - added CEPAC and ERNE stuff 12 Jan 2009 EBZ - added VIRGO latchup from 1/11/09 01 May 2009 EBZ - updated LASCO, SUMER, CEPAC info 06 May 2009 EBZ - udpated CELIAS, UVCS, SWAN, MDI 22 Sep 2009 EBZ - updated GOLF's emergency list with newest one List of All Instrument Emergencies ================================== All Instruments Off: Feb 05 2002 - All off due to main bus undervoltage Jan 27 2007 - All off due to SUMER redundant LCL on ESRs: Dec 4 1995 ESR 1 - roll rate, failure was ACU reset Nov 19 1997 ESR 2 - off-pointing, ACU power failure Mar 3 1998 ESR 3 - off-pointing Mar 3 1998 ESR 4 - roll rate Jun 24 1998 ESR 5 - false roll rate Jun 25 1998 ESR 6 - roll rate Jun 25 1998 ESR 7 - off-pointing Sep 16 1998 ESR 8 - commanded as part of recovery Sep 22 1998 ESR 9 - false off-pointing Dec 21 1998 ESR 10 - off-pointing, loss of last gyro Feb 14 1999 ESR 11 - off-pointing Nov 28 1999 ESR 12 - triggerd by ACU reset monitoring Dec 1 1999 ESR 13 - triggered from ground, no roll control in CRP due to ACU TM buffer overrun Nov 29 2000 ESR 15 - ACU reset Jan 14 2001 ESR 16 - ACU reset Jul 28 2001 ESR 17 - ACU reset Feb 5 2002 ESR 18 - main bus voltage drop, probable cause "Tin Whiskers" *Tin Whiskers* --> crystalline build up on surfaces that use Tin as a final finish, electronic system failures have been attributed to short circuits caused by tin whiskers that bridge closely-spaced circuit elements maintained at different electrical potentials Jul 8 2003 ESR 19 - caused by FSPAAD Apr 21 2004 ESR 20 - caused by FSPAAD Apr 22 2004 ESR 21 - caused by FSPAAD Apr 22 2004 ESR 22 - caused by FSPAAD Apr 23 2004 ESR 23 - triggered by FPSS sun prescence monitoring Dec 8 2004 ESR 24 - triggered by CSPAAD (false trigger) Other: Mar 1996 - Daily pulse was changed and can't ever be changed again Dec 10 2007 - Switched to redundant PRTU due to a SWAN LCL issue, VIRGO was okay GOLF: Sep 11 1996 - GOLF software reset itself due to multiple SEUs in software, copied software from PROM over autonomously October 2002 - Mechanisms are stuck so they are observing the blue wing only May 14 2003 - GIPWA went red, power supply voltages went low, turned off, needed to decrease sub heater duty cycles, caused by SEU or unknown Sep 28 2004 - Temperatures rose a bit, problem on an identified electronic module attached to GOLF Sensor Unit used to drive the polarizer mechanism. This has induced a part of a watt power disipation increase. Dec 16 2004 - GIPWA getting higher due to the problem GOLF had with a mechanism driving electronics in September Mar 28 2005 - GOLF temperatures increased slowly like 2 previous times, want to decrease sensor unit temperatures by turning off mechanism electronics Apr 08 2005 - Quarter Wave Plate and Linear Polarizer electronics turned off and substitution heaters (circuit 68 and 69) on at 30% Jan 2007 - OCD to make changes to halon_m and G03 for GIPWA and GSMECMOD, not implemented yet Mar 06 2008 - GIPWA and many mnemonics violated, turned GOLF off, SEU??? Sep 20 2009 - same as March 2008 VIRGO: Jun 28 1996 - DIARAD cover half open, open it Sep 09 1996 - Safed itself probably due to a short over-current (latch-up), GOLF and MDI saw temp and pointing changes, restarted on Sep 12 May 07 (?) 1997 - Latch-up May 20 1997 - Latch-up Mar 25/26 1998 - VIPWA red low - powered off, overcurrent possibly caused by a latch-up tripped the fuse in the power supply and stopped the instrument Mar 30 2001 - Latch-up Nov 11 2001 - Latch-up May 06 2002 - Latch-up Dec 07 2002 - Temperature CR2 was red because they changed the operation of DIARAD for one hour for a longer exposure Jul 14 2003 - Latch-up Apr 02 2004 - Safed itself, probably due to a short over-current (latch-up) like in 1996 "As to your remark about the LM139, we have indeed one for the current switch-off in the DC-DC, ......... But as I said above once per 8 years is not yet supporting any statistics." Feb 13 2005 - Latch-up Feb 20 2006 - Latch-up Aug 02 2005 - Current increase, cause unknown but probably false readings Apr 14 2006 - problem with DIARAD after a test - it got stuck with the right shutter open and the left closed and did not respond to follow-up command - created and uplinked a DIARAD reset TSTOL Jan 28 2007 - Violations and possible latch-up, related to turn-on and timing of v_fl_synchr "ignore all limits for 10 minutes after the first midnight after the v_fl_synchr is run' Jun 22 2008 - Latch-up Jan 11 2009 - Latch-up CDS: Oct 04 1996 - GIS detector off Dec 1996 - CTM18 alarms so run procedure, made into C-02 Feb 1997 Mar 29 1997 - Processor hung June 1997 - Sensor for CUVMCP sometimes faulty so made C-06 June 1997 - CTM29 yellow alarms are not a problem so ignore Jul 17 1997 - Processor hung Nov 10 1997 - Processor hung Dec 17 1997 - Processor hung Nov 28 1999 - CDS hung during submode change in ESR Dec 27 1999 - Processor hung Jan 29 2000 - S/W crash August 2001 - Slit mechanism malfunction Dec 26 2001 - CDS recovery Mar 31 2004 - Backup MCU timeout so cmds weren't effective - disable MCU till reboot Jun 11 2004 - Processor hung Jul 01 2004 - Watchdog Nov 08 2004 - CDS reboot Mar 23 2005 - Door opening caused FPSS/RW torque anomaly on s/c Jul 15 2005 - CIVMCP violations, many tests but not conclusive Nov 15 2005 - CIVMCP again, same as July Mar 10 2006 - UVCS extra current from nearby electronics box caused QTCR3 to get warmer Nov 03 2006 - CDS crash - cause unknown Jul 31 2008 - CDS crash Aug 08 2008 - CDS crashed during submode change LASCO: October 1996 - shutter hang that caused the config mons for ETE1 and ETE4 to be made? ocd 358????? Jul 21 1997 - S/W crash during submode change November 199 ocd 673??? made L03 Apr 26 1998 - LEB hung May 30 1998 - Hung EIT sector wheel Dec 11 1998 - Hung EIT sector wheel C1 never came back after mission interruption Feb 06 1999 - Hung EIT sector wheel Jul 07 1999 - Hung EIT sector wheel Mar 28 2000 - LEB hung Jul 03 2000 - Hung EIT sector wheel Aug 08 2000 - Hung EIT sector wheel Mar 10 2001 - Hung EIT sector wheel Mar 10 2001 - LEB hung Jul 23 2001 - Hung EIT sector wheel, camera error Jun 18 2002 - Occulter moved and exposed some sun, no damage Sep 04 2003 - Hung EIT Sector Wheel, during filter change Nov 30 2003 - Hung EIT Sector Wheel Apr 17 2004 - LEB bug and observation schedule messed up Oct 06 2004 - Hung EIT Sector Wheel, during filter change Jan 14 2005 - Hung EIT sector wheel Sep 02 2005 - LSPONOF violated, probably a glitch Nov 22 2005 - LEB hung, buffer overflow that shutdown OBE and LEB Nov 28 2005 - LEB hung Dec 2005 - LEB memory problem with calibrations Jan 15 2006 - Hung EIT Sector Wheel, shutter error Oct 18 2006 - Hung EIT sector wheel, like in January, ICE flipped off Nov 04 2006 - Hung EIT sector wheel, during filter change Mar 19 2007 - Hung EIT sector wheel Nov 16 2007 - LEB hung, software froze while commanding then LASCO crashed Mar 31 2008 - LEB anomaly Jun 12 2008 - LASCO rejected a command then crashed Aug 04 2008 - LASCO anomaly Aug 23 2008 - LASCO software error Sep 27 2008 - LASCO anomaly SUMER: Jan 17 1997 - SUMER reboot Feb 23 1999 - Red high temps, powered OFF/ON at SUMER request Feb 26 1999 - wrote ocd to disable thermal monitoring of 2 heaters (76 and 82) Nov 13 2001 - DPU failure - glitch, increase time to 4 minutes before CS needed Oct 26 2002 - SUMER overheating - operator error (retaining current switched on when slit position was adjusted) Apr 15 2004 - SUMER reset due to incorrect write in memory test May 19 2004 - Detector A row problem started Sep 09 2005 - limit violations due to spurious reboot, SUMER powered off Nov 2006 - Detector A declared dead Nov 20 2006 - wrote ocd to enable thermal monitoring for circuit 82 Jul 11 2008 - many limit violations, tried to safe using s_fl_safe and SUMER rebooted itself halfway through - DPU spurious error Sep 21 2008 - DPU error, team had FOT cycle power April 2009 - Raster scan mechanism degradation during April 2009 campaign - sit and stare studies only, but pointing still possible CEPAC: Feb 5 1995 - ESU anomaly - lots of info in folder for this Mar 1996 - LION will be turned off for any MDI door closings decision made May 28 1996 - ERNE communication anomaly - lots of info in folder for this Feb 15 1997 - CEPAC ESU temperature triggers thermal reconfig Feb 25 1997 - CEPAC temperatures limit violations Oct 25 1997 - CEPAC ESU in non-nominal state Nov 21 1997 - CEPAC ESU in non-nominal state Mar 4 1997 - LION failure just like 1998 one, turn off then on Dec 4 1998 - LION failure until power cycling 1999 - CEPAC developed patch for ESU data request errors Nov 2000 - CEPAC ERNE problems, power cycle ESU - lots of info in folder for this Jan 11 2001 - Non-typical ESU data request error Jul 02 2001 - ERNE software update Aug 17 2001 - ERNE Anomaly Type 2 (this is the 6th one so far) Jan 03 2002 - ERNE Anomaly Type 2 Jan 24 2002 - ERNE software update Feb 12 2002 - CEPAC off ??? Jan 14 2003 - ERNE software update Mar 11 2003 - ERNE Anomaly Type 2 Jan 19 2004 - ERNE Anomaly Type 2 (this is the 9th one so far) Apr 24ish 2004 - ERNE Anomaly Type 2 Aug 12 2004 - ERNE Anomaly Type 2 Jul 31 2005 - ERNE Anomaly Type 2 Apr 24 2006 - Anom 2??? Aug 13 2006 - ERNE Anomaly Type 2 May 10 2007 - ERNE Anomaly Type 2 May 22 2008 - ERNE Anomaly Type 2 Oct 15 2008 - ERNE Anomaly Type 2 Dec 15 2008 - ERNE Anomaly Type 2 ---- CELIAS: Aug 18 1996 - CTOF turned off due to high current and temps Apr 16 1998 - MTOF power off May 08 1999 - MTOF sensor emergency Sep 18 1999 - MTOF sensor emergency Sep 28 1999 - CTOF limit violation when HVPS is switched off Mar 01 2000 - MTOF sensor emergency Jul 14 2000 - CTOF limit violation, sensor powered OFF Nov 05 2001 - FSCBUC YH limits, CTOF powered off Jan 24 2002 - DPU failure/reboot Jan 15 2005 - DPU failure Mar 23 2007 - limit violations, CELIAS powered off, DPU reset did folder check, limits MDI: Apr 12 1999 - Motor current violation Apr 14 1999 - Motor current violation Apr 20 1999 - Motor current violation Apr 20 1999 - Motor current violation Mar 23 2000 - ISS open loop failure during a mag Jul 23 2006 - MDI not responding to TC, TM stuck at zero Mar 15 2007 - MDI problem - cause unknown did folder check, limits SWAN: Jul 27 1996 - SWAN off (why?????) Aug 15 1996 - HV power supply on +Z tripped, reset Oct 27 1996 - mechanism in -Z stuck, reset electronic board Nov 28 1996 - CPU reset due to SEU Apr 10 1997 - SWAN Sensor Unit 1 power supply tripped Jul 03 1997 - SWAN Sensor Unit 2 power supply tripped Feb 12 1998 - SWAN Sensor Unit 2 power supply tripped Jul 01 1999 - Sensor Unit power trip Sep 21 1999 - motor stopped Dec 09 1999 - Sensor Unit 2 power supply tripped Jan 03 2000 - bad motor offset in delayed load Aug 03 2000 - -Z sensor inner motor stuck Oct 05 2000 - Sensor Unit 1 power supply tripped Aug 25 2001 - Sensor Unit 2 power supply tripped probably due to a radiation hit Jan 01 2002 - Sensor Unit 1 power supply tripped Jun 24 2003 - came out of lock during keyhole Aug 2003 - motor overcurrent due probably to skipped step by s/w Oct 12 2003 - SWAN s/w crashed for the first time in the mission Oct 21 2003 - Sensor Unit 1 power supply tripped Jan 2 2005 - software/processor failure Feb 28 2005 - current rise b/c radiation caused latchup in motor output Sep 21 2005 - SWAN motor stuck Sep 06 2006 - ESR limit violation, instrument powered off Dec 09 2007 - Experiment LCL monitoring triggered, SWAN powered off, probably rad hit Mar 03 2008 - -Z mechanism got stuck due to rad hit in SU did email check, folder check, limits UVCS: Sep 18 1997 - UVCS unplanned SAFE mode Nov 08 1997 - Unplanned SAFE mode Apr 12 1998 - Detectors unexpected OFF 1999 - Not using Ly-a detector Nov 04 2001 - Unplanned SAFE mode Dec 04 2002 - Unplanned SAFE mode July 2004 - VLD turned off Jan 2006 - Ly-alpha row problem May 2006 - OVI row problem did email check, did folder check, limits (9) check emails current issues file folder logbooks - not doing events list - done check with team cross-check with Bud's s/c anomaly list - done limits binder contingency scripts at end, check with J&B, give copy to FOT