SOC Action Items for Experiment Safing - initial, date, time (UTC) when done -------------------------------------- 1. If after hours and SOC gets call at home about ESR, go into work _______ right away, stopping by IMOC to tell FOT the SOC is at work You might want to call another SOC in but if the hour is late, let one SOC sleep late to be rested. If Tero is not around, call Bernhard *** Be thinking about if any instrument was in a non-nominal state prior to the ESR b/c this will change the below plans - check ins.status for info on instruments. Also be thinking of what submode we should go into (remember that when UVCS is on the OVI detector then the UVCS Ly-a detector is off so they will cool down faster - but no extra actions are needed) (remember that SUMER and GOLF substitution heaters are normally on) *** Note: ARTU & SRTU have been reconfigured (consequences with MDI & VIRGO) See info files on RTU reconfigurations *** Note: All limit violation sheets will be ignored 2. Email the alerts email distribution list saying that we are in ESR __________ and they will get lots of limit violations. Note: Remember to call teams (see below for list) within an hour or so 3. Get out the Gyroless Scripts binder as well as any OCDs that _______ have not yet been incorporated into the scripts (check the undone OCD listing at the front of the OCD binder for these). Also verify that the versions we have are the most recent ones. Check that the sections below on FOT action items are using the most recent versions. Log all details in logbook Start an ESR folder and put copies of all papers and emails in it 4. When reasonable, ask FOT for some information (or get it from _______ the POCC terminal): CDS value of CKCMODE Who was on the offpoint flag list before the problem (should be known) 7. call SWAN on-call person, tell them when llockin was run (get from email) _______ 8. call on-call LASCO person _______ (should normally come in to finish closing LASCO doors & restart EIT observing during 20min NRT and give permission to be removed from ESR flag list when EIT is observing again - Step 16) 9. call the on-call MDI person _______ (should come in to restart structure during 20min NRT - Step 16 ESR Recovery Script) 10. call the on-call UVCS person and notify of ESR _______ 11. call the on-call CDS person, tell about CKCMODE status and heaters _______ 12. if SUMER is observing, call the SUMER planner (local or Germany), _______ ask them about what submode change they want to be in for recovery Note: if SUMER not observing, send email about the submode change 13. Email GOLF with a reminder about their GTREARAD violations in Low Rate ________ (if it happens) 14. check with Bernhard and see if we should call Europe teams - the _______ default is NOT to call them 15. call rest of teams if Bernhard says so _______ CELIAS, CEPAC COSTEP, CEPAC ERNE, GOLF, VIRGO 16. email minutes list about ESR ______ 17. check with instruments when they come in about their temperatures _______ - do they need substitution heaters turned on (if so, is the default setting okay - default substitution heater settings in ESR Notes binder) * NOTE: ESR Initial Operations Script Step 8 safed them (except when noted) UVCS - check with FOT that heaters are on and at what setting ______ CDS - check with FOT that heaters are on and at what setting ______ CELIAS STOF - check on email if/when delayed command was uplinked ______ MDI - check with MDI that iss loop opened, structure stopped, _______ VCID -> 0 (done in Steps 1 & 8) SWAN - check email for the time the safing happened ______ SUMER - check on email that they were removed from ESR flag if observing _______ VIRGO - check with FOT that their 2 doors were closed, their _______ time tag loads were not uplinked (both Step 8), and that their time tags onboard were deleted (Step 4 embedded in k_ttag_ck) MDI, LASCO, UVCS, SUMER - when do you want your offpoint flag enabled? ______ * all of the FOT items may have to wait until FOT has more time 18. Follow scripts and verify all instrument actions have been performed, noting activity times in SOC logbook, then email teams. ______ Send emails to special_ops email distribution list, which has: cds-esr, celias-esr, cepac-esr, eitlasco-esr, golf-esr, mdi-esr, sumer-esr, swan-esr, uvcs-esr, virgo-esr 19. Have teams turn in limit violation sheets for any limits that are violating unexpectedly. ______ 20. Note that Step 16 of the ESR Recovery Script has the ESR flags enabled; OBT distributed to MDI, LASCO, CELIAS, CEPAC, SWAN, UVCS; v_fl_synchr; and 20 minutes of NRT for MDI to restart structure, EIT/LASCO to finish closing LASCO door and restart observing, and SUMER to adjust heaters if wanted. Then the ESR flag is disabled for EIT/LASCO. Make sure this step happens after the final RTU reconfiguration has been done. Make sure that you tell MDI, LASCO/EIT, SUMER when we have our 1 hour before NRT notice from the FOT. If LASCO/EIT is removed from the ESR warning flag list, they should always load a full resolution plan. Otherwise, subsequent ESRs will not flush, EIT will get behind, and when trying to catch up it may become overheated. This will never happen with a full resolution plan. 21. Check with FOT and see what time VIRGO's v_fl_synchr was run in the ESR _______ Recovery Script (Step 16) 22. FOT will upload VIRGO time tag commands that were lost and will ________ send a command to VIRGO to execute 15 minutes after the first midnight UT following v_fl_synchr SOC Action Items for Experiment Recovery - check off when done ----------------------------------------- - Find out what version of recovery file CELIAS wants to use _________ (contact Ops Coordinator first if you can) - Tell SWAN that we will run n_fl_manrec unless they say not to _______ (lock-out done 2 hours after last burn) - Verify with FOT the UVCS default heater settings AFTER they are recovered _______ - Find out if MDI needs VC2 to do leg alignment _______ - Find out when CDS and UVCS will open doors and tell LASCO _________ - Find out when CDS, LASCO, UVCS, SWAN want their limits monitored normally _______ (ESR Initial Operations Script switches them to ESR limits) _ If/when 1hr or more of NRT becomes available, CDS may want to do a partial recovery by having the substitution heaters disabled (NOTE: OE needed) and then go to READY-BOTH mode. When they are in that state, with doors still closed, the ESR flag should be disabled. If another ESR happens after that, before a full recovery & ESR flag enabling, NO ACTION will be required for CDS (FOT included). - Find out who wants to have their offpoint flag enabled (Man. End, #5) _______ (choices are MDI, LASCO, SUMER, UVCS) As a rule, SUMER does not want the offpoint flag enabled (Klaus Wilhelm & Dietmar 4/21/04). The list will be the same as whoever was in it before the ESR but verify this with the teams - Follow the instrument recovery scripts and inform teams as done * Set up POCC terminal: Set up the POCC terminal in the ECS office so that teams can monitor their housekeeping data. We will be in Low Rate Telemetry, so only bring up instrument pages that are of: *_exphk_* syntax. CDS c_exphk_tm1, c_exphk_tm2 CELIAS f_exphk_tm1 GOLF g_exphk_tm1 CEPAC h_exphk_tm1, h_exphk_tm2 EIT e_exphk_tm1 MDI m_exphk_tm1, SUMER s_exphk_tm1 SWAN n_exphk_tm1 UVCS u_exphk_tm1, u_exphk_tm2 VIRGO v_exphk_tm1 A *_exphk_* page exists for each instrument team except LASCO. Other pages may be displaying stale telemetry values so do not bring up those pages. By stale, we mean that the values have not updated since prior to ESR occurance and are designated with an "S" next to the parameters. Don't look at these. The table below outlines instrument responses, actions, etc. during an ESR. ------------------------------------------------------------------------------------ FOT Action Items for Experiment Safing - this is for reference only -------------------------------------- * Note - check the version numbers for the below listed scripts * As soon as the ESR flag is triggered, the monitoring is automatically disabled (no change to recipient list) and the offpoint flag is disabled ESR Initial Operations Script, V9.1, CS-06B, Gyroless Scripts Binder Appendix C1: configure ground system for ESR and transition to medium rate telemetry use ESR limits configure MDI for transition from LR-MR (m_fl_lrtomr) ask SOC what submode we want to go into (default is last active one) collect data start ranging configure AOCS for ESR operations delete VIRGO on-board time tag commands (in k_ttag_ck) ground based roll control esr data collection yaw/pitch braking experiment safing: adjust UVCS substitution heater 84 to 60% and enable heater read value of CKCMODE for CDS and then based on the number, adjust their substitution heaters and disable substitution heater monitoring if required by script uplink delayed command file CEL_ESR_SAFE to safe STOF record SWAN telemetry remove SUMER from ESR warning flag list safe SWAN (n_fl_esrllin) close VIRGO doors and don't uplink new time tag loads FOT Action Items for Experiment Recovery ---------------------------------------- ESR Recovery Script, V12.2, CS-06A, Gyroless Scripts Binder Appendix C1 Step 16: after we have gone to CRP, FPSS is commanded to normal, ARTU is recovered (Step 14 is ARTU reset), COMS backup is recovered (Step 15), then: re-enable monitoring of ESR flag distribute OBT to MDI, LASCO, UVCS, SWAN, CELIAS, CEPAC synchronize VIRGO with daily pulse (v_fl_synchr), uplink time tags, send command to open covers enable NRT for 20 minutes for instrument safety check and to restart structure (MDI), finish closing doors (LASCO), restart observing (EIT), to change heater settings (SUMER) Remove EIT/LASCO from ESR flag upon permission NOTE: do only after the final RTU reconfiguration is done then go to RMW then NM Maneuver End Script, V15.0, Gyroless Scripts Binder Appendix A Steps 5 and 6: reset offpointing flags with whomever was on there before enable experiment offpointing monitoring close MDI ISS loop with permission acquire more tracking stars if SWAN was safed, unlock SWAN mechanisms 2 hours after last thruster firing. NOTE: SWAN nominal limit set is to be taken into use immediately before and no earlier than 5 minutes before the mechanism unlocking. TSTOL procedure n_fl_manrec is to be run after at least a 3 minute wait after the mechanism unlocking. Send CELIAS recovery load enable NRT open CDS door upon direction ESR Instrument Recovery Script, V5.0, CS-06C, Gyroless Scripts Binder Appendix C1 SOC directs when/what step to use Step 1: CELIAS - uplink STOF recovery load if needed Step 2: UVCS - execute U05 Step 3: CDS - disable substitution heaters if they were enabled, adjust duty cycles, enable heater monitoring if it was disabled Step 4: LASCO/EIT - enable offpoint flag, enable ESR flag Step 5: SUMER - enable offpoint flag, enable ESR warning flag if needed Step 6: SWAN - run n_fl_llockout then n_fl_manrec (may need to add going back to nominal limits here) Step 7: VIRGO - upload any future time tag commands that were erased by ESR, run v_fl_synchr, send cover open command Once a team has recovered and wants its limits monitored normally, we tell the FOT. This needs to be determined for: CDS, LASCO, UVCS, SWAN ------------------------------------------------------------------------------------ *** ESR flag enabled to all instruments on the default list in Step 16 of CS-06A unless otherwise specified in the scripts ***
| CDS | Upon receipt of ESR flag |
CDS door will close and they go to a low power mode (standby mode), and CDS will gradually cool down. |
|---|---|---|
| Actions During ESR |
CDS stays in standby mode during ESR, however temperature monitoring must
continue and sub heaters may need adjusting upon direction from team.
To prevent CDS from becoming too cold, the FOT will turn on substitution heaters
according to the status of CKCMODE parameter (in ESR Initial Ops Script, Step 8):
|
|
| ESR recovery actions |
CDS will not open doors until 24 hours after last momentum management thruster firing was completed. CDS recovery will be done via NRT. Substitution heaters must be disabled prior to CDS beginning its recovery. It will take ~6 hours for CDS to become thermally stable and resume science observations once recovery has been completed. | |
| CELIAS | Upon receipt of ESR flag |
Instrument receives ESR flag, but is currently programmed not to respond to the flag. All detectors remain operational. |
| Actions During ESR |
Some ESRs have taken place during a planned spacecraft maneuver when STOF was already in a safe configuration. It is not necessary to determine the state of STOF, prior to sending the delayed command file. The safing file can be sent a second, third time, etc., with no problem at all. Since CELIAS can no longer use its GSE computer, SOCs may be asked by the team to check the status of fsdesrd and fsdpoc with expected values for fsdesrd=0 and for fsdpoc=1. SOCs may also be asked to check the housekeeping web pages ( http://umtof.umd.edu/hk and http://umtof.umd.edu/hk/hk.used) |
|
| ESR recovery actions |
Check with STOF team if they want to submit an updated recovery file, as their configuration changes from time to time. Recover CELIAS STOF via uplinking: CELnn_RECOVR.DEL no earlier than 6 hours after last thruster firing, where nn is the version number of their recovery file. (presently: CEL12_RECOVR.DEL, 1 July 2008), (ESR Instrument Recovery Script, Step 1 and in ESR Recovery Script, Step ). NOTE: if the recovery load is outdated, it is not a problem to uplink it anyway - the team will send another delayed load later to fine-tune the instrument if needed. | |
| CEPAC | Instruments (COSTEP and ERNE) receives ESR flag, but does not respond to it. Note: LION shall be switched off for any MDI door closures via h_fl_loff.prc 1 to 60 minutes before MDI door closure, and switched on via h_fl_lon.prc 1 to 60 minutes after MDI door opening | |
| EIT | Upon receipt of ESR flag |
Instrument receives ESR flag. Their buffer gets flushed and images will stop being taken. NOTE: if LASCO was already safed they may have been removed from the ESR flag list and thus wouldn't get the ESR flag or be flushed. |
| Actions During ESR |
In Step 16, give NRT to EIT to restart observations and then remove LASCO/EIT from ESR warning flag list. NOTE: If LASCO/EIT is removed from the ESR warning flag list, they should always load a full resolution plan. Otherwise, subsequent ESRs will not flush, EIT will get behind, and when trying to catch up it may become overheated. This will never happen with a full resolution plan. | |
| ESR recovery actions |
The offpoint flag is re-enabled in the ESR Instrument Recovery Script Step 4. The ESR flag will need to be re-enabled if it was disabled earlier. | |
| GOLF | Instrument receives ESR flag, but does not respond to it. GTREARAD may violate when in Low Rate. (should be fixed with next database update) | |
| LASCO | Upon receipt of ESR flag |
Doors close as a response to expected to ESR flag. (They may not close all the way and sort of "bounce" a bit but for the most part they close). |
| Actions During ESR |
The FOT will enable NRT after MR telemetry has been acquired (in ESR Recovery Script, Step 16). During NRT they may send some commands to check instrument and close the doors tightly. | |
| ESR recovery actions |
Will open doors in NRT 1 hour after CDS and UVCS have, and recover in NRT. The offpoint flag is re-enabled in the ESR Instrument Recovery Script Step 4. | |
| MDI | Upon receipt of ESR flag |
Instrument receives ESR flag, and the MDI ISS loop opens in response to it. Then the MDI ISS loop is open, structure is stopped, and the VCID is set to 0 in ESR Initial Ops Script, Steps 1 and 8. Their door remains open. Their LOBT may be disrupted when the AOCS RTUs (Attitude and Orbit Control System Remote Terminal Units) reconfigure in either direction, stopping "structure" their basic science program. |
| Actions During ESR |
During the period of NRT after MR telemetry acquired in Step 16, they like to send some commands to check instrument and restart their structure program (helps thermally * slightly * to have it running). | |
| ESR recovery actions |
MDI may need to do a leg alignment in VC2. Afterwards, MDI can close
their ISS loop in NRT if they didn't have the FOT do it after
maneuvers. Their LOBT may be disrupted when the AOCS RTUs (Attitude and Orbit Control System Remote Terminal Units) reconfigure in either direction, stopping "structure" their basic science program. Note: the TSTOL procedure to open the ISS loop is m_fl_issopen, and the TSTOL procedure to close the ISS loop is m_fl_issclos. |
|
| SUMER | Upon receipt of ESR flag |
If door is open and ESR flag is enabled, door will close in response to ESR flag. FOT will remove SUMER from ESR warning flag list (in ESR Initial Operations Script, Step 8). |
| Actions During ESR |
In NRT in Step 16, SUMER may modify their heater settings. Note: they may cool down too rapidly for this to be plausible. s_fl_safe can be used if they are too cold before the NRT/Medium Rate is available. | |
| ESR recovery actions |
Will open door, etc. in NRT when ready to resume observations. ESR and Offpoint Flags will be enabled as part of the NRT to open the door (RPR is s_fl_flag_ena.prc) | |
| SWAN | Upon receipt of ESR flag |
When the ESR flag is automatically sent to the instruments, SWAN should receive it and place both sensors automatically into lock position. In case the distribution to SWAN was accidentally disabled, the procedure forces the action. The flight software will ignore the command if the sensors were already safed, so no harm can be done. The last command of the new procedure version is a copy of the old disabrest. |
| Actions During ESR |
Note: n_fl_esrllin may not always work if their motor position calibration is off. If SWAN has a bad motor offset they can send a delayed command to put them in IDLE (with the heaters still on) and then they can send a delayed command to restart things - SOC would do OCD's |
|
| ESR recovery actions |
Go to nominal limits. Run n_fl_llockout procedure (in Maneuver End script (and also Step 6 of ESR Instrument Recovery Script)) and run procedure n_fl_manrec no earlier than 2 hours after the burn. Note: do not run n_fl_llockout until at least 2 hours after the last thruster firing. | |
| UVCS | Upon receipt of ESR flag |
Will receive the ESR warning flag. UVCS also has internal safing mechanisms. Upon receipt of ESR flag, the door will close, the detectors will be turned off and they will get cold quickly. |
| Actions During ESR |
FOT will turn on UVCS TSU substitution heater circuit #84 to 60%
(in ESR Initial Ops Script, Step 8)
They also have contingency script U-02, to decrease duty cycle by one decrement of 10% if UTTHEC1 or UTTHEC2 violates red high limits If they are still too cold, they may turn on their detector and internal heaters and have the FOT turn off their sub heaters |
|
| ESR recovery actions |
UVCS will usually perform some preliminary NRT commanding and then ask for their substitution heater to be switched off. When off, verify the default settings of this heater to be 60% on both the prime and redundant sides (in ESR Instrument Recovery Script, Step 2). Further NRT commanding will be needed and it will take 1-2 days for UVCS to become thermally stable and resume science observations. The offpoint flag is enabled upon direction (in ESR Instrument Recovery Script Step 2). | |
| VIRGO | Upon receipt of ESR flag |
Instrument does not receive ESR flag. However, will fall out of sync with Daily Pulse when the AOCS Remote Terminal Unit (AOCS RTU) becomes reconfigured. |
| Actions During ESR |
VIRGO time tag commands are all deleted (Step 4 of ESR Inital Operations
script). VIRGO covers closed and a note for the FOT not to uplink time
tag commands (Step 8 of ESR Initial Operations Script).
Note: If VIRGO ever wanted to close SPM and PMO covers,
the FOT would run the procedure v_fl_covclose.prc The procedure v_fl_synchr is run after AOCS Remote Terminal Unit (AOCS RTU) reconfigurations (Step 16 of the ESR Recovery Script). E-mail to the VIRGO team (VIRGO_latchup) the time the action has been performed. |
|
| ESR recovery actions |
If VIRGO had to safe, they would recover with a delayed command file.
FOT uplinks all future time tag command loads deleted from the time
tag buffer (Step 7 of ESR Instrument Recovery Script and 16 of ESR
Recovery Script). FOT will send a command to open PMO6-A cover 15
minutes after midnight after daily pulse has happened (Step 16 of ESR
Recovery Script). (KNGEVZML, 800C) |