Additional issues arising from Cycle 2017/01¶

Plucked from the eLog.
Issues already fixed are in italics.
Actions/assignments are in bold.

Post-Classification

---

Other¶

• A/C Unit 3 alarm
  • What do we do when John's away - P Masterton, ISIS

• "On Run Status, the Integrated TOF1 (requested) Triggers and Requested Triggers numbers differ by 2: 76810 vs 76812."
  • On various runs we've seen differences of up to 10 triggers. Why? Action DR

• "U/S Tracker - LED system isn't working; possibly a switch got flipped." Check during run-up - checklist for Run-up A.Kurup

• Tracker volume flush read-back broken - add to checklist for run-up
  _
• MOM checklist should include checking that the State Machines are in the appropriate states. (which states are those?) AKurup will write a procedure on how to restart SM's, ALH and then mask out known false alarms ; MOM to work through it at start of running ; MOM to hold the list of known issues. ; Action SBoyd to modify MOM checklist

• "EPICS reports that A/C unit 2 - which I believe is actually the reading from unit 3 - is reporting 26.3°C suggesting that the East end of the Hall is getting warm"
  • We need to understand the Hall temperatures: numbers presently range from 22 to 27 ° Action AK to check environment monitoring

"We have selected "LH2 Empty" in Run Control: it STILL wrote "LH2" into the CDB! CDB viewer's reporting "LH2" as the material and "Empty" as the shape which I think is the wrong way round - not sure if this is the viewer or Run Control." Need to check CDB viewer displays correctly (DK/JM) ; Need to update for Neon (which is what we actually had) DR

• "discovered during this that the overtemperature alarms had been LEFT SILENCED at some point, without this being passed on to the next shift."
  Shifters silencing alarms rather than acknowledging, and/or not expanding the tree to identify the individual alarm MOM/SteveBoyd
  • e.g. If you silence the Proton Absorber Status because it alarms when changing Beamline settings, how will you know if it droops open during the run (c.f. December)?
  • Status of Shift Operation Checklist? (SBoyd) Action on SB to add alarm check to shift changeover checklist

• "MOM/DC Configuration Checklist" omitted add to MOM/DC handover Action SB

Controls and Monitoring¶

• We noticed today that the TKU, TKD, SSU, SSD and FC Alarm Handlers were all running but disabled 'D'
  This should now be fixed: State Machines and SM IOC's restarted; ALH Groups have masking "Force PV's" which have been re-set. Action AK.

• Calibration of compressed air pressure sensors - in progress

• Readouts from A/C units 2 and 3 swapped

• Massive number of bogus alarms (e.g. on a PV that is inValid):
  • "Tracker 1 Cryostat - vacuum gauge - alarm as reading is inValid"
  • "I have disabled the "QPS Signals", "QPS Interlocks" and "Interlocks" trees on the SSU Alarm Handler, to match the SSD Alarm Handler State." - should be covered in the SM fix
  • "I've also disabled the alarm on the Neutron Monitor - which is no longer connected so gives a permanent error." - Taken out completely
  • "AC5 status disabled in ENV alarm handler" - Unit will not be networked, PVs to be removed from spreadsheet
  • "Henry has disabled the following in the alarm handler in consultation with MOM-Ed:
    • TKU: GHe, Cryo1 vacuum
    • TKD: GHe, Cryo1/2 (really 3/4) vacuum, Cryo2-LCRB-temp2 (actually Cryo4LC-temp2), Cryo2-LCRB-AFEheater2 (also mislabelled)"
      GHe monitoring boxes restarted; vacuum gauges replaced and the readout isn't working (AK/CMw); temp and AFE: need to check again; mislabelling fixed
  • "I've disabled the Q4 and Q7 MPS water flows in the Alarm Handler because of the false alarms ... I am suspicious of the fact that we only get these false alarms on those two supplies" - Comes from the readback glitches - Time filter settings have been re-applied and should prevent the alarms
  • FCD Load Cells - see below - HW disconnect - not a C&M thing
  • What is the plan for SSU CC1 and 2? see below - Need to check cable - comms issue between compressor and IOC

• "the limits on SSU CC3 High-Side Pressure have been adjusted to get rid of the frequent alarms we were getting a couple of days ago. (now just to get CC1 and 2 sorted out)."
  • What is the plan for SSU CC1 and 2? - Need to check cable - comms issue between compressor and IOC

• "SSU Alarm Handler DISABLED in all channels" - should be covered in the SM fix

• "Run Control is hung up; restarted in separate terminal" - Run Control has long-standing deep-seated issues

• "Will also check ... reported water leak in trench. "
  "About 13:00 Ajit restarted the Environment IOC - this should have got the air temperature and other sensors going."
  • SecurityProbe system not being read out by the IOC - needs some sort of heartbeat alarm: check the Read-back Status of the PV ; Stale info from SecurityProbe system has safety implications

• "Noticed power supply for EMR Single Anode PMT # 26 was tripped off and probably had been for a while."
  "Noticed also that in the Detectors ALH the entire EMR tree is Disabled, although Ed did put the Detectors into Running yesterday." - should be covered in the SM fix
  • Not clear when the ALH EMR tree was turned off. Who is responsible for checking the Alarm Handlers over before running: experts or C&M? - see below
  • MOM checklist should include checking that the State Machines are in the appropriate states. (which states are those?) AKurup will write a procedure on how to restart SM's, ALH and then mask out known false alarms ; MOM to work through it at start of running ; MOM to hold the list of known issues. ; SBoyd to oversee MOMs

• "Roof water temperature caused a red alarm, upper limit. - Returned to normal before a cause to be found. Assumed to be a measurement fluctuation."
  • We see glitches in the roof water PV MICE-SER-H2O-01:TPrimary: 6:06 31 May 2017 Checked Archiver - glitch in output variable isn't accompanied by any glitch in its inputs. In progress.

• "We have selected "LH2 Empty" in Run Control: it STILL wrote "LH2" into the CDB! CDB viewer's reporting "LH2" as the material and "Empty" as the shape which I think is the wrong way round - not sure if this is the viewer or Run Control." Need to check what is being written out (and displayed) by Run Control (AK) (related CDB action below)

• "I just realised that one reason the Alarm Handler is such a pain to use is that the software is incapable of scrolling and making a beep noise at the same time!" Test if scrolling and beeping works on miceiocpctest - VNC? On miceiocpctest you cannot scroll while is beeping (PF).

• "There was a major alarm from DATE at about 1360 spills/ 83k triggers, but by the time we found it we couldn't see any cause or error message. No sign of a trip in the beamloss strip chart. The transfer of the ISIS beamloss numbers from ISIS into EPICS is still very unpredictable - sometimes we get the values updated on 2 successive spills, sometimes the values are stay frozen for 9 or 10 spills." - Can't see a 10-spill plateau in the Archiver Review the limits for the BLM PVs

• Repeating alarms need thresholds adjusting putting them on a strip-chart makes it easier to see if there's a real problem
  • "SSU CC3 High-Side Pressure"
  • DS PSU voltage Check with Mike
  • "Q4 and Q7 are showing a consistent warning on the MPS water flow" #1880
  • "alarm from Q2 deltaT" #1880
  • "Q8 reports temperature change warning alarms." #1880
  • "Q8 has a warning on the voltage" #1880
  • "Supply air pressure reach 6.8 Bar before returning to normal." - increase Limit Hi 7.0, HiHi 7.2
  • "Alarms in beam line: quad delta T touches limit" : #1880

• "Heartbeat alarms for miceecserv1 and miceraid 5. The help pop-ups flash up an illegible error message." - Now Fixed
  • In Alarm Handlers, we need working pop-ups ...
  • Changed during run?

• "After acknowledging the miceraid5 alarm, the PV remains shown as red, but the alarm keeps returning. This seems to be because the Heartbeat re-sets the status to zero before making the next test, which fails again. Probably, this is Wrong as it operates differently to the majority of other alarms."
  • Is the Heartbeat test operation correct? AK/Paolo to see if it can be made consistent ; SBoyd & AK to decide on procedure for dealing with persistent alarms

• "temperature warnings from the CS1 VME crate."
  • Need to clarify temperatures of crates - are they really warmer than expected, else tweak limits AK to track down appropriate expert. ; There is an action on getting the crates cleaned out

Beamline¶

• Beamstop circuit 1 leak - Henry / Hall Manager

• "We have pulled out the target one notch to 35.75 mm BCD, as the losses are now much spikier since the work ISIS did this morning"
  • Beamline Co-ordinator has raised with ISIS...

• DS Shield Pressures DS team cleaning gas over summer
• "Decay solenoid supply pressure PI21 - fluctuated back to normal"
  • DS needs gas cleaning and TLC DS team cleaning gas over summer

• Beamstop Restraint possibly has stripped bolt - Henry / Hall Manager

• Tweak MPS temperatures difference limits: #1880

• Target laser saga, and control PC / R78

• Question from ISIS about the MICE 3ms spill gate - why not 1 ms?
  • Possible balance between inefficiency from dead-time vs. activation of ring for no data
  • How are the good muons distributed within the MICE spill?