Project

General

Profile

Actions
History

Ops/Comm Scheduling meeting : 7/2/2017

Location : CR21 in R1 at 2pm

Phone access : ReadyTalk number
US 866-740-1260
UK 0800-496-0576
Swiss 0800-70-53-51
Italy 800182592
Access Code: 2155402

List requested yesterday:

Below, at Craig's request is the list from memory of some of the issues seen last running period:

  • Alarm Handler: what is the difference between "E" and "V" white alarms? Not covered in shifter training. Can they really just be ignored? AK to explain; add to shifter training
  • Q2 alarms on negative voltage
  • #1878, #1891 Trench water pump control - pressure appears stable; shifter lockout in progress
  • #1881 Water system PVs not being archived - Services/Environment/Computing spreadsheet
  • #1906 Alarm Handler disabled for DS - Alarm Handler checks should be added to Run Preparation checklist: PF to send list of Alarm Handlers to SB
  • Alarm Handler immobilised by mouse middle-click middle mouse button immobilised
  • #1880 Alarm Handler not reacting to conventional magnet interlock - AK aware
  • miceraid5 disk full - auto-cleaning already requested at CM44 - done manually, in progress (EO)
  • #1903, #1881 micestore disk full, also needs to be in Nagios warning checks in place but need testing (hence old data left in place)
  • #1881 File Compactor failure - shifters reported only the "previous error" message; should be in Alarm Handler. Move File Compactor output to top of RC screen? Also waiting on Services/Environment/Computing spreadsheet
    • What was the original error? DR: micestore disk full. Has been dealt with.
  • Trigger Mismatch (nEvts 0!=33). Unpacking exception in void InputCppDAQData::getCurEvent(MAUS::Data *data) errors from OnRec - Had a series of runs with a lot of these. If we get too many then DATE should stop itself. EO: we do need to keep an eye on this
  • DS pressure instability (initially unnoticed as Alarm Handler disabled) - MT investigating
  • Proton Absorber 29mm slab not registering raised - not reproduced dodgy electrical connection?
  • Diffuser iris 4 not opening - not reproduced, SSU high-field test still needed
  • #1888 air pressure reading zero in diffuser GUI - fixed ad-hoc by PMH; is fix in production? C&M to close ticket when done
  • Diffuser GUI not controlling/displaying actual diffuser state - believed fixed by PMH
  • DS current - the dynamically generated Alarm Handler limits excluded the set value - AK aware
  • #1875 insufficient water cooling as Chiller 1 had tripped - state now visible and controlled with new Manager
  • target1ctl dead and so no spare target control PC; target2ctl only had older software and so not publishing all data, especially laser levels - new target1ctl in difficult progress (EO)
  • Target laser levels decaying - at least one instance of target shutting itself down from this - fibres swapped
  • Target RATS daemon crash - add "Plot From Memory" as a PV and to ALH?
  • Target VME crate overheating - RF Hall too warm, outside MICE control
  • VME crates in Hall overheating; crate monitoring looking at wrong crate Services/Environment/Computing spreadsheet, GUI needs fixing
  • Beamstop circuit 1 leak - no redundancy
  • PPS search drop - not reproduced, cause not as claimed - in fact the search was not actually dropped, contrary to implication of eLog 270 "The MOM was informed and will come down and help us re-search the Hall."
  • PPS search records incomplete - last record sheet in binder was mid-October [confirmed after meeting]
  • Site Emergency: procedure for shifters during run - should be added to emergency actions posters (SB). Note there are two different posters in MLCR
  • Data Mover failure on large files increase timeouts
  • Tracker calibration procedure not carried out - should be added to Standby-to-Running checklist (SB)
  • Tracker calibration procedure not clearly explained to shifters at training, resulting in invalid calibrations - need to mention stopping the target: update instructions (EO); covered by Standby-to-Running checklist during 16-hour running
  • Tracker cryo temperature alarms - in progress; new cold head?
  • Tracker cryo: AFE Heater alarms: how to handle known false recurring alarms - each alarm instance should still be recorded in logs, but need to improve shift handover to save repeated escalations
  • Hall probe failure causing C&M to misunderstand Channel settings - in progress
  • QPS and windows updates debacle
  • MOM DC Configuration Checklist eLog tab has out-of-date info - DR: it is NOT obsolete. Add reminder to the paper DC-MOM handover forms to fill it out (AN/DC). [It sounds like it needs post-hoc updating, also to include the Tracker Helium issues, but no clear who that action should be on]
Since then:
  • compressed air failover - this failed a couple of weeks ago, after which the DS fridge turbine refused to restart - The service engineer failed to clear the "service" status on the compressor after his work; JG now investigating a possible change of contractor. JG will do a failover test before the DS is powered.
  • webcams - images not visible on micewww, probably due to micenet lockout #1399 but low priority
  • boots in controlled-entry airlock - stop using boots as matter-of-course; only wear for specific tasks (AN). Move old boots elsewhere (JG)
  • ISIS variables and status not in EPICS - restart IOC. Possibly need to get ISIS end running.
  • Escalation of issues during running: Shifters call MOM
    • For the Beamline (Target -> Diffuser) the MOM calls the BLOC (else Henry) and the BLOC calls the experts (else Henry)
    • For other subsystems the MOM calls the experts
  • Neutron monitor & restart docs - MICE to take the CMS ION to Poole to be fixed ourselves. MLCR needs to know how to restart remote viewing of the temporary monitor.
  • Expert list - update (SB). There are (still) 3 separate lists in the MLCR
  • Data Mover certificate expires 11 Feb.
  • Quiet target bleep - check NIM crate voltages
  • [Q6 losing overtemperature limit from PV]
  • [Instructions for resetting Data Mover token - in progress]

I've crossed out some I think are done but have to go and test the diffuser now.

  • Operations :
    • Monday 6th : Preparation meeting with Ed/Ajit/Colin/Josef/Others to go over readiness for running
    • Tuesday 7th (subject to meeting on Monday) : Bring up magnets to running configuration, take some pulser data and take magnets down
    • Saturday 11th : Expert led run without solenoids + Beam tune
    • Monday 13th : Triage actions arising from expert led run
    • Tuesday 14th : Bring up magnets in the evening and monitor overnight by reserve shift.
    • Wednesday 15th : Data-taking start at 08:00
  • Shift signup for first half of Feb is viable now
Shift Block ID # signed up # shifters still needed
1 6 0
2 5 1
3 5 1
4 5 1
  • BLOCs are needed : Feb 27 - Mar 1
  • AOB:

Minutes

  • CC Magnet status
    While preparing the QPS an external trip light needed resetting by closing the contactors, but the latter caused the QPS to see voltages. Discovered SSU M1 was at 180 A. Believed to be because controller was already trying to run current (set manually last week) - with contactors open controller had set to max. voltage. Need to ensure controllers are all set to zero, at start of checklist.
  • Hydrogen status (V Bayliss)
    Decision has been made to install in May. Will do a test cooldown in R9 this week for practice/to repeat the instability phase, then start taking apart next week. Will move Hydrogen turret only to Hall and install in FC1 which is already there, starting 7th March after data run.
    Controlled Entries will be possible during the commissioning and operation phase, but the rules thereof need finalising.
  • Issues arising from previous running
    Went through shutdown task list at the Readiness Meeting, ran out of time.
    Issues arising from previous running: went through list above (see here for details)
    [These are only those Henry knows about: beamline or happened while Henry on shift. Estimate the full list is 3 times as long...]
  • The Alarms PC in the ISIS MCR has been re-instated, in response to last week's water glitch.
  • The Beamline IOC resets the Proton Absorber to zero (i.e. restarting the software causes a uncommanded physical movement)
    Should have a procedure for keeping IOC while people are working on the PA: e.g. central NFS lockfile for IOCs, create manually to prevent starting.
  • Operations :
    • Monday 6th : Preparation meeting with Ed/Ajit/Colin/Josef/Others to go over readiness for running
    • Tuesday 7th (subject to meeting on Monday) : Bring up magnets to running configuration, take some pulser data and take magnets down
    • Saturday 11th : Expert led run without solenoids + Beam tune, beam time 11:00 for "few hours"
    • Monday 13th : Triage actions arising from expert led run; BLOC training
    • Tuesday 14th : Bring up magnets in the evening and monitor overnight by reserve shift.
    • Wednesday 15th : Data-taking start at 08:00
  • AOB:
    • Behaviour of Tracker Cryo 1 changed significantly at 08:00 today - any ideas?
      Hall was locked - first entry 08:05, so no-one in Hall. No sign of water glitch
    • Data breach of STFC dosimetry service - if you get sent an email, it's NOT spam.

Actions

  • JBoehm to get SS operating info from PHanlet (ongoing)
  • JBoehm to get/impose consensus from SC Magnets group on running of "Central CC GUI"
  • DC to collate list of issues arising from previous running
  • Status of miceraid5 disk cleanup (stands)
  • DC to ensure Health Physics have checked temporary neutron monitor (stands)
  • SBoyd - ensure MICE-wide discussion of MOM C&M panel
  • Henry to get gaps in BLOC rota plugged
  • Also actions in list above

Updated by Boyd, Steven almost 7 years ago · 22 revisions