Step 4 Readiness (summer 2016) for MDR7¶

Raw list includes some things marked as already done as they need re-doing (e.g. new unpacking including Hall probes) and a lot of repetition (and some items may even be done already).

List of issues at summer 2016, culled from

Step 4 Requirements
24 June Readiness Meeting
28 June Operations Meeting and issues from MDR7
21 June Operations Meeting
14 June Operations Meeting
7 June Operations Meeting
February 2016 List

Beamline (c.f. #1854 )¶

Target

• Target Alarm handler #1613 update 22 (CAM)
• ISIS data items #1669 (CAM)
• Activation Study (activation seen: limit of 2 V at MS/64 stands)
• Beambump tune
• Updated target manual (EO/JL)
• Updated target checklists (Henry)

Conventional Magnets

• review existing water flows and temperature range (MBC)
  • need limit in control software? (CAM)
• Calibrate D1 & D2 current reading and update IOC software (DL)
• Q1 MPS start-up recalcitrance (DL) Q1 MPS replaced
  • Replaced the PS. Does it have correct polarity?
• Q1 alarm on voltage at zero amps (CAM)
• Q4 oscillation below 10A and anomalous ramp rate (DL)
• Trench water leak / Q5 water leak : Q5 has been fixed.
• Q6 leak : too fragile, wouldn't touch it - shifters to watch
• Leak sensors and drip trays
• Water, temperatures and MPS water flows in ALH (CAM)
  • *Q6 temperature alarm limits missing!*

Decay Solenoid

• DS readback current requires calibration (DL)
• Overheating investigation (DK/DL)
• Calibrate current reading and update IOC software (DL)
• 120A trip investigation (DK/DL)
• Replace final battery cell (DL - confirmed still outstanding )
• Diodes in BBU circuit (DL)
• Boosted air (MBC)
  • Trench sensor (awaiting read-out)
  • Compressor sensor
  • Backup compressor for DS
• Linde PC install: awaiting validation and IOC (DK) - licence issues
• DS ALH/archiver corrections #1710 (CAM)
  • DS Fridge instability : Has been investigated by Mark. Monitor.

+ Proton Absorber / Beamstop / Diffuser

• Proton absorber sticking : Hasn't been reproduced lately - shifters will need to monitor it.

Other

• BLOC training and experience (MBC)
• BLOC roster
  • Put BLOCs in CHEESE (Steve) : Done

Detectors, Monitoring and Control¶

• Tracker leak (Melissa/Tracker group)
• GVA1 status (???) -- Durga/John N. to look at effect on beamline -- Removed: (no longer) in geometry?
• CKOV monitoring : Should now be operational. Will be checked at expert run (actually, I think it was the KL monitor).
• HV channels (?) : Some HV channels are reporting as -1 in the 527. Check at expert run.
• MDR7: SY527 issues
• MDR7: TOF2: current trip in TOF2_103
• Get Helium flushing the tracker now (John)
  • Need a procedure for the MOM to check the Helium level for the tracker to know when to change the bottle (John)
• Monitoring GUI for the tracker volumes needs to be added in the AppLauncher
• issue with the EMR DAQ (restarted the crate)
• CDB vs Spreadsheet (All) -- resolved to remove spreadsheet by end of user run
• Alarm Handlers and State Machines
  • Go over FC and DS alarm handler spreadsheets (Pierrick,Mark,John C)
• Expert panel readiness : Done
• Feedback from OnMon to RC to tell ALH that data has become corrupted. #1802 (Rhys, Pierrick)
• SSH lockout procedure documentation (Paolo) and details of implementation (Steve)
• Test of network reconfiguration (Pierrick, Paolo)
• Devising a way to record the absorber details in the CDB (Pierrick, Durga)
• Confirm ISIS PVs and applications OK in MLCR (Ops)
• Check that PRY movement monitor measurements are in the archiver (Pierrick) : Stands

Cooling Channel¶

• "Commissioning Manager" and planning of that process
• RA, Operating Instructions, etc.
  • Progress on SS procedure documentation
• Install FC monitoring in ISIS MCR (Pierrick,Paolo)
• Hall probe communication problems
  • John G to think about a shielding solution for the Hall probe connectors
• There needs to be someone on call for the SC magnets. It will be a combination between Josef/Mark

Hall and infrastructure¶

• Cooling water system
• PID alignment and survey
• PPS issues #1818 ?
• Fencing under South Mezzanine
• All valid PPS meshing and gates in place
• camhall03 inverted
• Neutron Monitor remote readout
• Shared services/environment (water, air, computing/networking, UPS ( #1213 ) in Alarm Handler and in Archiver (CAM)
• Water fail document
• Additional ambient air temperature sensors
• Post-MDR7: Aircon unit assignment incorrect in monitoring GUI

General Computing¶

• DATE/OnMon/OnRec error messages all need timestamps, to allow correlation* Data Quality Flags #1383 (All, Janusz, Pierrick)
• DATE crash on first run of the day : Done? (fix actually deployed?)
• Data Quality Flags #1383
• Reco: Data Quality flags #1585 (Rhys, Durga, Pierrick)
• onrec disk full - cron in place?
  • miceraid5 disk full : Paolo has updated Nagios so it will keep an eye on this. We are down to 250 GB so we should do a preemptive cleanup. Get in touch with Yordan on self-cleanup.
• CDB : do we have all necessary information in the CDB
  • Get Target Depth, Target Delay, Total Beam Loss to read out (currently 0 or blank) - not appropriate for CDB?
  • Solenoid currents
  • Solenoid polarity
  • Focus coil current
  • Focus coil polarity
  • Absorber material
  • Absorber pressure (if necessary)
  • Tracker gas and pressure
• MDR7: issue writing in the CDB the new tables
• CDB viewer needs to have the cooling channel magnets and absorber status. Action : Janusz
• Implement a search function for the CDB (Someone)
• File Compactor:
  • Alarm Handler #1236
  • Data Quality Flags #1383 (Henry, Pierrick)
  • Compactor file corruption : Looks like two processes were running simultaneously which shouldn't happen.
  • MDR7: Data mover issue after the re-installation of miceonrec01: SOLVED and TESTED
• Automated RECO mover:
  • Automated mover and concentrator running
  • RECO-robot on hardware token
• Formal freeze of online software : Monday 27th June
• Access to EPICS archiver over the net (Cobb)
• Nagios/Heart beat alarms : Done (who gets notified, and when?)
• Backups not working for a fortnight : Refer to Paolo to see if we need an alarm for backups.

Operations¶

• Address values of the alarm limits (Steve, Paul H)
• What are SS settings for reference runs (All) : Decide to continue use current conventional magnet reference settings regardless of the SS currents
• Run 7851 - what happened and has the mess been fixed?
• Padlock-able overnight keypress -- in progress
• Start working through Hall prep list
• SC Controlled Entry procedure (mezzanine access, ODH, et al.)
• Ensure up-to-date documentation and checklists in MLCR:
  • Experts to review documentation and training needs during MDR7
    • especially Control Room Operations which is incomplete and out of date
    • need Alarm Handler training (screenshots, etc.?) and File Compactor
  • Update operations checklist
  • All shifter documentation up-to-date (???) and in MLCR binders (Boyd)
    • Shift checklist update : suggestions
  • Emergency procedures and documentation for shifters and experts
  • Expert list
  • Procedure whilst control room is unmanned : CC kept energised with MCR oversight.
• Training plan : http://micewww.pp.rl.ac.uk/projects/operations/wiki/TrainingPlan201602
  • New elements
    • Whatever is needed for the SS and FC
    • Alarm Handler
  • PPS Door lock (bouncing) : train people not to bang doors.
• Shifter Roster
• Shifters skipping training
  • Same shifters due this cycle
• Sick shifters (?) : Don't come in if dying. Tell MOM if sick. remind third-parties
• Permissive list of MOM-authorisable actions -- in progress
• Tidy up MLCR

Readiness_To_Do_List

Issue	Responsible Party	System	Issue Description	Data Noted	Status	Date Completed
MDR1-DET-05	M.Uchida	Tracker	Sufficient spares for the tracker readout should be commissioned so that they are hot-swappable	2015-01-21	In progress
MDR1-DET-06	M.Uchida	Tracker	The tracker cryogenics temperature fine control should be repaired	2015-01-21
MDR1-DET-10	M.Uchida	Tracker	The tracker internal and external LED systems should be made operational	2015-01-21
MDR1-DAQ-02	Y.Karadzhov	DATE	A new unpacking should be released to online monitoring and online reconstruction developers	2015-01-21	2016: Hall probe readout
MDR1-DAQ-03	Y.Karadzhov	DATE	The crashes in the InfoBrowser should be understood and fixed. This is the principal tool for DAQ debugging and is needed for both shifters and experts	2015-01-21
MDR1-CAM-03	P.Hanlet	EMR	Remote operation of the EMR should be established	2015-01-21	In progress
MDR1-CAM-04	M.Uchida	Tracker	Remote operation of the tracker should be integrated with the rest of the controls software	2015-01-21	In progress
MDR1-CAM-05	P.Hanlet	Diffuser	The "diffuser set" indicator should be fixed in the diffuser controls GUI	2015-01-21	In progress
MDR1-CAM-11	P.Hanlet	RunControl	Add buttons to Arm/Start/Pause/Stop DATE	2015-01-21	Near completion; still needs lock files
MDR1-CAM-12	P.Hanlet	RunControl	Consider adding Online Monitoring and Online Reconstruction notifications #1802	2015-01-21	In progress
MDR1-CAM-13	P.Hanlet	RunControl	Confirm configuration write to CDB after first trigger, Begin/End run comments and tag run	2015-01-21	Requires beam
MDR1-CAM-14	P.Hanlet	Target	Add target optical readouts to Alarm Handler	2015-01-21	Awaiting Matt's fixing PV updating
MDR1-CMP-04	A.Dobbs	MAUS	A new version of MAUS should be installed (released?) when an updated DAQ unpacking becomes available	2015-01-21	2016: Hall probe readout
MDR1-CMP-05	J.Martyniak	CDB	The Configuration Database get_tagged_beamline read should be fixed	2015-01-21