Project

General

Profile

Actions
History

Step 4 Readiness (summer 2016) after MDR7

Summary of the actions (Steve/MOM 11/07/2016)

Raw list includes some things marked as already done as they need re-doing (e.g. new unpacking including Hall probes) and some items may even be done already.

In this version I've marked the ones I know are done - where "done" means the fix has been identified, tested and deployed.

List of issues at summer 2016, culled from

Step 4 Requirements
24 June Readiness Meeting
28 June Operations Meeting and issues from MDR7
21 June Operations Meeting
14 June Operations Meeting
7 June Operations Meeting
February 2016 List

Henry has NOT gone back through the eLog.

Beamline (c.f. #1854 )

Target
  • Target Alarm handler #1613 update 22 (CAM)
  • ISIS data items #1669 (CAM) - ISIS-to-EPICS gateway tested
  • Activation Study (activation seen: limit of 2 V at MS/64 stands)
  • Beambump tune
  • Updated target manual (EO/JL)
  • Updated target checklists (Henry)
Conventional Magnets
  • review existing water flows and temperature range (MBC)
    • need limit in control software? (CAM)
  • Calibrate D1 & D2 current reading and update IOC software (DL)
  • Q1 MPS start-up recalcitrance (DL) Q1 MPS replaced
    • Replaced the PS. Does replacement have correct polarity? Reference run results? The TOFs calibration is fine
    • Soak test / smell test Smells OK; soaked Q2 supply OK :(
  • Diagnose failed Danfysik 9000 supply from Q2 (ISIS)
  • Q1 alarm on voltage at zero amps (CAM) - gone since PSU changes?
  • Q4 oscillation below 10A and anomalous ramp rate (DL)
  • Trench water leak / Q5 water leak : Q5 has been fixed.
  • Q6 leak : too fragile, wouldn't touch it - shifters to watch
  • Leak sensors and drip trays
  • Water, temperatures and MPS water flows in ALH (CAM)
    • Q6 temperature alarm limits missing!
    • Magnets trips should be reported in the ALH (Pierrick)
Decay Solenoid
  • Overheating investigation (DK/DL)
  • Calibrate readback current reading and update IOC software (DL)
  • 120A trip investigation (DK/DL)
  • Replace final battery cell (DL - confirmed still outstanding )
  • Diodes in BBU circuit (DL)
  • Boosted air (MBC)
    • Trench sensor (awaiting read-out)
    • Compressor sensor
    • Backup compressor for DS
  • Linde PC install: awaiting validation and IOC (DK) - licence issues
  • DS ALH/archiver corrections #1710 (CAM) - covered by DS spreadsheet action below?
    • DS Fridge instability : Has been investigated by Mark. Monitor.
Proton Absorber / Beamstop / Diffuser
  • Proton absorber sticking : Hasn't been reproduced lately - shifters will need to monitor it.
  • Diffuser: see MDR1-CAM-05 below
Other
  • BLOC training and experience (Henry)
  • BLOC roster
    • Put BLOCs in CHEESE (Steve) : Done

Detectors, Monitoring and Control

  • GVA1 status Durga/John N. to look at effect on beamline Removed: in geometry?
  • CKOV monitoring : Should now be operational. Will be checked at expert run (actually, I think it was the KL monitor).
  • HV channels (?) : Some HV channels are reporting as -1 in the CAEN 527. Check at expert run.
  • MDR7: SY527 issues: card required a reboot
  • MDR7: TOF2: current trip in TOF2_103
  • Tracker leak (Melissa/Tracker group) as good as it will get?
  • Get Helium flushing the tracker now (John)
  • Monitoring GUI for the tracker volumes needs to be added in the AppLauncher (Pierrick)
  • Issue with the EMR DAQ (restarted the crate) - future?: is there an usable spare? (Francois)
  • CDB vs Spreadsheet (All) -- resolved to remove spreadsheet by end of user run
  • Alarm Handlers and State Machines
    • Go over FC and DS alarm handler spreadsheets (Pierrick,Mark,John C)
    • Other systems
  • Expert panel readiness : Done? Cooling channel?
  • Feedback from OnMon to RC to tell ALH that data has become corrupted. #1802 (Pierrick)
  • SSH lockout procedure documentation (Paolo) https://micewww.pp.rl.ac.uk/projects/cm/wiki/Iocs#MICENet-lockout - and -details of implementation (Steve). Mimic for the shifters (Pierrick)
  • Test of network reconfiguration (Pierrick, Paolo)
  • Devising a way to record the absorber details in the CDB (Pierrick, Durga)
  • Confirm ISIS PVs and applications OK in MLCR (Ops)
  • Check that PRY movement monitor measurements are in the archiver (Pierrick)

Cooling Channel

  • "Commissioning Manager" and planning of that process
  • RA, Operating Instructions, etc.
    • Progress on SS procedure documentation
  • Install FC monitoring in ISIS MCR (Pierrick,Paolo)
  • Hall probe communication problems
    • John G to think about a shielding solution for the Hall probe connectors
  • There needs to be someone on call for the SC magnets. It will be a combination between Josef/Mark

Hall and infrastructure

  • Cooling water system no more changes till August
  • PID alignment and survey - Survey done. Alignment runs will be taken during 2016/02 cycle
  • PPS issues #1818 ?
  • Fencing under South Mezzanine
  • All valid PPS meshing and gates in place
  • camhall03 inverted
  • Neutron Monitor remote readout
  • Additional ambient air temperature sensors
  • Shared services/environment (water, air, computing/networking, UPS ( #1213 ) in Alarm Handler and in Archiver (CAM)
  • Water fail document
  • Post-MDR7: Aircon unit assignment incorrect in monitoring GUI: 2 and 3 are swapped
  • Aircon unit 4 has some mechanical issue - occasional squealing from fan belt (John G.)

General Computing

  • DATE/OnMon/OnRec error messages all need timestamps, to allow correlation (Durga)
  • DATE crash on first run of the day : Done? (fix actually deployed?)
  • Data Quality Flags #1383 (All, Janusz, Pierrick)
  • Reco: Data Quality flags #1585 (Durga, Pierrick)
  • onrec disk full - cron in place?
  • miceraid5 disk full : Paolo has updated Nagios so it will keep an eye on this. We are down to 250 GB so we should do a preemptive cleanup. Get in touch with Yordan on self-cleanup (clean with cron/tmpwatch everything untouched for over a month). (Yordan)
  • CDB : do we have all necessary information in the CDB
    • Get Target Depth, Target Delay, Total Beam Loss to read out (currently 0 or blank) - not appropriate for CDB? (Durga/Pierrick)
    • Solenoid currents
    • Solenoid polarity
    • Focus coil current
    • Focus coil polarity
    • Absorber material
    • Absorber pressure (if necessary)
    • Tracker gas and pressure: is in the check list
  • MDR7: issue writing in the CDB the new tables (prod DB?)
  • CDB viewer needs to have the cooling channel magnets and absorber status. Action : (Janusz)
  • Implement a search function for the CDB (Someone)
  • CDB corrections backlog from previous user runs
  • File Compactor:
    • Alarm Handler #1236
    • Data Quality Flags #1383 (Henry, Pierrick)
    • Compactor file corruption : Looks like two processes were running simultaneously which shouldn't happen.
    • MDR7: Data mover issue after the re-installation of miceonrec01: SOLVED and TESTED
  • Automated RECO mover:
    • Automated mover and concentrator running
    • RECO-robot on hardware token
  • Formal freeze of online software : Monday 27th June
  • Access to EPICS archiver over the net (Cobb) It works, but sometimes is very slow.
  • Nagios/Heart beat alarms : Done (who gets notified, and when?)
  • Backups not working for a fortnight : Refer to Paolo to see if we need an alarm for backups.

Operations

  • Address values of the alarm limits (Steve, Paul H) spreadsheets
  • What are SS settings for reference runs (All) : Decide to continue use current conventional magnet reference settings regardless of the SS currents
  • Run 7851 - what happened and has the mess been fixed?
  • Padlock-able overnight keypress - in progress- broken?
  • Start working through Hall prep list
  • SC Controlled Entry procedure (mezzanine access, ODH, et al.)
  • Ensure up-to-date documentation and checklists in MLCR:
    • Experts to review documentation and training needs during MDR7
      • especially Control Room Operations which is incomplete and out of date
      • need Alarm Handler training (screenshots, etc.?) and File Compactor
    • Update operations checklist
    • All shifter documentation up-to-date (???) and in MLCR binders (Boyd)
      • Shift checklist update : suggestions
    • Emergency procedures and documentation for shifters and experts
    • Expert list - MLCR copy missing Target expert, at least
    • Procedure whilst control room is unmanned : CC kept energised with MCR oversight.
  • Training plan : http://micewww.pp.rl.ac.uk/projects/operations/wiki/TrainingPlan201602
    • New elements
      • Whatever is needed for the SS and FC
      • Alarm Handler
    • Level of PPS training needed
    • PPS Door lock (bouncing) : train people not to bang doors.
  • Shifter Roster
  • Shifters skipping training
    • Same shifters due this cycle
  • Sick shifters (?) : Don't come in if dying. Tell MOM if sick. remind third-parties
  • Permissive list of MOM-authorisable actions -- in progress
  • Web Whiteboard
  • Settings and Run Plans defined
  • Tidy up MLCR - Is on MOM

Readiness_To_Do_List

Issue Responsible Party System Issue Description Data Noted Status Date Completed
MDR1-DET-05 M.Uchida Tracker Sufficient spares for the tracker readout should be commissioned so that they are hot-swappable 2015-01-21 In progress
MDR1-DET-06 M.Uchida Tracker The tracker cryogenics temperature fine control should be repaired 2015-01-21
MDR1-DET-10 M.Uchida Tracker The tracker internal and external LED systems should be made operational 2015-01-21
MDR1-DAQ-02 Y.Karadzhov DATE A new unpacking should be released to online monitoring and online reconstruction developers 2015-01-21 2016: Hall probe readout
MDR1-DAQ-03 Y.Karadzhov DATE The crashes in the InfoBrowser should be understood and fixed. This is the principal tool for DAQ debugging and is needed for both shifters and experts 2015-01-21
MDR1-CAM-03 P.Hanlet EMR Remote operation of the EMR should be established 2015-01-21 In progress
MDR1-CAM-04 M.Uchida Tracker Remote operation of the tracker should be integrated with the rest of the controls software 2015-01-21 In progress
MDR1-CAM-05 P.Hanlet Diffuser The "diffuser set" indicator should be fixed in the diffuser controls GUI 2015-01-21 In progress
MDR1-CAM-11 P.Hanlet RunControl Add buttons to Arm/Start/Pause/Stop DATE 2015-01-21 Near completion; still needs lock files
MDR1-CAM-12 P.Hanlet RunControl Consider adding Online Monitoring and Online Reconstruction notifications #1802 2015-01-21 In progress
MDR1-CAM-13 P.Hanlet RunControl Confirm configuration write to CDB after first trigger, Begin/End run comments and tag run 2015-01-21 Requires beam
MDR1-CAM-14 P.Hanlet Target Add target optical readouts to Alarm Handler 2015-01-21 Awaiting Matt's fixing PV updating
MDR1-CMP-04 A.Dobbs MAUS A new version of MAUS should be installed (released?) when an updated DAQ unpacking becomes available 2015-01-21 2016: Hall probe readout
MDR1-CMP-05 J.Martyniak CDB The Configuration Database get_tagged_beamline read should be fixed 2015-01-21

Updated by Franchini, Paolo over 6 years ago ยท 71 revisions