Step4Readiness » History » Revision 31

« Previous | Revision 31/71 (diff) | Next »
Nebrensky, Henry, 01 July 2016 11:39
Complete list of issues - a few await marking as done

Step 4 Readiness (summer 2016) for MDR7

Raw list includes some things marked as already done as they need re-doing (e.g. new unpacking including Hall probes) and a lot of repetition (and some items may even be done already).

List of issues at summer 2016, culled from

Step 4 Requirements
24 June Readiness Meeting
28 June Operations Meeting and issues from MDR7
21 June Operations Meeting
14 June Operations Meeting
7 June Operations Meeting
February 2016 List

Beamline (c.f. #1854 )

  • Target Alarm handler #1613 update 22 (CAM)
  • ISIS data items #1669 (CAM)
  • Activation Study (activation seen: limit of 2 V at MS/64 stands)
  • Beambump tune
  • Updated target manual (EO/JL)
  • Updated target checklists (Henry)
Conventional Magnets
  • review existing water flows and temperature range (MBC)
    • need limit in control software? (CAM)
  • Calibrate D1 & D2 current reading and update IOC software (DL)
  • Q1 MPS start-up recalcitrance (DL) Q1 MPS replaced
    • Replaced the PS. Does it have correct polarity?
  • Q1 alarm on voltage at zero amps (CAM)
  • Q4 oscillation below 10A and anomalous ramp rate (DL)
  • Trench water leak / Q5 water leak : Q5 has been fixed.
  • Q6 leak : too fragile, wouldn't touch it - shifters to watch
  • Leak sensors and drip trays
  • Water, temperatures and MPS water flows in ALH (CAM)
    • *Q6 temperature alarm limits missing!*
Decay Solenoid
  • DS readback current requires calibration (DL)
  • Overheating investigation (DK/DL)
  • Calibrate current reading and update IOC software (DL)
  • 120A trip investigation (DK/DL)
  • Replace final battery cell (DL - confirmed still outstanding )
  • Diodes in BBU circuit (DL)
  • Boosted air (MBC)
    • Trench sensor (awaiting read-out)
    • Compressor sensor
    • Backup compressor for DS
  • Linde PC install: awaiting validation and IOC (DK) - licence issues
  • DS ALH/archiver corrections #1710 (CAM)
    • DS Fridge instability : Has been investigated by Mark. Monitor.
+ Proton Absorber / Beamstop / Diffuser
  • Proton absorber sticking : Hasn't been reproduced lately - shifters will need to monitor it.
  • BLOC training and experience (MBC)
  • BLOC roster
    • Put BLOCs in CHEESE (Steve) : Done

Detectors, Monitoring and Control

  • Tracker leak (Melissa/Tracker group)
  • GVA1 status (???) -- Durga/John N. to look at effect on beamline -- Removed: (no longer) in geometry?
  • CKOV monitoring : Should now be operational. Will be checked at expert run (actually, I think it was the KL monitor).
  • HV channels (?) : Some HV channels are reporting as -1 in the 527. Check at expert run.
  • MDR7: SY527 issues
  • MDR7: TOF2: current trip in TOF2_103
  • Get Helium flushing the tracker now (John)
    • Need a procedure for the MOM to check the Helium level for the tracker to know when to change the bottle (John)
  • Monitoring GUI for the tracker volumes needs to be added in the AppLauncher
  • issue with the EMR DAQ (restarted the crate)
  • CDB vs Spreadsheet (All) -- resolved to remove spreadsheet by end of user run
  • Alarm Handlers and State Machines
    • Go over FC and DS alarm handler spreadsheets (Pierrick,Mark,John C)
  • Expert panel readiness : Done
  • Feedback from OnMon to RC to tell ALH that data has become corrupted. #1802 (Rhys, Pierrick)
  • SSH lockout procedure documentation (Paolo) and details of implementation (Steve)
  • Test of network reconfiguration (Pierrick, Paolo)
  • Devising a way to record the absorber details in the CDB (Pierrick, Durga)
  • Confirm ISIS PVs and applications OK in MLCR (Ops)
  • Check that PRY movement monitor measurements are in the archiver (Pierrick) : Stands

Cooling Channel

  • "Commissioning Manager" and planning of that process
  • RA, Operating Instructions, etc.
    • Progress on SS procedure documentation
  • Install FC monitoring in ISIS MCR (Pierrick,Paolo)
  • Hall probe communication problems
    • John G to think about a shielding solution for the Hall probe connectors
  • There needs to be someone on call for the SC magnets. It will be a combination between Josef/Mark

Hall and infrastructure

  • Cooling water system
  • PID alignment and survey
  • PPS issues #1818 ?
  • Fencing under South Mezzanine
  • All valid PPS meshing and gates in place
  • camhall03 inverted
  • Neutron Monitor remote readout
  • Shared services/environment (water, air, computing/networking, UPS ( #1213 ) in Alarm Handler and in Archiver (CAM)
  • Water fail document
  • Additional ambient air temperature sensors
  • Post-MDR7: Aircon unit assignment incorrect in monitoring GUI

General Computing

  • DATE/OnMon/OnRec error messages all need timestamps, to allow correlation* Data Quality Flags #1383 (All, Janusz, Pierrick)
  • DATE crash on first run of the day : Done? (fix actually deployed?)
  • Data Quality Flags #1383
  • Reco: Data Quality flags #1585 (Rhys, Durga, Pierrick)
  • onrec disk full - cron in place?
    • miceraid5 disk full : Paolo has updated Nagios so it will keep an eye on this. We are down to 250 GB so we should do a preemptive cleanup. Get in touch with Yordan on self-cleanup.
  • CDB : do we have all necessary information in the CDB
    • Get Target Depth, Target Delay, Total Beam Loss to read out (currently 0 or blank) - not appropriate for CDB?
    • Solenoid currents
    • Solenoid polarity
    • Focus coil current
    • Focus coil polarity
    • Absorber material
    • Absorber pressure (if necessary)
    • Tracker gas and pressure
  • MDR7: issue writing in the CDB the new tables
  • CDB viewer needs to have the cooling channel magnets and absorber status. Action : Janusz
  • Implement a search function for the CDB (Someone)
  • File Compactor:
    • Alarm Handler #1236
    • Data Quality Flags #1383 (Henry, Pierrick)
    • Compactor file corruption : Looks like two processes were running simultaneously which shouldn't happen.
    • MDR7: Data mover issue after the re-installation of miceonrec01: SOLVED and TESTED
  • Automated RECO mover:
    • Automated mover and concentrator running
    • RECO-robot on hardware token
  • Formal freeze of online software : Monday 27th June
  • Access to EPICS archiver over the net (Cobb)
  • Nagios/Heart beat alarms : Done (who gets notified, and when?)
  • Backups not working for a fortnight : Refer to Paolo to see if we need an alarm for backups.


  • Address values of the alarm limits (Steve, Paul H)
  • What are SS settings for reference runs (All) : Decide to continue use current conventional magnet reference settings regardless of the SS currents
  • Run 7851 - what happened and has the mess been fixed?
  • Padlock-able overnight keypress -- in progress
  • Start working through Hall prep list
  • SC Controlled Entry procedure (mezzanine access, ODH, et al.)
  • Ensure up-to-date documentation and checklists in MLCR:
    • Experts to review documentation and training needs during MDR7
      • especially Control Room Operations which is incomplete and out of date
      • need Alarm Handler training (screenshots, etc.?) and File Compactor
    • Update operations checklist
    • All shifter documentation up-to-date (???) and in MLCR binders (Boyd)
      • Shift checklist update : suggestions
    • Emergency procedures and documentation for shifters and experts
    • Expert list
    • Procedure whilst control room is unmanned : CC kept energised with MCR oversight.
  • Training plan :
    • New elements
      • Whatever is needed for the SS and FC
      • Alarm Handler
    • PPS Door lock (bouncing) : train people not to bang doors.
  • Shifter Roster
  • Shifters skipping training
    • Same shifters due this cycle
  • Sick shifters (?) : Don't come in if dying. Tell MOM if sick. remind third-parties
  • Permissive list of MOM-authorisable actions -- in progress
  • Tidy up MLCR


Issue Responsible Party System Issue Description Data Noted Status Date Completed
MDR1-DET-05 M.Uchida Tracker Sufficient spares for the tracker readout should be commissioned so that they are hot-swappable 2015-01-21 In progress
MDR1-DET-06 M.Uchida Tracker The tracker cryogenics temperature fine control should be repaired 2015-01-21
MDR1-DET-10 M.Uchida Tracker The tracker internal and external LED systems should be made operational 2015-01-21
MDR1-DAQ-02 Y.Karadzhov DATE A new unpacking should be released to online monitoring and online reconstruction developers 2015-01-21 2016: Hall probe readout
MDR1-DAQ-03 Y.Karadzhov DATE The crashes in the InfoBrowser should be understood and fixed. This is the principal tool for DAQ debugging and is needed for both shifters and experts 2015-01-21
MDR1-CAM-03 P.Hanlet EMR Remote operation of the EMR should be established 2015-01-21 In progress
MDR1-CAM-04 M.Uchida Tracker Remote operation of the tracker should be integrated with the rest of the controls software 2015-01-21 In progress
MDR1-CAM-05 P.Hanlet Diffuser The "diffuser set" indicator should be fixed in the diffuser controls GUI 2015-01-21 In progress
MDR1-CAM-11 P.Hanlet RunControl Add buttons to Arm/Start/Pause/Stop DATE 2015-01-21 Near completion; still needs lock files
MDR1-CAM-12 P.Hanlet RunControl Consider adding Online Monitoring and Online Reconstruction notifications #1802 2015-01-21 In progress
MDR1-CAM-13 P.Hanlet RunControl Confirm configuration write to CDB after first trigger, Begin/End run comments and tag run 2015-01-21 Requires beam
MDR1-CAM-14 P.Hanlet Target Add target optical readouts to Alarm Handler 2015-01-21 Awaiting Matt's fixing PV updating
MDR1-CMP-04 A.Dobbs MAUS A new version of MAUS should be installed (released?) when an updated DAQ unpacking becomes available 2015-01-21 2016: Hall probe readout
MDR1-CMP-05 J.Martyniak CDB The Configuration Database get_tagged_beamline read should be fixed 2015-01-21

Updated by Nebrensky, Henry almost 8 years ago · 31 revisions