Actions
Step 4 Readiness (summer 2016) after MDR7¶
Summary of the actions (Steve/MOM 11/07/2016)
Raw list includes some things marked as already done as they need re-doing (e.g. new unpacking including Hall probes) and some items may even be done already.
In this version I've marked the ones I know are done - where "done" means the fix has been identified, tested and deployed.
List of issues at summer 2016, culled from
Step 4 Requirements
24 June Readiness Meeting
28 June Operations Meeting and issues from MDR7
21 June Operations Meeting
14 June Operations Meeting
7 June Operations Meeting
February 2016 List
Henry has NOT gone back through the eLog.
Beamline (c.f. #1854 )¶
Target- Target Alarm handler #1613 update 22 (CAM)
ISIS data items #1669 (CAM) - ISIS-to-EPICS gateway testedActivation Study (activation seen: limit of 2 V at MS/64 stands)Beambump tune- Updated target manual (EO/JL)
Updated target checklists (Henry)
review existing water flows and temperature range (MBC)- need limit in control software? (CAM)
- Calibrate D1 & D2 current reading and update IOC software (DL)
Q1 MPS start-up recalcitrance (DL)Q1 MPS replacedReplaced the PS. Does replacement have correct polarity? Reference run results? The TOFs calibration is fineSoak test / smell testSmells OK; soaked Q2 supply OK :(
- Diagnose failed Danfysik 9000 supply from Q2 (ISIS)
- Q1 alarm on voltage at zero amps (CAM) - gone since PSU changes?
- Q4 oscillation below 10A and anomalous ramp rate (DL)
Trench water leak / Q5 water leak : Q5 has been fixed.- Q6 leak : too fragile, wouldn't touch it - shifters to watch
- Leak sensors and drip trays
- Water, temperatures and MPS water flows in ALH (CAM)
Q6 temperature alarm limits missing!- Magnets trips should be reported in the ALH (Pierrick)
Overheating investigation (DK/DL)- Calibrate readback current reading and update IOC software (DL)
120A trip investigation (DK/DL)- Replace final battery cell (DL - confirmed still outstanding )
- Diodes in BBU circuit (DL)
- Boosted air (MBC)
- Trench sensor (awaiting read-out)
- Compressor sensor
- Backup compressor for DS
- Linde PC install: awaiting validation and IOC (DK) - licence issues
DS ALH/archiver corrections #1710 (CAM) - covered by DS spreadsheet action below?DS Fridge instability : Has been investigated by Mark. Monitor.
Proton absorber sticking : Hasn't been reproduced lately - shifters will need to monitor it.Diffuser: see MDR1-CAM-05 below
- BLOC training and experience (Henry)
- BLOC roster
Put BLOCs in CHEESE (Steve) : Done
Detectors, Monitoring and Control¶
- GVA1 status
Durga/John N. to look at effect on beamline Removed:in geometry? CKOV monitoring : Should now be operational. Will be checked at expert run (actually, I think it was the KL monitor).HV channels (?) : Some HV channels are reporting as -1 in the CAEN 527. Check at expert run.MDR7: SY527 issues: card required a reboot- MDR7: TOF2: current trip in TOF2_103
Tracker leak (Melissa/Tracker group) as good as it will get?Get Helium flushing the tracker now (John)Need a procedure for the MOM to check the Helium level for the tracker to know when to change the bottle (John)Documented for succeeding MOMs? Draft: https://micewww.pp.rl.ac.uk/projects/operations/wiki/Helium, in the MOM handover page and in the MOM Standing Duties List.
- Monitoring GUI for the tracker volumes needs to be added in the AppLauncher (Pierrick)
- Issue with the EMR DAQ (restarted the crate) - future?: is there an usable spare? (Francois)
- CDB vs Spreadsheet (All) -- resolved to remove spreadsheet by end of user run
- Alarm Handlers and State Machines
Go over FC and DS alarm handler spreadsheets (Pierrick,Mark,John C)Other systems
Expert panel readiness : Done? Cooling channel?Feedback from OnMon to RC to tell ALH that data has become corrupted. #1802 (Pierrick)SSH lockout procedure documentation (Paolo) https://micewww.pp.rl.ac.uk/projects/cm/wiki/Iocs#MICENet-lockout - and -details of implementation (Steve). Mimic for the shifters (Pierrick)Test of network reconfiguration (Pierrick, Paolo)Devising a way to record the absorber details in the CDB (Pierrick, Durga)- Confirm ISIS PVs and applications OK in MLCR (Ops)
Check that PRY movement monitor measurements are in the archiver (Pierrick)
Cooling Channel¶
- "Commissioning Manager" and planning of that process
- RA, Operating Instructions, etc.
- Progress on SS procedure documentation
- Install FC monitoring in ISIS MCR (Pierrick,Paolo)
Hall probe communication problemsJohn G to think about a shielding solution for the Hall probe connectors
- There needs to be someone on call for the SC magnets. It will be a combination between Josef/Mark
Hall and infrastructure¶
Cooling water systemno more changes till AugustPID alignment and survey - Survey done. Alignment runs will be taken during 2016/02 cycle- PPS issues #1818 ?
Fencing under South Mezzanine- All valid PPS meshing and gates in place
- camhall03 inverted
Neutron Monitor remote readoutAdditional ambient air temperature sensors- Shared services/environment (water, air, computing/networking, UPS ( #1213 ) in Alarm Handler and in Archiver (CAM)
Water fail document- Post-MDR7: Aircon unit assignment incorrect in monitoring GUI: 2 and 3 are swapped
- Aircon unit 4 has some mechanical issue - occasional squealing from fan belt (John G.)
General Computing¶
DATE/OnMon/OnRec error messages all need timestamps, to allow correlation (Durga)DATE crash on first run of the day : Done? (fix actually deployed?)- Data Quality Flags #1383 (All, Janusz, Pierrick)
- Reco: Data Quality flags #1585 (Durga, Pierrick)
onrec disk full - cron in place?- miceraid5 disk full : Paolo has updated Nagios so it will keep an eye on this. We are down to 250 GB so we should do a preemptive cleanup. Get in touch with Yordan on self-cleanup (clean with cron/tmpwatch everything untouched for over a month). (Yordan)
CDB : do we have all necessary information in the CDB- Get Target Depth, Target Delay, Total Beam Loss to read out (currently 0 or blank) - not appropriate for CDB? (Durga/Pierrick)
Solenoid currentsSolenoid polarityFocus coil currentFocus coil polarityAbsorber materialAbsorber pressure (if necessary)Tracker gas and pressure: is in the check list
MDR7: issue writing in the CDB the new tables (prod DB?)- CDB viewer needs to have the cooling channel magnets and absorber status. Action : (Janusz)
- Implement a search function for the CDB (Someone)
- CDB corrections backlog from previous user runs
- File Compactor:
- Alarm Handler #1236
- Data Quality Flags #1383 (Henry, Pierrick)
Compactor file corruption : Looks like two processes were running simultaneously which shouldn't happen.MDR7: Data mover issue after the re-installation of miceonrec01: SOLVED and TESTED
- Automated RECO mover:
- Automated mover and concentrator running
RECO-robot on hardware token
Formal freeze of online software : Monday 27th JuneAccess to EPICS archiver over the net (Cobb)It works, but sometimes is very slow.Nagios/Heart beat alarms : Done(who gets notified, and when?)Backups not working for a fortnight : Refer to Paolo to see if we need an alarm for backups.
Operations¶
Address values of the alarm limits (Steve, Paul H)spreadsheetsWhat are SS settings for reference runs (All): Decide to continue use current conventional magnet reference settings regardless of the SS currentsRun 7851 - what happened and has the mess been fixed?Padlock-able overnight keypress -in progress- broken?Start working through Hall prep listSC Controlled Entry procedure (mezzanine access, ODH, et al.)Ensure up-to-date documentation and checklists in MLCR:Experts to review documentation and training needs during MDR7- especially Control Room Operations which is incomplete and out of date
- need Alarm Handler training (screenshots, etc.?) and File Compactor
Update operations checklistAll shifter documentation up-to-date (???)and in MLCR binders (Boyd)Shift checklist update : suggestions
Emergency procedures and documentation for shifters and experts- Expert list - MLCR copy missing Target expert, at least
Procedure whilst control room is unmanned: CC kept energised with MCR oversight.
Training plan : http://micewww.pp.rl.ac.uk/projects/operations/wiki/TrainingPlan201602- New elements
- Whatever is needed for the SS and FC
- Alarm Handler
Level of PPS training neededPPS Door lock (bouncing) : train people not to bang doors.
- New elements
Shifter RosterShifters skipping trainingSame shifters due this cycle
Sick shifters (?) : Don't come in if dying. Tell MOM if sick. remind third-parties- Permissive list of MOM-authorisable actions -- in progress
- Web Whiteboard
- Settings and Run Plans defined
Tidy up MLCR- Is on MOM
Issue | Responsible Party | System | Issue Description | Data Noted | Status | Date Completed |
MDR1-DET-05 | M.Uchida | Tracker | Sufficient spares for the tracker readout should be commissioned so that they are hot-swappable | 2015-01-21 | In progress | |
M.Uchida | Tracker | The tracker cryogenics temperature fine control should be repaired | 2015-01-21 | |||
M.Uchida | Tracker | The tracker internal and external LED systems should be made operational | 2015-01-21 | |||
|
Y.Karadzhov | DATE | A new unpacking should be released to online monitoring and online reconstruction developers | 2015-01-21 | 2016: Hall probe readout | |
Y.Karadzhov | DATE | The crashes in the InfoBrowser should be understood and fixed. This is the principal tool for DAQ debugging and is needed for both shifters and experts | 2015-01-21 | |||
MDR1-CAM-03 | P.Hanlet | EMR | Remote operation of the EMR should be established | 2015-01-21 | In progress | |
M.Uchida | Tracker | Remote operation of the tracker should be integrated with the rest of the controls software | 2015-01-21 | In progress | ||
P.Hanlet | Diffuser | The "diffuser set" indicator should be fixed in the diffuser controls GUI | 2015-01-21 | In progress | ||
P.Hanlet | RunControl | Add buttons to Arm/Start/Pause/Stop DATE | 2015-01-21 | Near completion; still needs lock files | ||
MDR1-CAM-12 | P.Hanlet | RunControl | Consider adding |
2015-01-21 | In progress | |
P.Hanlet | RunControl | Confirm configuration write to CDB after first trigger, Begin/End run comments and tag run | 2015-01-21 | Requires beam | ||
MDR1-CAM-14 | P.Hanlet | Target | Add target optical readouts to Alarm Handler | 2015-01-21 | Awaiting Matt's fixing PV updating | |
|
A.Dobbs | MAUS | A new version of MAUS should be installed (released?) when an updated DAQ unpacking becomes available | 2015-01-21 | 2016: Hall probe readout | |
J.Martyniak | CDB | The Configuration Database get_tagged_beamline read should be fixed | 2015-01-21 |
Updated by Franchini, Paolo about 7 years ago ยท 71 revisions