Response Procedure in case of target failure¶

• Table of contents
• Response Procedure in case of target failure
  • Has the BPS been tripped?
  • RATS daemon crash
  • Laser Level Issue
  • USB link loss
  • Anything worse

Has the BPS been tripped?¶

• If BPS tripped then confirm that the frame is raising itself out as expected (stepper driver display alternating between 'J' and 'c')
• Once the frame is fully raised the target is always out of ISIS
• Confirm frame is fully raised and stationary (stepper driver display 'U')
• Remove and return MTENBL key to ISIS MCR only after target frame has raised completely
• ISIS can resume running immediately on receipt of key
  • Aim is to get this back upstairs within 10 minutes

The MICE shift crew should phone BLOC first, then notify ISIS MCR that expertise is en route.

RATS daemon crash¶

• The target controller will carry on dipping for a short time.
• It will then automatically stop actuation, raise out the target frame slowly, and park the target.
• Shifters should confirm that the frame has started raising, and contact the BLOC.
• If ISIS is off, just check that there isn't a BPS error.
• In any case, remove and return MTENBL key to ISIS MCR only after target frame has raised completely

BLOC should

• confirm frame is raised and MTENBL key back in ISIS
• restart daemon, or if necessary cycle controller
• If OK, run target with frame raised for a bit and then resume as normal
  • else call expert

Laser Level Issue¶

The position read-back of the target shuttle depends on sufficient light being returned to the sensors. This is monitored by the controller and will result in a "Quadrature Error" and BPS trip if it falls below threshold. In the case of such a failure shifters should follow the procedure above.

Standing procedures require the shifters to contact the BLOC should the readout fall below the warning level of 51 000. They should leave the target in hold until they get a response.

The target will operate safely as long as the readout is over 49 000. BLOC should

• Use the EPICS Archiver to check when and how the laser levels have dropped
• Contact the Target experts and agree whether to increase laser power locally or the expert will increase gain remotely
• (If needed raise frame and return MTENBL key to ISIS MCR)
• To increase laser power:
  • park Target
  • put in hold and check lasers, return to park and check lasers - to get an averaged value
  • turn laser power up one notch of control knob
  • check lasers as above
  • repeat until that channel is in line with the others

USB link loss¶

• The target controller will carry on dipping for a short time.
• It will then automatically stop actuation, raise out the target frame slowly, and park the target.
• Shifters should confirm that the frame has started raising, and contact the BLOC.
• If ISIS is off, just check that there isn't a BPS error.
• In any case, remove and return MTENBL key to ISIS MCR only after target frame has raised completely

BLOC should

• confirm frame is raised and MTENBL key back in ISIS
• on target1ctl as root: check if /var/log/messages is reporting USB errors similar to
  

  Sep 22 10:30:22 target1ctl kernel: usb 1-1: reset high speed USB device using ehci_hcd and address 6
  Sep 22 10:30:53 target1ctl last message repeated 6 times

• if so, disconnect target controller USB cable from server
  • reconnect cable via a powered USB hub (steal from MOM desk in R76)
  • see if messages cease (they appear about once a minute)
  • if yes, restart RATS daemon
    • run target with frame raised for a bit and then resume as normal
  • else
    • get rid of USB hub (it isn't helping) and re-connect controller
    • do a cold-reboot of the server:
      • shut down target1ctl, remove mains plug, wait 5 mins, switch back on
    • RATS daemon will start automatically
    • run target with frame raised for a bit and then resume as normal
• otherwise
  • call expert

Contact expert if unsure!

Anything worse¶

If need to swap control PC, there are notes on target control PC [[computing-software:Target_Setup|build]] and [[online:TGTCTRLMachine|setup]].