Trouble Shooting - FADs

Crate Reset

There exist a script which will do the Crate Reset automatically:

• dimctrl --quit --cmd ".js scripts/doCrateReset.js crate[n]=true"
• n = number of crate you want to reset (e.g. crate2=true)

Crate Reset manually:

• disable all FTUs (in the FTU tab of your screen session) ftmctrl/ENABLE_FTU -1 0
• disconnect the 10 FADs in the crate, by clicking the LEDs
• ftmctrl/RESET_CRATE x (x corresponding crate)
• enable all FTUs (in the FTU tab of your screen session) ftmctrl/ENABLE_FTU -1 1
• reconnect the 10 FADs, one by one with 3 seconds waiting in between (the FAD needs this time to boot)

start up - connection problem

Symptom

• problem occurs usually during start up
• after FAD_CONTROL/START or pushing FAD -> START button in the GUI, not all 40 FAD LEDs are green
• fadctrl is in state Connecting (instead of Connected)

Solution

• stop dimscripts
  • dimctrl --stop from a bash
• do a crate reset:

Not Helping

• disconnect / reconnect to the FAD.
• waiting
• reset other / all crates (might just create another of these bugs in another FAD)
• stopping or killing fadctrl
• power cycling the camera (might just create another of these bugs in another FAD)

In-run-FAD-loss

Symptom

• during a run:
  • 4 adjacent *strange* patches in the events tab of the GUI
  • orange warnings in fadctrl-console (eventbuilder realises, that one FAD stopped sending data)
  • trigger rate drop to 0
• when the run ended and a new one is started
  • MCP hang in state Configuring3
  • fadctrl hang in state Configuring2

Solution

• find out which FAD board is disconnected
  • the one board in the FAD tab which has the strange behaviour
  • the one without a thick on its LED in the GUI (is only possible to see, when a new run is started and the system hangs in ConfiguringN)
• stop dimscript (by dimctrl --stop from a bash)
  • only possible when dimctrl was started as a server instance (dimctrl --server)
  • otherwise you have to kill dimctrl via ctrl + c
• reset MCP (MCP/RESET)
• Disconnect the problematic FAD (clicking on the corresponding LED in the GUI
• wait 3 to 5 sec
• Reconnect the problematic FAD (again clicking)

Not Helping

• waiting
• reset any crates (might just create another start up connection problem)
• stopping or killing fadctrl
• power cycling the camera (might just create another start up connection problem)

DRS underflow problem

Symptom

• happens usually during start up
• during first data taking one single patch in the events tab appears different (e.g. all dark blue)
• after the 3 calibration runs are taken by the system, the calibration constants behave different:
  • the magenta line (normally around +1000mV) has pretty long error bars and is curved over the whole canvas

Solution

• stop dimscript
  • dimctrl --stop from a bash
• Reset crate (see <A HREF="#tscratereset">above</A>)
  • this doesn't helped often, but it doesn't need so much time as the power cycle
• if the problem appears again: power cycle the camera (see <A HREF="#tshardware">below</A>)

Not Helping

• all the rest

startup - no proper connection problem

Symptom

• happens usually during start up
• several FADs are in state "Waiting" (orange in the GUI)

Solution

• disenable the corresponding FADs by clicking in the GUI on them
• wait a short time
• enable the corresponding FADs by clicking again on them (wait 3 seconds between two clicks)

Not Helping

• killing / quiting of fadctrl

fadctrl hangs in state configuring by taking an external lp run

Symptom

• fadctrl hangs in state configuring when it starts the external lp run during datataking (point 17. under datatakingdetails#normaldata normal datataking procedure the 9. run)
• all FADs are connected (if not all FADs are connected it is a FAD connection loss (see above)

Solution

• normally the shifter just forgot to open the lid, so
  • stop script
  • open the lid (see preparation#startup Start Up procedure , point 9)

Not Helping

• all the rest