Trouble Shooting - FADs

Crate Reset

Please use Smartfact or the script for a Camera Reset (former: "Crate" Reset). The following step-by-step instructions is mainly meant for experts during debugging.

There exists a script which will do the Crate Reset automatically:

• dimctrl --quit --cmd ".js scripts/doCrateReset.js crate[n]=true"
• n = number of crate you want to reset (e.g. crate2=true)

Note that the current setup does not allow anymore to reset individual crates. So whatever option you give, the whole camera will be reset. This previous instructions explain how to start the script directly in a local instance of dimctrl. This is not recommended. The recommended way is to use dimctrl to start the script in the dimserver. If you decide to start the script locally, make sure that the script that you start is the right one (i.e. in operator/scripts on newdaq/newdata).

Crate Reset manually:

• disable all FTUs (in the FTU tab of your screen session) ftmctrl/ENABLE_FTU -1 0
• disconnect all FADs, either by FAD_CONTROL/STOP or if that fails by disconnecting them one-by-one
• FPGAFTM_CONTROL/RESET_CAMERA
• wait for 2min(!)
• enable all FTUs (in the FTU tab of your screen session) ftmctrl/ENABLE_FTU -1 1
• reconnect the FADs with FAD_CONTROL/START or one-by-one

start up - connection problem

Symptom

• problem occurs usually during start up
• after FAD_CONTROL/START or pushing FAD -> START button in the GUI, not all 40 FAD LEDs are green
• fadctrl is in state Connecting (instead of Connected)

Solution

• stop dimscripts
  • dimctrl --stop from a bash
• do a crate reset:

Not Helping

• disconnect / reconnect to the FAD.
• waiting
• reset other / all crates (might just create another of these bugs in another FAD)
• stopping or killing fadctrl
• power cycling the camera (might just create another of these bugs in another FAD)

In-run-FAD-loss

Symptom

• during a run:
  • 4 adjacent *strange* patches in the events tab of the GUI
  • orange warnings in fadctrl-console (eventbuilder realises, that one FAD stopped sending data)
  • trigger rate drop to 0
• when the run ended and a new one is started
  • MCP hang in state Configuring3
  • fadctrl hang in state Configuring2

Solution

• find out which FAD board is disconnected
  • the one board in the FAD tab which has the strange behaviour
  • the one without a thick on its LED in the GUI (is only possible to see, when a new run is started and the system hangs in ConfiguringN)
• stop dimscript (by dimctrl --stop from a bash)
  • only possible when dimctrl was started as a server instance (dimctrl --server)
  • otherwise you have to kill dimctrl via ctrl + c
• reset MCP (MCP/RESET)
• Disconnect the problematic FAD (clicking on the corresponding LED in the GUI
• wait 3 to 5 sec
• Reconnect the problematic FAD (again clicking)

Not Helping

• waiting
• reset any crates (might just create another start up connection problem)
• stopping or killing fadctrl
• power cycling the camera (might just create another start up connection problem)

DRS underflow problem

Symptom

• happens usually during start up
• during first data taking one single patch in the events tab appears different (e.g. all dark blue)
• after the 3 calibration runs are taken by the system, the calibration constants behave different:
  • the magenta line (normally around +1000mV) has pretty long error bars and is curved over the whole canvas

Solution

• stop dimscript
  • dimctrl --stop from a bash
• Reset crate (see <A HREF="#tscratereset">above</A>)
  • this doesn't helped often, but it doesn't need so much time as the power cycle
• if the problem appears again: power cycle the camera (to do so please follow the procedure in Hardware )

Not Helping

• all the rest

startup - no proper connection problem

Symptom

• happens usually during start up
• several FADs are in state "Waiting" (orange in the GUI)

Solution

• disenable the corresponding FADs by clicking in the GUI on them
• wait a short time
• enable the corresponding FADs by clicking again on them (wait 3 seconds between two clicks)

Not Helping

• killing / quiting of fadctrl

fadctrl hangs in state configuring by taking an external lp run

Symptom

• fadctrl hangs in state configuring when it starts the external lp run during datataking (point 17. under datatakingdetails#normaldata normal datataking procedure the 9. run)
• all FADs are connected (if not all FADs are connected it is a FAD connection loss (see above)

Solution

• normally the shifter just forgot to open the lid, so
  • stop script
  • open the lid (see preparation#startup Start Up procedure , point 9)

Not Helping

• all the rest