wiki:TroubleShootingFads

Trouble Shooting - FADs

Crate Reset

There exist a script which will do the Crate Reset automatically:

  • dimctrl --quit --cmd ".js scripts/doCrateReset.js crate[n]=true"
  • n = number of crate you want to reset (e.g. crate2=true)

Crate Reset manually:

  • disable all FTUs (in the FTU tab of your screen session) ftmctrl/ENABLE_FTU -1 0
  • disconnect the 10 FADs in the crate, by clicking the LEDs
  • ftmctrl/RESET_CRATE x (x corresponding crate)
  • enable all FTUs (in the FTU tab of your screen session) ftmctrl/ENABLE_FTU -1 1
  • reconnect the 10 FADs, one by one with 3 seconds waiting in between (the FAD needs this time to boot)

start up - connection problem

Symptom

  • problem occurs usually during start up
  • after FAD_CONTROL/START or pushing FAD -> START button in the GUI, not all 40 FAD LEDs are green
  • fadctrl is in state Connecting (instead of Connected)

Solution

  • stop dimscripts
    • dimctrl --stop from a bash
  • do a crate reset:

Not Helping

  • disconnect / reconnect to the FAD.
  • waiting
  • reset other / all crates (might just create another of these bugs in another FAD)
  • stopping or killing fadctrl
  • power cycling the camera (might just create another of these bugs in another FAD)

In-run-FAD-loss

Symptom

  • during a run:
    • 4 adjacent *strange* patches in the events tab of the GUI
    • orange warnings in fadctrl-console (eventbuilder realises, that one FAD stopped sending data)
    • trigger rate drop to 0
  • when the run ended and a new one is started
    • MCP hang in state Configuring3
    • fadctrl hang in state Configuring2

Solution

  • find out which FAD board is disconnected
    • the one board in the FAD tab which has the strange behaviour
    • the one without a thick on its LED in the GUI (is only possible to see, when a new run is started and the system hangs in ConfiguringN)
  • stop dimscript (by dimctrl --stop from a bash)
    • only possible when dimctrl was started as a server instance (dimctrl --server)
    • otherwise you have to kill dimctrl via ctrl + c
  • reset MCP (MCP/RESET)
  • Disconnect the problematic FAD (clicking on the corresponding LED in the GUI
  • wait 3 to 5 sec
  • Reconnect the problematic FAD (again clicking)

Not Helping

  • waiting
  • reset any crates (might just create another start up connection problem)
  • stopping or killing fadctrl
  • power cycling the camera (might just create another start up connection problem)

DRS underflow problem

Symptom

  • happens usually during start up
  • during first data taking one single patch in the events tab appears different (e.g. all dark blue)
  • after the 3 calibration runs are taken by the system, the calibration constants behave different:
    • the magenta line (normally around +1000mV) has pretty long error bars and is curved over the whole canvas

Solution

  • stop dimscript
    • dimctrl --stop from a bash
  • Reset crate (see <A HREF="#tscratereset">above</A>)
    • this doesn't helped often, but it doesn't need so much time as the power cycle
  • if the problem appears again: power cycle the camera (to do so please follow the procedure in Hardware )

Not Helping

  • all the rest

startup - no proper connection problem

Symptom

  • happens usually during start up
  • several FADs are in state "Waiting" (orange in the GUI)

Solution

  • disenable the corresponding FADs by clicking in the GUI on them
  • wait a short time
  • enable the corresponding FADs by clicking again on them (wait 3 seconds between two clicks)

Not Helping

  • killing / quiting of fadctrl

fadctrl hangs in state configuring by taking an external lp run

Symptom

Solution

Not Helping

  • all the rest
Last modified 4 years ago Last modified on Jun 4, 2015, 9:57:13 PM