Data Taking Procedure - Main Page

Remarks

• This is just a short introduction/summary for doing a shift. For more informations (and for becoming an expert) follow the links!!
• There are always changes to the system. Therefore please check these pages and the ​ShiftForum regularly.
• Also in this document, you can find some useful information ​http://www.fact-project.org/meetings/Dortmund2014/dorner_status_operations.pdf
• The system is being automatized more and more, but this doesn't mean that there is nothing to do anymore. Monitoring the system is very important. Check especially:
  • system status: ​Smartfact
  • weather (links on ​main page)
  • quick look analysis: ​QLA (​new version - still in test phase)

---

Check List Preparation and Datataking

DataTakingCheckList: Check list for a whole shift (preparation and all nights).

Remarks

• Read the complete manual, not only the check list!
• Prepare for your shift well in advance!
• Start well in advance so that you still have time to contact the experts in case of problems or questions.

---

Startup

Please use your private account for data taking. Login as *fact* user only when you really have to access the screen session with the programs, which normally not the case. You can use the *follow.sh programname* to follow the log-file of any program or the web-interface. You can do all necessary operations like starting a script from your user account or smartfact. You can check the container temperature in Munin. Using the *fact* account you only risk to interfere unintentionally with data taking by e.g. pressing Ctrl-C in the wrong screen-tab.

0. open Smartfact - you can monitor and operate the system from there
1. open the gtc-skycamera image and the weather station of Magic (see ​https://fact-project.org/). More very helpful information like the ING infrared skycam or the TNG dust measurement can be also found there in the section "Weather Info"
2. open the page ​http://www.fact-project.org/showlog/index.php (for checking the logfiles)
   • use your own account for this and not the fact user!
   • different logs can be checked by adding the name of the program, e.g. ​http://www.fact-project.org/showlog/?log=dimserver or ​https://www.fact-project.org/showlog/?log=magiclidar
   • alternatively change directory to ~fact/operation and do *follow.sh programname*, e.g. *follow.sh dimserver* (to follow the output of the dimserver which is processing the scripts)

additional information:

• ​Introduction for a VPN session gives a helpful introduction for a VPN session to La Palma - you need this to access internatl websites and don't need tunnels to access the vnc-sessions
• The screen session on newdaq has most programs running
• See ScreenSessionHelpfulCommands for a list of helpful commands for screen sessions
• On aux there is a vnc-session for the drivesystem (with the tpoint camera) and on gui one for the old gui. More details on TroubleShootingSoftware To connect to a vnc-session you may use a vncviewer of your choice, e.g. vinagre aux:1 To find out which desktop number to connect to, you can do ps aux | grep vnc on aux and look for the :X in the command where X is the number of the desktop.

4. start the Shiftsummary in the logbook:

Logbook - Shift summary

The shift summary (first entry) has to contain the following information:

• contact details
• planned schedule
• achieved schedule
• problems
• ORM alerts

See Template for Shift Summary

5. plan the schedule:

Schedule

• fill the schedule according to the thread "observations (month JJJJ)" in the ​shift forum
• use the ​visibility and current prediction plots to plan the sources to observe
• for finding a good position for a ratescan, you can use software planetariums like stellarium or KStars
  • good position for a ratescan: zd < 10°, no bright star in the field of view
  • a tutorial for choosing a position with xephem can be found in this ​thread in the shift forum
  • A detailed explanation of the different ratescan-types that can be scheduled is given here
• edit ~/operation/scripts/schedule.js
  • you can find the formate of the schedule in the ScheduleTemplate
  • don't forget to change the date of the observation tasks
  • schedule the startup ~10 minutes before start of astronomical twilight, the first source observation 1 minute after start of astronomical twilight and the shutdown ~8 minutes before end of astronomical twilight
  • you can find the start and end of the astronomical twilight in smartfact->weather->sun
  • you can add several tasks to one observation, for example a ratescan following by normal datataking
  • if you specify the coordinate, notice: ra is in decimal hours and dec in decimal degree

Remark: When there is no time left for a first-drs-calib until the start of astronomical twilight, please skip it (i.e. the single pe run), as data taking has first priority.

6. insert schedule to database either from ​https://www.fact-project.org/schedule/ or with dimctrl: in /home/fact/operation : dimctrl --quit --cmd ".js scripts/updateSchedule.js"
7. Check status of the Telescope in the TPoint Camera (in cosy on aux; switchable on 10.0.1002.30 La Palma Network (-> you need vpn connection)) or Lid Camera ​http://fact-project.org/cam/lidcam.php and the ​IR Camera
8. Check status in the container (via the program temperature or munin)
   • if the temperature in the container exceed 40°C don't start the drive
   • wait until the temperature decreases below 40°C, maybe you can ask someone on La Palma to open the container door
9. Check weather conditions, by checking the informations from the magicweatherstation, from the tng dust measurement and from the GTC allsky camera
   • insert the informations (including the image from the allsky cam) in the logbook entry
10. switch on drive and unlock it: either from smartfact or from dimctrl:
    • start dimctrl client in any terminal: /home/fact/operation/dimctrl
    • Power Drive:
      • PWR_CONTROL/TOGGLE_DRIVE
    • unlock Drive:
      • DRIVE_CONTROL/UNLOCK
    • stop dimctrl: .q
11. All other task of the startup are performed by the Main.js script (see first point of Datataking)
    • The list of the ManuallyStartup is currently not needed
    • Note that once the script is started, there is no need to stop it to change the schedule. You can change the schedule at any time. You can even start Main.js before you update the schedule (but make sure the last schedule in the database is at least 10 hours ago.

---

Datataking

• start Datataking script Main.js from smartfact or from a bash with following command in /home/fact/operation:
  • dimctrl --start scripts/Main.js
• if you have to start a dim-script, java-script or command for yourself, can call it with following command from a bash:
  • dimctrl --batch ScriptsForDimCtrl/<scriptname>.dim (for a dim script)
  • dimctrl --start scripts/<scriptname>.js (for a javascript script)
  • dimctrl --quit --command ".js scripts/updateSchedule.js" (javascript via command)
  • dimctrl --quit --command "DRIVE_CONTROL/UNLOCK" (single command)
• Monitor the system and in special the currents
  • limits for the currents are defined in the DatatakingLimits
  • if the limits for the currents are exceeded you have to stop datataking:
  • stop the script (dimctrl --stop from a bash)
  • stop Trigger (MCP/STOP or FTM_CONTROL/STOP_TRIGGER
  • ramp down the voltage (BIAS_CONTROL/SET_ZERO_VOLTAGE)
  • close open files (FAD_CONTROL/CLOSE_OPEN_FILES)
  • adapt your schedule (edit ~/operation/scripts/schedule.js) and update the schedule (dimctrl --quit --cmd ".js scripts/updateSchedule.js") Note, that only entries which are schedules for *now* or the future are transferred to the database. All previous entries stay like they are.
  • you can write also your own schedule file and transfer its content to the database by given the filepath to the command: (dimctrl --quit --cmd ".js scripts/updateSchedule.js file=<!pathToScheduleFile>")
• Be aware of all other DatatakingLimits
• If you have to track manually (the script sends the tracking commands, so the manually tracking is not needed normally), have a look at the CurrentPointingPositions
• To control the hardware remotely, please have a look at RemoteControlHardware
• For details of the different procedures of the datataking, see the DatatakingProcedures
  • be aware that all steps mentioned there are currently proceeded by the scripts and it is not needed to perform them manually.
• You can find the list of the taken runs at: ​https://www.fact-project.org/smartfact/#observations
• Monitor the ​ QLA-website and give an alert in case of high excess rates
  • look in the ​ Shift-Forum to get a trigger criterium for an alert
• don't forget to write the corresponding logbook entries and the comments in the run-comment DB:

Logbook - Datataking entries

General remarks:

• Datataking has first priority, so first solve an occuring problem, then write the logbook entries
• please fill the logbook as soon as possible (so first Problem solving, then immediatly logbook entries), so that the timestamp of the entries are correct
• make a single entry for every measurement you do (one entry for every source, for every technical measurement ...)
• insert following informations
  • weather: skybrightness, clouds, calima, ... i.e. any information that cannot be retrieved from the auxiliary files
  • please report also mistakes or missing information in the manuals so that they can be fixed/added
• for a ratescan please insert the following informations:
  • position (Ra, Dec)
  • Zd angle
  • screenshot of the ratescan (in smartfact or in the GUI)
• if a problem occurs, please write down following:
  • what happens, please do not only post the error message, but also some lines above
  • what you did, to solve the problem
  • how log you need to solve the problem

TemplateNormalDatatakingSet

TemplateQuotingLogFiles

Run comment DB - Entries

• problems / information for single runs have to be inserted in the ​run-comment DB
  • these are for example:
  • fad losses
  • tracking stops
  • car flash

---

Trouble Shooting

• never stop a not hanging program with ctrl+c
  • when a restart of the program is really necessary use .q instead
  • if the program is started without console, you can try to restart with *dimctrl --restart SERVER_NAME* (SERVER_NAME might, e.g., by FAD_CONTROL) or to exit do *dimctrl --cmd "SERVER_NAME/EXIT 0"*
  • restarting a program is in most cases not a solution and only increase the risk to trigger more problems. So avoid restarting programs as long as possible
• TroubleShooting
• List with known Hardware problems: https://trac.fact-project.org/wiki/KnownProblems)

Most common problems:

• troubleshooting#tsfadloss FAD losses
• troubleshooting#tsdrivectrl Drive Error

---

Shutdown

manual shutdown procedure, in case it cannot be done with Main.js

0. After the shutdown task in the schedule was proceeded you have to do the the following task to finish the shutdown:
1. stop datataking script with following command from a bash:
   • dimctrl --stop
2. make sure that
   • no script is running
   • the bias is off
   • the trigger is off
   • the lid is closed
     • check TPoint Camera
     • if it is too dark outside you can switch on the IR-Lights via the ​FACT IR-Webcam page
     • stop TPoint Camera after check (via the page 10.0.100.230 from the internal network)
   • the telescope is parked
     • check ​IR Camera
   • biasctrl disconnected
   • all logbook entries are finished
3. stop the drive by:
   • in dimctrl: PWR_CONTROL/TOGGLE_DRIVE
4. Please fill out the shutdown Checklist: ​http://fact-project.org/Checklist/
   • make sure you have already chosen and informed someone to check you shutdown everything ok: use them in the "Nominated Checker's Email" field.

If you have to shutdown the hardware on the island, please look at HardwareManuallyShutdown