wiki:DataTakingMainPage

Version 58 (modified by ghughes, 10 years ago) ( diff )

--

Data Taking Procedure - Main Page

Remarks

  • This is just a short introduction/summary for doing a shift. For more informations (and for becoming an expert) follow the links!!
  • There are always changes to the system. Therefore please check this pages and the ​shift forum regularly.
  • The system is being automatized more and more, but this doesn't mean that there is nothing to do anymore. Monitoring the system is very important.

Datataking Check List

DataTakingCheckList: Check List for a whole shift. Be aware that you have to read the following points also, not only the Check list!

Preparation

Preparation before going on shift:

Preparation before nightly shift

  • read ALL ​logbook and ​shift forum entries
  • start well in advance so that you still have time to contact the experts in case of problems or questions
  • make sure that you have the important telephone numbers also offline for emergency cases

Startup

Please use your private account for data taking. Login as *fact* user only when you really have to access the screen session with the programs, which normally not the case. You can use the *follow.sh programname* to follow the log-file of any program or the web-interface. You can do all necessary operations like starting a script from your user account or smartfact. You can check the container temperature in Munin. Using the *fact* account you only risk to interfere unintentionally with data taking by e.g. pressing Ctrl-C in the wrong screen-tab.

  1. open Smartfact
  2. open the gtc-skycamera image and the weather station of Magic (see ​https://fact-project.org/). More very helpful information like the ING infrared skycam or the TNG dust measurement can be also found there in the section "Weather Info"
  3. open a ssh-connection to newdaq to submit scripts
  4. open the page ​http://www.fact-project.org/showlog/index.php (for checking the logfiles)
  5. ​Introduction for a VPN session gives a helpful introduction for a VPN session to La Palma
  6. connect to the screen session on newdaq. See ScreenSessionHelpfulCommands for a list of helpful commands for screen sessions
  7. open VNC-session on gui (address: 10.0.100.24,currently port :1)
  8. open VNC-session on aux (address: 10.0.100.22,currently port :2)
  9. start the Shiftsummary in the logbook:

Logbook - Shift summary

The shift summary (first entry) has to contain the following information:

  • contact details
  • planned schedule
  • achieved schedule
  • problems
  • ORM alerts

See Template for Shift Summary

  1. plan the schedule:

Schedule

  • fill the schedule according to the thread "observations (month JJJJ)" in the ​shift forum
  • use the ​visibility and current prediction plots to plan the sources to observe
  • for finding a good position for a ratescan, you can use software planetariums like stellarium or KStars
    • good position for a ratescan: zd < 10°, no bright star in the field of view
    • a tutorial for choosing a position with xephem can be found in this ​thread in the shift forum
  • edit ~/operation/scripts/schedule.js
    • you can find the formate of the schedule in the ScheduleTemplate
    • don't forget to change the date of the observation tasks
    • schedule the startup ~10 minutes before start of astronomical twilight, the first source observation 1 minute after start of astronomical twilight and the shutdown ~8 minutes before end of astronomical twilight
    • you can find the start and end of the astronomical twilight in smartfact->weather->sun
    • you can add several tasks to one observation, for example a ratescan following by normal datataking
    • if you specify the coordinate, notice: ra is in decimal hours and dec in decimal degree

Remark: When there is no time left for a first-drs-calib until the start of astronomical twilight, please skip it (i.e. the single pe run), as data taking has first priority.

  1. start dimctrl client in any terminal: /home/fact/operation/dimctrl
    • Power Drive:
      • PWR_CONTROL/TOGGLE_DRIVE
    • unlock Drive:
      • DRIVE_CONTROL/UNLOCK
    • stop dimctrl: .q
  2. insert schedule to database: in /home/fact/operation : dimctrl --quit --cmd ".js scripts/updateSchedule.js"
  3. Start TPoint-camera
    • visit the page 10.0.100.230 from the La Palma Network
    • Login and start the TPoint Camera
  4. Check status of the Telescope in the TPoint Camera and the ​IR Camera
  5. Check status in the container (via the program temperature)
    • if the temperature in the container exceed 40°C don't start the drive
    • wait until the temperature decreases below 40°C, maybe you can ask someone on La Palma to open the container door
  6. Check weather conditions, by checking the informations from the magicweatherstation, from the tng dust measurement and from the GTC allsky camera
    • insert the informations (including the image from the allsky cam) in the logbook entry
  7. All other task of the startup are performed by the Main.js script (see first point of Datataking)
    • The list of the ManuallyStartup is currently not needed
    • Note that once the script is started, there is no need to stop it to change the schedule. You can change the schedule at any time. You can even start Main.js before you update the schedule (but make sure the last schedule in the database is at least 10 hours ago.

Datataking

  • start Datataking script from a bash with following command in ~/operation:
    • dimctrl --start scripts/Main.js
  • if you have to start a dim-script, java-script or command for yourself, can call it with following command from a bash:
    • dimctrl --batch ScriptsForDimCtrl/<scriptname>.dim (for a dim script)
    • dimctrl --start scripts/<scriptname>.js (for a javascript script)
    • dimctrl --quit --command ".js scripts/updateSchedule.js" (javascript via command)
    • dimctrl --quit --command "DRIVE_CONTROL/UNLOCK" (single command)
  • Monitor the system and in special the currents
    • limits for the currents are defined in the DatatakingLimits
    • if the limits for the currents are exceeded you have to stop datataking:
    • stop the script (dimctrl --stop from a bash)
    • stop Trigger (MCP/STOP or FTM_CONTROL/STOP_TRIGGER
    • ramp down the voltage (BIAS_CONTROL/SET_ZERO_VOLTAGE)
    • close open files (FAD_CONTROL/CLOSE_OPEN_FILES)
    • adapt your schedule (edit ~/operation/scripts/schedule.js) and update the schedule (dimctrl --quit --cmd ".js scripts/updateSchedule.js") Note, that only entries which are schedules for *now* or the future are transferred to the database. All previous entries stay like they are.
    • you can write also your own schedule file and transfer its content to the database by given the filepath to the command: (dimctrl --quit --cmd ".js scripts/updateSchedule.js file=<!pathToScheduleFile>")
  • Be aware of all other DatatakingLimits
  • If you have to track manually (the script sends the tracking commands, so the manually tracking is not needed normally), have a look at the CurrentPointingPositions
  • To control the hardware remotely, please have a look at RemoteControlHardware
  • For details of the different procedures of the datataking, see the DatatakingProcedures
    • be aware that all steps mentioned there are currently proceeded by the scripts and it is not needed to perform them manually.
  • You can find the list of the taken runs at: ​https://www.fact-project.org/smartfact/#observations
  • Monitor the ​ QLA-website and give an alert in case of high excess rates
  • don't forget to write the corresponding logbook entries and the comments in the run-comment DB:

Logbook - Datataking entries

General remarks:

  • Datataking has first priority, so first solve an occuring problem, then write the logbook entries
  • please fill the logbook as soon as possible (so first Problem solving, then immediatly logbook entries), so that the timestamp of the entries are correct
  • make a single entry for every measurement you do (one entry for every source, for every technical measurement ...)
  • insert following informations
    • weather: skybrightness, clouds, calima, ... i.e. any information that cannot be retrieved from the auxiliary files
    • please report also mistakes or missing information in the manuals so that they can be fixed/added
  • for a ratescan please insert the following informations:
    • position (Ra, Dec)
    • Zd angle
    • screenshot of the ratescan (in smartfact or in the GUI)
  • if a problem occurs, please write down following:
    • what happens, please do not only post the error message, but also some lines above
    • what you did, to solve the problem
    • how log you need to solve the problem

TemplateNormalDatatakingSet

TemplateQuotingLogFiles

Run comment DB - Entries

  • problems / information for single runs have to be inserted in the ​run-comment DB
    • these are for example:
    • fad losses
    • tracking stops
    • car flash

Trouble Shooting

  • never stop a not hanging program with ctrl+c
    • when a restart of the program is really necessary use .q instead
    • if the program is started without console, you can try to restart with *dimctrl --restart SERVER_NAME* (SERVER_NAME might, e.g., by FAD_CONTROL) or to exit do *dimctrl --cmd "SERVER_NAME/EXIT 0"*
    • restarting a program is in most cases not a solution and only increase the risk to trigger more problems. So avoid restarting programs as long as possible
  • TroubleShooting

Most common problems:

Shutdown

  1. After the shutdown task in the schedule was proceeded you have to do the the following task to finish the shutdown:
  2. stop datataking script with following command from a bash:
    • dimctrl --stop
  3. make sure that
    • no script is running
    • the bias is off
    • the trigger is off
    • the lid is closed
      • check TPoint Camera
      • if it is too dark outside you can switch on the IR-Lights via the ​FACT IR-Webcam page
      • stop TPoint Camera after check (via the page 10.0.100.230 from the internal network)
    • the telescope is parked
    • biasctrl disconnected
    • all logbook entries are finished
  4. stop the drive by:
    • in dimctrl: PWR_CONTROL/TOGGLE_DRIVE
  5. Please fill out the shutdown Checklist: ​http://fact-project.org/Checklist/
    • make sure you have already chosen and informed someone to check you shutdown everything ok: use them in the "Nominated Checker's Email" field.

If you have to shutdown the hardware on the island, please look at HardwareManuallyShutdown

Note: See TracWiki for help on using the wiki.