= Data Taking Procedure - Main Page = **Remarks** * This is just a short introduction/summary for doing a shift. For more informations (and for becoming an expert) follow the links!! * There are always changes to the system. Therefore please check this pages and the [[https://www.fact-project.org/logbook/forumdisplay.php?fid=5|shift]] forum regularly. * The system is being automatized more and more, but this doesn't mean that there is nothing to do anymore. **Monitoring** the system is **very important**. ---- == Datataking Check List == DataTakingCheckList: Check List for a whole shift. Be aware that you have to read the following points also, not only the Check list! ---- == Preparation == Preparation before going on shift: - PreparationBeforeGoingOnShift Preparation before nightly shift - read ALL [[https://www.fact-project.org/logbook/forumdisplay.php?fid=2|logbook]] and [[https://www.fact-project.org/logbook/forumdisplay.php?fid=5|shift]] forum entries - start well in advance so that you still have time to contact the experts in case of problems or questions - make sure that you have the important telephone numbers also offline for emergency cases ---- == Startup == **Please use your private account for data taking. Login as *fact* user only when you really have to access the screen session with the programs, which normally not the case. You can use the *follow.sh programname* to follow the log-file of any program or the web-interface. You can do all necessary operations like starting a script from your user account or smartfact. You can check the container temperature in Munin. Using the *fact* account you only risk to interfere unintentionally with data taking by e.g. pressing Ctrl-C in the wrong screen-tab.** 0. open Smartfact 0. open the gtc-skycamera image and the weather station of Magic (see https://fact-project.org/). More very helpful information like the ING infrared skycam or the TNG dust measurement can be also found there in the section "Weather Info" 0. open a ssh-connection to newdaq to submit scripts 0. open the page http://www.fact-project.org/showlog/index.php (for checking the logfiles) - use your own account for this and not the fact user! - different logs can be checked by adding the name of the program, e.g. http://www.fact-project.org/showlog/?log=dimserver or https://www.fact-project.org/showlog/?log=magiclidar - alternatively change directory to ~fact/operation and do *follow.sh programname*, e.g. *follow.sh dimserver* (to follow the output of the dimserver which is processing the scripts) 0. [[https://www.fact-project.org/logbook/showthread.php?tid=828|Introduction for a VPN session]] gives a helpful introduction for a VPN session to La Palma 0. connect to the screen session on newdaq. See ScreenSessionHelpfulCommands for a list of helpful commands for screen sessions 0. open VNC-session on gui (address: 10.0.100.24,currently port :1) - start GUI, if not already running) (see ManuallyStartup) 0. open VNC-session on aux (address: 10.0.100.22,currently port :2) - start cosy, if not already running (see ManuallyStartup) 0. start the Shiftsummary in the logbook: === Logbook - Shift summary === The shift summary (first entry) has to contain the following information: - contact details - planned schedule - achieved schedule - problems - ORM alerts See [[TemplateShiftSummary|Template for Shift Summary]] 10. plan the schedule: === Schedule === - fill the schedule according to the thread "observations (month JJJJ)" in the [[https://www.fact-project.org/logbook/forumdisplay.php?fid=5|shift]] forum - use the [[https://fact-project.org/scheduling/scheduling.php|visibility and current prediction plots]] to plan the sources to observe - for finding a good position for a ratescan, you can use software planetariums like stellarium or KStars - good position for a ratescan: zd < 10°, no bright star in the field of view - a tutorial for choosing a position with xephem can be found in this [[https://www.fact-project.org/logbook/showthread.php?tid=1771|thread]] in the shift forum - edit ~/operation/scripts/schedule.js - you can find the formate of the schedule in the ScheduleTemplate - don't forget to change the date of the observation tasks - schedule the startup ~10 minutes before start of astronomical twilight, the first source observation 1 minute after start of astronomical twilight and the shutdown ~8 minutes before end of astronomical twilight - you can find the start and end of the astronomical twilight in smartfact->weather->sun - you can add several tasks to one observation, for example a ratescan following by normal datataking - if you specify the coordinate, notice: ra is in decimal hours and dec in decimal degree **Remark**: When there is no time left for a first-drs-calib until the start of astronomical twilight, please skip it (i.e. the single pe run), as data taking has first priority. 11. start dimctrl client in any terminal: ''/home/fact/operation/**dimctrl**'' - Power Drive: - PWR_CONTROL/TOGGLE_DRIVE - unlock Drive: - DRIVE_CONTROL/UNLOCK - stop dimctrl: .q 0. insert schedule to database: in /home/fact/operation : dimctrl --quit --cmd ".js scripts/updateSchedule.js" 0. Start TPoint-camera - visit the page 10.0.100.230 from the La Palma Network - Login and start the TPoint Camera 0. Check status of the Telescope in the TPoint Camera and the [[https://fact-project.org/cam/index.php|IR Camera]] 0. Check status in the container (via the program temperature) - if the temperature in the container exceed 40°C don't start the drive - wait until the temperature decreases below 40°C, maybe you can ask someone on La Palma to open the container door 0. Check weather conditions, by checking the informations from the magicweatherstation, from the tng dust measurement and from the GTC allsky camera - insert the informations (including the image from the allsky cam) in the logbook entry 0. All other task of the startup are performed by the Main.js script (see first point of Datataking) - The list of the ManuallyStartup is currently **not** needed - Note that once the script is started, there is no need to stop it to change the schedule. You can change the schedule at any time. You can even start Main.js before you update the schedule (but make sure the last schedule in the database is at least 10 hours ago. ---- == Datataking == - start Datataking script from a bash with following command in ''~/operation'': - dimctrl --start scripts/Main.js - if you have to start a dim-script, java-script or command for yourself, can call it with following command from a bash: - dimctrl --batch !ScriptsForDimCtrl/.dim (for a dim script) - dimctrl --start scripts/.js (for a javascript script) - dimctrl --quit --command ".js scripts/updateSchedule.js" (javascript via command) - dimctrl --quit --command "DRIVE_CONTROL/UNLOCK" (single command) - Monitor the system and in special the currents - limits for the currents are defined in the DatatakingLimits - if the limits for the currents are exceeded you have to stop datataking: - stop the script (dimctrl --stop from a bash) - stop Trigger (MCP/STOP or FTM_CONTROL/STOP_TRIGGER - ramp down the voltage (BIAS_CONTROL/SET_ZERO_VOLTAGE) - close open files (FAD_CONTROL/CLOSE_OPEN_FILES) - adapt your schedule (edit ~/operation/scripts/schedule.js) and update the schedule (dimctrl --quit --cmd ".js scripts/updateSchedule.js") Note, that only entries which are schedules for *now* or the future are transferred to the database. All previous entries stay like they are. - you can write also your own schedule file and transfer its content to the database by given the filepath to the command: (dimctrl --quit --cmd ".js scripts/updateSchedule.js file=") - Be aware of all other DatatakingLimits - If you have to track manually (the script sends the tracking commands, so the manually tracking is **not** needed normally), have a look at the CurrentPointingPositions - To control the hardware remotely, please have a look at RemoteControlHardware - For details of the different procedures of the datataking, see the DatatakingProcedures - be aware that all steps mentioned there are currently proceeded by the scripts and it is **not** needed to perform them manually. - You can find the list of the taken runs at: https://www.fact-project.org/smartfact/#observations - Monitor the [[http://fact-project.org/dch/qla.php?expert=yes | QLA-website]] and give an alert in case of high excess rates - look in the [[http://www.fact-project.org/logbook/forumdisplay.php?fid=5 | Shift-Forum]] to get a trigger criterium for an alert - don't forget to write the corresponding logbook entries and the comments in the run-comment DB: === Logbook - Datataking entries === General remarks: - Datataking has first priority, so first solve an occuring problem, then write the logbook entries - please fill the logbook as soon as possible (so first Problem solving, then immediatly logbook entries), so that the timestamp of the entries are correct - make a single entry for every measurement you do (one entry for every source, for every technical measurement ...) - insert following informations - weather: skybrightness, clouds, calima, ... i.e. any information that cannot be retrieved from the auxiliary files - please report also mistakes or missing information in the manuals so that they can be fixed/added - for a ratescan please insert the following informations: - position (Ra, Dec) - Zd angle - screenshot of the ratescan (in smartfact or in the GUI) - if a problem occurs, please write down following: - what happens, please do not only post the error message, but also some lines above - what you did, to solve the problem - how log you need to solve the problem TemplateNormalDatatakingSet TemplateQuotingLogFiles === Run comment DB - Entries === - **problems / information for single runs have to be inserted in the [[https://www.fact-project.org/run_db/db/run_comment.php|run-comment DB]]** - these are for example: - fad losses - tracking stops - car flash ---- == Trouble Shooting == - **never** stop a not hanging program with ''ctrl+c'' - when a restart of the program is really necessary use .q instead - if the program is started without console, you can try to restart with *dimctrl --restart SERVER_NAME* (SERVER_NAME might, e.g., by FAD_CONTROL) or to exit do *dimctrl --cmd "SERVER_NAME/EXIT 0"* - restarting a program is in most cases not a solution and only increase the risk to trigger more problems. So avoid restarting programs as long as possible - TroubleShooting Most common problems: - [[troubleshooting#tsfadloss FAD losses]] - [[troubleshooting#tsdrivectrl Drive Error ]] ---- == Shutdown == 0. After the shutdown task in the schedule was proceeded you have to do the the following task to finish the shutdown: 0. stop datataking script with following command from a bash: - dimctrl --stop 0. make sure that - no script is running - the bias is off - the trigger is off - the lid is closed - check TPoint Camera - if it is too dark outside you can switch on the IR-Lights via the [[https://fact-project.org/cam/index.php|FACT IR-Webcam page]] - stop TPoint Camera after check (via the page 10.0.100.230 from the internal network) - the telescope is parked - check [[https://fact-project.org/cam/index.php|IR Camera]] - biasctrl disconnected - all logbook entries are finished 0. stop the drive by: - in dimctrl: PWR_CONTROL/TOGGLE_DRIVE 0. Please fill out the shutdown Checklist: http://fact-project.org/Checklist/ If you have to shutdown the hardware on the island, please look at HardwareManuallyShutdown