# # ChangeLog for trunk/DataCheck/Transfer # # Generated by Trac 1.5.2 # 11/11/24 14:14:10 Tue, 04 Jun 2019 19:34:29 GMT Daniela Dorner [19549] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) add grb-run Thu, 04 Apr 2019 17:12:05 GMT Daniela Dorner [19487] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) print only warning if the missing file is older than a few days Sun, 21 May 2017 09:42:02 GMT Daniela Dorner [18875] * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (modified) adapted to use newdata instead of data for transfer Sun, 21 May 2017 09:41:23 GMT Daniela Dorner [18874] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) added additional check for QLA results Sun, 21 May 2017 09:40:55 GMT Daniela Dorner [18873] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) fixed small bug in logging Sun, 21 May 2017 09:34:57 GMT Daniela Dorner [18872] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) implemented option to use /data1 instead of /scratch Sun, 21 May 2017 09:33:52 GMT Daniela Dorner [18871] * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) adapted to use proper rsync-temp-dir for each transfer Sun, 14 May 2017 22:38:52 GMT Daniela Dorner [18867] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) adapted logging to usage of newdata Sun, 14 May 2017 20:05:33 GMT Daniela Dorner [18863] * trunk/DataCheck/Transfer/CleanupNewdata.sh (added) added (script to clean up newdata) Sun, 14 May 2017 20:01:03 GMT Daniela Dorner [18856] * trunk/DataCheck/Transfer/BackupQLA.sh (modified) updated for newdata as download machine in LP Sun, 14 May 2017 19:59:09 GMT Daniela Dorner [18854] * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) updated to run on newdata and create a second copy Sun, 14 May 2017 19:58:29 GMT Daniela Dorner [18853] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) updated to run on newdata and create a second copy Sun, 14 May 2017 19:57:08 GMT Daniela Dorner [18852] * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (modified) * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) updated to use newdata instead of data for transferring data Sun, 14 May 2017 01:29:35 GMT Daniela Dorner [18851] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) adapted as discussed in telcon (implemented warning if files in ... Sat, 08 Apr 2017 02:17:03 GMT Daniela Dorner [18807] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved logging Thu, 02 Mar 2017 02:37:07 GMT Daniela Dorner [18778] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) fixed typo Tue, 21 Feb 2017 21:21:56 GMT Daniela Dorner [18775] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) fixed bug and added check Sat, 04 Feb 2017 19:51:45 GMT Daniela Dorner [18747] * trunk/DataCheck/Transfer/BackupQLA.sh (modified) removed debugging output Sat, 04 Feb 2017 16:41:48 GMT Daniela Dorner [18746] * trunk/DataCheck/Transfer/BackupQLA.sh (modified) implemented speedup (rsync from/to ISDC took 1.5-2h each) Sat, 04 Feb 2017 16:27:30 GMT Daniela Dorner [18744] * trunk/DataCheck/Transfer/BackupDatabase.sh (modified) fix in path Wed, 25 Jan 2017 23:43:24 GMT Daniela Dorner [18739] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved logging Sun, 08 Jan 2017 17:21:53 GMT Daniela Dorner [18723] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) files in the fails folder are considered now "fine" in the context of ... Thu, 05 Jan 2017 00:56:47 GMT Daniela Dorner [18709] * trunk/DataCheck/Transfer/BackupQLA.sh (modified) added backup of QLA to Wue Wed, 21 Dec 2016 12:37:49 GMT Daniela Dorner [18701] * trunk/DataCheck/DataCheck/InfoForMWL.sh (modified) * trunk/DataCheck/QuickLook/SendData.sh (modified) * trunk/DataCheck/QuickLook/SendData_Internal.sh (modified) * trunk/DataCheck/Sourcefile.sh (modified) * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) fixed bug (variable pwfile occured twice -> renamed) Wed, 21 Dec 2016 12:36:32 GMT Daniela Dorner [18700] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved logging Sun, 18 Dec 2016 20:54:38 GMT Daniela Dorner [18686] * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) improved logging Sun, 18 Dec 2016 20:50:56 GMT Daniela Dorner [18685] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved logging Wed, 02 Nov 2016 03:33:54 GMT Daniela Dorner [18671] * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) changed back to not using the rsync server which was gone after the ... Wed, 02 Nov 2016 03:33:32 GMT Daniela Dorner [18670] * trunk/DataCheck/Transfer/BackupQLA.sh (modified) changed back to not using the rsync server which was gone after the ... Wed, 02 Nov 2016 03:32:52 GMT Daniela Dorner [18669] * trunk/DataCheck/Transfer/BackupDatabase.sh (modified) fixed syntax of mysqldump for new mysqlversion Sat, 10 Sep 2016 17:31:04 GMT Daniela Dorner [18582] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved output; fixed regular expression for finding rawfiles Sat, 13 Feb 2016 17:47:02 GMT Daniela Dorner [18435] * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) added output for case of certaindate Sat, 13 Feb 2016 17:46:02 GMT Daniela Dorner [18434] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved logging, additional check for DB (jobs that were reset while ... Sat, 13 Feb 2016 17:42:50 GMT Daniela Dorner [18433] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) implemented usage of certaindate Sat, 13 Feb 2016 17:41:36 GMT Daniela Dorner [18432] * trunk/DataCheck/Transfer/BackupAuxToWue.sh (modified) implemented usage of variable certaindate Sat, 02 Jan 2016 06:30:05 GMT Daniela Dorner [18397] * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (modified) removed wrong comment Mon, 31 Aug 2015 00:53:15 GMT Daniela Dorner [18328] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) commented-out what is needed if daq is unavailable Mon, 31 Aug 2015 00:23:29 GMT Daniela Dorner [18319] * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) removed check of certaindate-variable (now done in getdate()) Mon, 31 Aug 2015 00:22:44 GMT Daniela Dorner [18318] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) added commented-out the paths in case daq is not available Wed, 18 Feb 2015 09:28:14 GMT Daniela Dorner [18137] * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (modified) added usage of function getdata() Wed, 18 Feb 2015 09:27:30 GMT Daniela Dorner [18136] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) ignore runs with a run number with more than 3 digits Tue, 27 May 2014 21:15:04 GMT Daniela Dorner [17905] * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) * trunk/DataCheck/Transfer/ZipRawData.sh (modified) included and/or improved check of variable certaindate Mon, 26 May 2014 15:10:34 GMT Daniela Dorner [17890] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) do not use function checklock anymore as script is only started when ... Mon, 26 May 2014 15:09:30 GMT Daniela Dorner [17889] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved logging Mon, 17 Mar 2014 13:18:25 GMT Daniela Dorner [17619] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) improved logging in case of full disk Mon, 17 Mar 2014 12:40:04 GMT Daniela Dorner [17618] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved logging, re-enabled check if script is already running Mon, 17 Mar 2014 12:38:48 GMT Daniela Dorner [17617] * trunk/DataCheck/Monitoring/CheckDU.sh (modified) * trunk/DataCheck/QuickLook/Step1.sh (modified) * trunk/DataCheck/Transfer/ZipRawData.sh (modified) improved logging in case of full disk Sun, 16 Mar 2014 23:47:30 GMT Daniela Dorner [17616] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) added DISK tag to logging, improved logging, added certaindate and ... Sun, 16 Mar 2014 23:42:56 GMT Daniela Dorner [17612] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) added check for full disk - at 100GB transfer is stopped Sun, 02 Mar 2014 01:20:11 GMT Daniela Dorner [17594] * trunk/DataCheck/Transfer/BackupAuxToWue.sh (modified) * trunk/DataCheck/Transfer/BackupDatabase.sh (modified) * trunk/DataCheck/Transfer/BackupQLA.sh (modified) * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (modified) * trunk/DataCheck/Transfer/RsyncDBToWue.sh (modified) * trunk/DataCheck/Transfer/RsyncRawLP.sh (modified) * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) changed logging for connection problems Sun, 02 Mar 2014 00:35:31 GMT Daniela Dorner [17590] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) added copying of already compressed files Sat, 01 Mar 2014 23:59:27 GMT Daniela Dorner [17576] * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) added feature to run script manually for a certain night, added check ... Sat, 01 Mar 2014 23:57:08 GMT Daniela Dorner [17575] * trunk/DataCheck/Transfer/BackupQLA.sh (modified) updated path to qla at isdc Sat, 01 Mar 2014 23:56:07 GMT Daniela Dorner [17574] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) added rsync of drs-file to avoid failing callistos Sat, 01 Mar 2014 23:46:31 GMT Daniela Dorner [17573] * trunk/DataCheck/Transfer/BackupAuxToWue.sh (modified) implemented usage of getdates function Sat, 01 Mar 2014 23:41:53 GMT Daniela Dorner [17570] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) updated output with qlainfo, implemented fz files, following links in ... Mon, 09 Sep 2013 14:55:29 GMT Daniela Dorner [17157] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) changed number of processes for pigz to be set from the general setup ... Mon, 09 Sep 2013 13:24:03 GMT Daniela Dorner [17149] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) comments what changes needed to use rsync server Fri, 06 Sep 2013 17:08:07 GMT Daniela Dorner [17114] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) implemented usage of getdates function Fri, 06 Sep 2013 16:04:19 GMT Daniela Dorner [17103] * trunk/DataCheck/Transfer/BackupQLA.sh (modified) * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) adapted for usage of rsync servers Thu, 05 Sep 2013 13:25:27 GMT Daniela Dorner [17079] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) added check if QLA finished callisto for all data runs Wed, 31 Jul 2013 16:31:10 GMT Daniela Dorner [16992] * trunk/DataCheck/Transfer/BackupAuxToWue.sh (modified) * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) added rsync-temp-dir to avoid temporary files from the rsync on the ... Mon, 03 Jun 2013 21:42:07 GMT Daniela Dorner [16679] * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) small fix in usage of lsof Mon, 03 Jun 2013 21:40:40 GMT Daniela Dorner [16678] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) reworked output, added summary Mon, 13 May 2013 09:57:16 GMT Daniela Dorner [15599] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) check files only in detail when number of files does agree Mon, 13 May 2013 09:50:59 GMT Daniela Dorner [15596] * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) allow more than one instance of script and moved control of number of ... Mon, 13 May 2013 09:47:58 GMT Daniela Dorner [15595] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) changed order of dates for processing to make sure that older data ... Mon, 13 May 2013 09:47:07 GMT Daniela Dorner [15594] * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) changed limit to 5 MB/s during night Wed, 08 May 2013 17:16:17 GMT Daniela Dorner [15559] * trunk/DataCheck/Transfer/BackupQLA.sh (added) added (script to backup results of QLA) Tue, 07 May 2013 22:22:58 GMT Daniela Dorner [15517] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) added pv to control rate of zipping, limited pigz to 2 processes Thu, 25 Apr 2013 12:12:54 GMT Daniela Dorner [15435] * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) * trunk/DataCheck/Transfer/RsyncRawLP.sh (modified) fixed bug which was caused by the introduction of the variable ... Sat, 13 Apr 2013 23:26:27 GMT Daniela Dorner [15328] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) changed interval for crashed jobs to 5 hours Mon, 04 Feb 2013 06:03:36 GMT Daniela Dorner [14841] * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) adapted to new structure in LP (newdaq, new transfer scheme) Mon, 04 Feb 2013 06:02:39 GMT Daniela Dorner [14840] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) adapted to new structure in LP (newdaq, new transfer scheme) Mon, 04 Feb 2013 05:16:50 GMT Daniela Dorner [14824] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) updated to changes in LP (newdaq), implemented PhiDo (still ... Mon, 17 Dec 2012 21:54:26 GMT Daniela Dorner [14757] * trunk/DataCheck/Transfer/ZipRawData.sh (modified) inverted order of dates (change from months ago) Mon, 17 Dec 2012 20:23:38 GMT Daniela Dorner [14756] * trunk/DataCheck/Transfer/RsyncRawLP.sh (modified) inverted order of dates (change from months ago) Tue, 19 Jun 2012 17:25:45 GMT Daniela Dorner [14192] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) bugfix for case file size is the same but file is in fails directory ... Thu, 24 May 2012 12:13:34 GMT Daniela Dorner [13864] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved logging Thu, 24 May 2012 12:09:22 GMT Daniela Dorner [13863] * trunk/DataCheck/Transfer/RsyncRawLP.sh (modified) * trunk/DataCheck/Transfer/ZipRawData.sh (modified) reduced time span for transfer processes again Thu, 24 May 2012 11:25:00 GMT Daniela Dorner [13862] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved logging, removed old code Tue, 08 May 2012 12:41:49 GMT Daniela Dorner [13600] * trunk/DataCheck/Transfer/RsyncRawLP.sh (modified) changed time range for rsync of raw files on LP from 3 to 9 days Tue, 08 May 2012 12:40:10 GMT Daniela Dorner [13599] * trunk/DataCheck/Processing/FillAuxData.sh (modified) * trunk/DataCheck/Transfer/BackupAuxToWue.sh (modified) * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (modified) changed time range for rsync of aux files from 3 to 9 days Mon, 07 May 2012 09:42:36 GMT Daniela Dorner [13580] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) run skript only for last three days or last X GB of data Mon, 16 Apr 2012 11:47:39 GMT Daniela Dorner [13342] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) improved logging Wed, 04 Apr 2012 16:12:21 GMT Daniela Dorner [13313] * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) * trunk/DataCheck/Transfer/RsyncRawLP.sh (modified) improved logging Wed, 04 Apr 2012 15:28:17 GMT Daniela Dorner [13312] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) adapted parameters to run script via cron Wed, 04 Apr 2012 11:44:47 GMT Daniela Dorner [13305] * trunk/DataCheck/Transfer/BackupDatabase.sh (added) added (script to backup databases) Thu, 22 Mar 2012 16:55:48 GMT Daniela Dorner [13171] * trunk/DataCheck/Transfer/CheckTransfer.sh (modified) improved output, fixed bugs Mon, 19 Mar 2012 17:20:12 GMT Daniela Dorner [13154] * trunk/DataCheck/Transfer/CheckTransfer.sh (added) added (script to check status of transfer for deleting data) Mon, 19 Mar 2012 17:09:05 GMT Daniela Dorner [13152] * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) bugfix for drsfiles Wed, 14 Mar 2012 19:26:48 GMT Daniela Dorner [13108] * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (modified) added check if directory exists Wed, 14 Mar 2012 19:25:37 GMT Daniela Dorner [13107] * trunk/DataCheck/Setup/setup.lp.fact (modified) * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) * trunk/DataCheck/Transfer/RsyncRawLP.sh (modified) added tmp path for rsyn Wed, 14 Mar 2012 09:03:19 GMT Daniela Dorner [13093] * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) added drs file to transfer Tue, 13 Mar 2012 14:49:22 GMT Daniela Dorner [13082] * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) added drs-files to rsync and check for existance of links Sun, 11 Mar 2012 17:10:36 GMT Daniela Dorner [13065] * trunk/DataCheck/Setup/setup.isdc.fact (modified) * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (modified) * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) added temp directory for rsync Fri, 09 Mar 2012 23:32:07 GMT Daniela Dorner [13058] * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (modified) Fri, 09 Mar 2012 22:27:53 GMT Daniela Dorner [13055] * trunk/DataCheck/Processing/CheckAuxFilesAvail.sh (modified) * trunk/DataCheck/Processing/CheckRawData.sh (modified) * trunk/DataCheck/Processing/CheckRawFilesAvail.sh (modified) * trunk/DataCheck/Processing/FillAuxData.sh (modified) * trunk/DataCheck/Setup/setup.lp.fact (modified) * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) * trunk/DataCheck/Transfer/RsyncAuxLP.sh (modified) * trunk/DataCheck/Transfer/RsyncRawLP.sh (modified) * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (modified) * trunk/DataCheck/Transfer/ZipRawData.sh (modified) adapted scripts to changes in directory structure Fri, 09 Mar 2012 22:18:35 GMT Daniela Dorner [13054] * trunk/DataCheck/Transfer/BackupAuxToWue.sh (modified) * trunk/DataCheck/Transfer/BackupRawToWue.sh (modified) reworked version of scripts doing backup to Wuerzburg Fri, 09 Mar 2012 22:15:38 GMT Daniela Dorner [13053] * trunk/DataCheck/Transfer/RsyncAuxToISDC.sh (added) * trunk/DataCheck/Transfer/RsyncRawToISDC.sh (added) added (scripts for transfer from LP to ISDC)