Changeset 9123 for trunk/MagicSoft


Ignore:
Timestamp:
08/15/08 15:56:15 (16 years ago)
Author:
Daniela Dorner
Message:
*** empty log message ***
Location:
trunk/MagicSoft/Mars
Files:
3 edited

Legend:

Unmodified
Added
Removed

  • trunk/MagicSoft/Mars/Changelog

    r9117 r9123  
    1818
    1919                                                 -*-*- END OF LINE -*-*-
     20
     21 2008/08/15 Daniela Dorner
     22
     23   * datacenter/macros/plotdb.C, datacenter/macros/plotoptical.C,
     24     datacenter/macros/plotstat.C, datacenter/macros/plotusage.C:
     25     - added call plot*(TString path) needed for the script
     26       dowebplots
     27     - added path to output file
     28
     29   * datacenter/macros/plotdb.C:
     30     - added path to call plotdb(dataset)
     31
     32   * datacenter/macros/plotoptical.C:
     33     - added path to call plotoptical(source)
     34
     35   * datacenter/macros/plotstat.C:
     36     - fixed bug in query
     37
     38   * datacenter/macros/plotusage.C:
     39     - adapted comment
     40     - removed not needed calls
     41
     42   * datacenter/scripts/dowebplots:
     43     - rewritten completely: script now handles not only creating
     44       web plots for ganymed, star and callisto, but also plotdb,
     45       plotusage, plotstat and plotoptical
     46     - implemented steering via DB for ganymed, star and callisto to
     47       make sure that all plots are produced
     48     - added rsync to this script
     49     - via command line options it is possible to choose which plots
     50       should be produced
     51     - redirect of the root error (used to output mean values) to
     52       txt files in the webdirectory
     53     - added writing of csv file (read by tabs.php)
     54
     55   * datacenter/scripts/run.condor:
     56     - added 'Arguments' (needed to submit dowebplots including
     57       command line opticon to condor)
     58
     59   * datacenter/scripts/scriptlauncher:
     60     - removed makedir of $listpath and $lockpath (done in setup
     61       already)
     62     - exchanged algorithm to loop over scripts to allow for scripts
     63       with command line option
     64
     65   * datacenter/scripts/sourcefile, datacenter/scripts/jobmanager:
     66     - moved definition of $column from sourcefile to jobmanager
     67       to allow for usage of the function getstatus by other scripts
     68       than the jobmanager
     69
     70   * datacenter/scripts/sourcefile:
     71     - removed definition of webpath (already defined in setup)
     72     - removed making of $listpath (not needed anymore)
     73     - added fail code for dowebplots (27)
     74     - additional if-clause in the getstatus function to allow
     75       for using it even if no column is defined
     76     - removed not needed 'numproc=0'
     77
     78   * resources/steps.rc:
     79     - added steps for webplots (fWebCalib, fWebStar, fWebGanymed,
     80       fWebPlotDBGanymed)
     81
     82
     83
     84 2008/08/14 Daniela Dorner
     85
     86   * datacenter/macros/plotusage.C:
     87     - added (macro to plot condor usage from DB)
     88
     89
     90
     91 2008/08/13 Daniela Dorner
     92
     93   * datacenter/scripts/setup:
     94     - removed listpath (not used anymore)
     95     - removed commented, not needed lines
     96
     97
    2098
    2199 2008/08/13 Stefan Ruegamer

  • trunk/MagicSoft/Mars/NEWS

    r9092 r9123  
    77   * Added informations about the DC currents (fMinCurrents, fMedCurrents
    88     and fMaxCurrents)
     9
     10 ;automatic analyis
     11
     12   * Rewritten producing plots in the web.
     13     New plots, namely the output of plotoptical.C (extinction curve from
     14     KVA), plotstat.C (status of the automatic analysis (percentage of
     15     processed, analysed data etc.)) and plotusage.C (statistics of the
     16     condor usage in the data center) has been are available now next to the
     17     plots of plotdb.C
     18     Mean values from the plots (output of the macro) are provided in a
     19     txt file now.
     20     The creation of the plots for callisto, star and ganymed in the web is
     21     now steered via database. Consequently the plots are faster and
     22     reliable up-to-date.
    923
    1024

  • trunk/MagicSoft/Mars/datacenter/scripts/dowebplots

    r9074 r9123  
    1818#
    1919#
    20 #   Author(s): Daniela Dorner  08/2005 <mailto:dorner@astro.uni-wuerzburg.de>
    21 #
    22 #   Copyright: MAGIC Software Development, 2000-2007
     20#   Author(s): Daniela Dorner  08/2008 <mailto:dorner@astro.uni-wuerzburg.de>
     21#
     22#   Copyright: MAGIC Software Development, 2000-2008
    2323#
    2424#
    … …  
    3131# files are produced.
    3232#
     33# The files from which plots are created can be devided into two categories:
     34# general: plots needed only once per telescope
     35# individual: plots done for each sequence / dataset
     36#
     37# For 'general' first the rootfiles are created with a macro from the
     38# database, then from this the ps via showplot, from this the pdf and the
     39# png files.
     40# For 'individual' running the macro is only needed for the plotdb for
     41# ganymed. The other steps are done accordingly.
     42#
     43# Via command line options the plots can be selected. For more information
     44# execute the script with the -h option.
     45#
    3346
    3447source `dirname $0`/sourcefile
    … …  
    3851set -C
    3952
     53
    4054scriptlog=$runlogpath/$program-$datetime.log
    4155date >> $scriptlog 2>&1
    4256
    43 while getopts p: opts
     57echo $@  >> $scriptlog 2>&1
     58
     59# definition of the plots to be created
     60generals=( plotdb optical condor status )
     61individuals=( calib signal star ganymed gplotdb )
     62# definition of the variables needed and varying for the plots
     63names=(     plotdb     optical    condor   status     calib     signal     star     ganymed           gplotdb )
     64pathnames=( plotdb     optical    condor   status  callisto   callisto     star     ganymed           ganymed )
     65macros=(    plotdb plotoptical plotusage plotstat        no         no       no          no                no )
     66filenames=( plotdb plotoptical plotusage plotstat     calib     signal     star     ganymed            plotdb )
     67columns=(       no          no        no       no fWebCalib fWebSignal fWebStar fWebGanymed fWebPlotDBGanymed )
     68digits=( 0 0 0 0 4 4 4 5 5 )
     69
     70# function to print the usage of the script
     71function usage()
     72{
     73   echo "Usage: $0 [options]"
     74   echo "options:"
     75   echo -n "  --general    name  "
     76   echo " where 'name' can be 'all' or one of the following: ${generals[@]} "
     77   echo -n "  --individual name  "
     78   echo " where 'name' can be 'all' or one of the following: ${individuals[@]} "
     79   echo -n "  -h                 "
     80   echo " print this usage "
     81   echo ""
     82   echo "Remark: if no commandline option is given the plots are produced for everything."
     83   echo ""
     84   exit
     85}
     86
     87# reading in the command line options
     88general=( )
     89individual=( )
     90while [ "$1" ]
     91do
     92   case $1 in
     93        --general) shift
     94                   if [ "$1" = "all" ]
     95                   then
     96                      general=( ${generals[@]} )
     97                   else
     98                      general=( $general $1 )
     99                   fi
     100                   ;;
     101     --individual) shift
     102                   if [ "$1" = "all" ]
     103                   then
     104                      individual=( ${individuals[@]} )
     105                   else
     106                      individual=( $individual $1 )
     107                   fi
     108                   ;;
     109              -h)  usage
     110                   exit
     111                   ;;
     112               *)  echo "unknown option $1 " >> $scriptlog 2>&1
     113                   usage  >> $scriptlog 2>&1
     114                   exit
     115                   ;;
     116   esac
     117   shift
     118done
     119
     120if [ ${#general[@]} -eq 0 ] || [ ${#individual[@]} -eq 0 ]
     121then
     122   all=( ${names[@]} )
     123else
     124   all=( ${general[@]} ${individual[@]} )
     125fi
     126
     127echo "For the following plots will be produced: "${all[@]} >> $scriptlog 2>&1
     128
     129
     130
     131# rsync subsystemdata, sequence and dataset files
     132echo "do rsync for subsystem files" >> $scriptlog 2>&1
     133#rsync -avn --delete $subsystempath/ $webpath/subsystemdata >> $scriptlog 2>&1
     134
     135echo "do rsync for sequence files" >> $scriptlog 2>&1
     136#rsync -avn --delete $sequencepath/ $webpath/sequences >> $scriptlog 2>&1
     137
     138echo "do rsync for dataset files" >> $scriptlog 2>&1
     139#rsync -avn --delete $datasetpath/ $webpath/datasets >> $scriptlog 2>&1
     140
     141
     142# using hour for lock => after 24 no new script is
     143#  started in case the previous is still running
     144hour=`date +%H`
     145lockfile=$lockpath/${program}-${hour}h.txt
     146checklock  >> $scriptlog 2>&1
     147
     148count=0
     149# loop over all scripts in $names
     150while [ 1 -gt 0 ]
    44151do
    45    case $opts in
    46       p)  type=$OPTARG
    47           echo "got programname: $type"  >> $scriptlog 2>&1
    48           ;;
    49       ?)  echo "usage: $(basename $0) -p programname"  >> $scriptlog 2>&1
    50           ;;
    51    esac
    52 done
    53 
    54 if [ "$type" = "" ]
    55 then
    56    echo "no program name given -> exit"  >> $scriptlog 2>&1
    57    echo "usage: $(basename $0) -p programname"   >> $scriptlog 2>&1
    58    date  >> $scriptlog 2>&1
    59    printprocesslog "INFO finished $0"
    60    exit
    61 fi
    62 
    63 # check if script is already running
    64 lockfile=$lockpath/lock-$program-$type.txt
    65 checklock  >> $scriptlog 2>&1
    66 
    67 #finding all rootfiles in the data directory, that were modified in the last 3 days
    68 #this are all statusdisplays
    69 rootfiles=`find $datapath/$type/ -maxdepth 10 -name '*.root' -mtime -3 | grep -v '_I_' | grep -v '_Y_' | grep -v 'summary'`
    70 
    71 #exit if no rootfiles are found
    72 if [ "$rootfiles" = "" ]
    73 then
    74    echo "nothing to do -> exit"  >> $scriptlog 2>&1
    75    finish >> $scriptlog 2>&1
    76 fi
    77 
    78 
    79 cd $mars
    80 
    81 #produce plots for each rootfile
    82 printprocesslog "INFO procduing plots for each $type rootfile"
    83 for rootfile in ${rootfiles[@]}
    84 do
    85    date >> $scriptlog 2>&1
    86    #get names of the psfile and the
    87    #pngfiles (1 per tab in the statusdiplay)
    88    psfile=`echo $rootfile | sed -e 's/.root$/.ps/g'`
    89    tabfile=`echo $rootfile | sed -e 's/.root$/-tab/g'`
    90    echo "rootfile: $rootfile" >> $scriptlog 2>&1
    91    echo "psfile:   $psfile"   >> $scriptlog 2>&1
    92    echo "tabfile:  $tabfile"  >> $scriptlog 2>&1
     152   check="ok"
     153   # get needed variables from the arrays
     154   name=${names[$count]}
     155   macro=${macros[$count]}
     156   pathname=${pathnames[$count]}
     157   filename=${filenames[$count]}
     158   column=${columns[$count]}
     159   digit=${digits[$count]}
    93160   
    94    #get date of root- and psfile
    95    daterootfile=`date +%Y%m%d -r $rootfile` >> $scriptlog 2>&1
    96    datepsfile=`date +%Y%m%d -r $psfile` >> $scriptlog 2>&1
    97    if [ "$datepsfile" = "" ]
     161   count=`expr $count + 1`
     162   if [ $count -gt ${#names[@]} ]
    98163   then
    99       echo "date of psfile is empty, i.e. the file $psfile doesn't exist" >> $scriptlog 2>&1
    100       echo " -> setting date to 0 and producing psfile..." >> $scriptlog 2>&1
    101       datepsfile=0
    102    fi
    103 
    104    echo "checking date..."  >> $scriptlog 2>&1
    105    echo "date of rootfile: $daterootfile" >> $scriptlog 2>&1
    106    echo "date of psfile:   $datepsfile"   >> $scriptlog 2>&1
    107    #if the psfile is newer than the rootfile
    108    #no plots have to be done -> continue
    109    if [ "$datepsfile" -gt "$daterootfile" ] >> $scriptlog 2>&1
    110    then
    111       echo "psfile is newer than rootfile -> continue " >> $scriptlog 2>&1
     164      count=0
    112165      continue
    113166   fi
    114 
    115    printprocesslog "INFO procduing plots for $rootfile"
    116    echo "producing psfile..." >> $scriptlog 2>&1
    117    ./showplot -b --save-as-ps=$psfile $rootfile >> $scriptlog 2>&1
    118 
    119    echo "creating temporary dir for pstoimg..." >> $scriptlog 2>&1
    120    temppath=`dirname $rootfile | sed -e 's/\/magic\/data//'`
    121    tempwebplotspath=/tmp/pstoimgtmp$temppath
    122    makedir $tempwebplotspath >> $scriptlog 2>&1
    123 
    124    echo "converting plots to png..." >> $scriptlog 2>&1
    125    pstoimg -antialias -aaliastext -interlaced -flip r270 -density 100 -tmp $tempwebplotspath -type png -multipage -out=$tabfile $psfile >> $scriptlog 2>&1
     167   # find out if these plots have to be created
     168   #  by looping over the complete array $all and comparing the entries to those of the array $names
     169   for (( a=0 ; a <= ${#all[@]} ; a++ ))
     170   do
     171      # exit the loop if the entries agree
     172      #  i.e. produce plots for this entry
     173      if [ "${all[$a]}" = "$name" ]
     174      then
     175         break
     176      fi
     177      # continue if the end of array is not yet reached
     178      if [ $a -lt ${#all[@]} ]
     179      then
     180         continue
     181      fi
     182      # exit both loops if no entry in the array is left
     183      #  i.e. there is nothing to do anymore
     184      if [ ${#all[@]} -eq 0 ]
     185      then
     186         break 2
     187      fi
     188      # continue with the next entry in the array $names
     189      #  in case the end of the array $all is reached without agreement
     190      continue 2
     191   done
     192
     193   # do things specific to 'general' and 'individual'
     194   if [ "$macro" = "no" ]
     195   then
     196      # 'individual'
     197      # remove name from list in case there is nothing left to do
     198      getstatus >> $scriptlog 2>&1
     199      if [ "$numproc" = "" ]
     200      then
     201         unset all[$a]
     202         all=( ${all[@]} )
     203         continue
     204      fi
     205      # get one sequence or dataset from the database for which plots have to be produced
     206      gettodo "1" >> $scriptlog 2>&1
     207      num=${primaries[0]}
     208      echo "executing $name for number "$num  >> $scriptlog 2>&1
     209      # lock the process
     210      lockfile=$lockpath/$program-$name-$num.txt
     211      checklock continue  >> $scriptlog 2>&1
     212      setstatus "start" >> $scriptlog 2>&1
     213      # set needed file and path names
     214      pathpart=$pathname/`printf %08d $num | cut -c 0-${digit}`/`printf %08d $num`
     215      inpath=$datapath/$pathpart
     216      outpath=$webpath/$pathpart
     217      tmppath=/tmp/pstoimg$pathpart
     218      rootfile=$inpath/$filename`printf %08d $num`.root
     219      psfile=$inpath/$filename`printf %08d $num`.ps
     220      csvfile=$inpath/$filename`printf %08d $num`.csv
     221      pdffile=$outpath/$filename`printf %08d $num`.pdf
     222      # rsync
     223      makedir $outpath >> $scriptlog 2>&1
     224      rsync -aLv --exclude=20[01][0-9]*_[YI]_*[.]root $inpath/ $outpath >> $scriptlog 2>&1
     225
     226      # for ganymed not only the ganymed*.root is processed but also a plotdb.root for the
     227      #  dataset. therefore the macro plotdb.C has to be run in this case
     228      if [ "$name" = "gplotdb" ]
     229      then
     230         # overwrite needed file and path names
     231         rootfile=$inpath/$filename.root
     232         psfile=$inpath/$filename.csv
     233         pdffile=$outpath/$filename.pdf
     234         datasetfile=$datasetpath/`printf %08d $num | cut -c 0-${digit}`/dataset`printf %08d $num`.txt
     235         
     236         echo "check1=root -q -b $macrospath/plotdb.C+\(\"$datasetfile\"\,\"$inpath/\"\) | tee -a $scriptlog | intgrep" >> $scriptlog 2>&1
     237         check1=`root -q -b $macrospath/plotdb.C+\("\"$datasetfile\""\,"\"$inpath/\""\) | tee -a $scriptlog | intgrep`
     238         case $check1 in
     239            1)   echo " check1=$check1 -> everything ok" >> $scriptlog 2>&1
     240                 printprocesslog "INFO rootfile $rootfile successfully produced"
     241                 ;;
     242            *)   echo " check1=$check1 -> ERROR -> couldn't create plots -> exit" >> $scriptlog 2>&1
     243                 printprocesslog "ERROR producing rootfile $rootfile failed"
     244                 com=$Fdowebplots
     245                 check=$check1
     246                 ;;
     247         esac
     248      fi
     249   else
     250      # 'general'
     251      echo "executing $name "  >> $scriptlog 2>&1
     252      # remove 'general' from list as they need to be executed only once
     253      unset all[$a]
     254      all=( ${all[@]} )
     255      # lock process
     256      lockfile=$lockpath/$program-$name.txt
     257      checklock continue >> $scriptlog 2>&1
     258      # set needed file and path names
     259      outpath=$webpath/$pathname
     260      tmppath=/tmp/$pathname
     261      rootfile=$outpath/$filename.root
     262      psfile=$outpath/$filename.csv
     263      pdffile=$outpath/$filename.pdf
     264      txtfile=$outpath/$filename.txt
     265     
     266      echo "check2=root -q -b $macrospath/$macro.C+\(\"$outpath/\"\) 2>> $txtfile | tee -a $scriptlog | intgrep" >> $scriptlog 2>&1
     267      check2=`root -q -b $macrospath/$macro.C+\("\"$outpath/\""\) 2>> $txtfile | tee -a $scriptlog | intgrep`
     268      case $check2 in
     269         1)   echo " check2=$check2 -> everything ok" >> $scriptlog 2>&1
     270              printprocesslog "INFO rootfile $rootfile successfully produced"
     271              ;;
     272         *)   echo " check2=$check2 -> ERROR -> couldn't produce root file $rootfile  -> exit" >> $scriptlog 2>&1
     273              printprocesslog "ERROR producing rootfile $rootfile failed"
     274              com=$Fdowebplots
     275              check=$check2
     276              ;;
     277      esac
     278   fi
    126279   
    127    echo "removing temporary dir..." >> $scriptlog 2>&1
    128    rmdir -pv $tempwebplotspath >> $scriptlog 2>&1
     280   # in case no error occured, the plots are produced from the rootfile
     281   if [ "$check" = "ok" ]
     282   then
     283      printprocesslog "INFO procduing plots (psfile and png) for $rootfile"
     284      if ! ls $rootfile >/dev/null 2>&1
     285      then
     286         echo "rootfile $rootfile does not exist" >> $scriptlog 2>&1
     287         printprocesslog "ERROR rootfile $rootfile does not exist"
     288         com=$Fdowebplots
     289         check=3
     290      else
     291         echo "producing psfile and csvfile..." >> $scriptlog 2>&1
     292         if ! ./showplot -b --save-as-ps=$psfile --save-as-csv=$csvfile $rootfile >> $scriptlog 2>&1
     293         then
     294            com=$Fdowebplots
     295            check=4
     296         else
     297            echo "creating temporary directory for pstoimg..." >> $scriptlog 2>&1
     298            makedir $tmppath >> $scriptlog 2>&1
     299           
     300            echo "converting plots to png..." >> $scriptlog 2>&1
     301            if ! pstoimg -antialias -aaliastext -interlaced -flip r270 -density 100 -tmp $tmppath -out=$outpath/$name -type png -multipage $psfile >> $scriptlog 2>&1
     302            then
     303               printprocesslog "WARN could not create png files for $psfile "
     304               com=$Fdowebplots
     305               check=5
     306            fi
     307           
     308            echo "removing temporary directory..." >> $scriptlog 2>&1
     309            rmdir -pv $tmppath >> $scriptlog 2>&1
     310           
     311            echo "converting $psfile to pdf..." >> $scriptlog 2>&1
     312         fi
     313         
     314         if ps2pdf $psfile $pdffile >> $scriptlog 2>&1
     315         then
     316            rm -v $psfile  >> $scriptlog 2>&1
     317         else
     318            printprocesslog "WARN could not convert $psfile to $pdffile"
     319            com=$Fdowebplots
     320            check=6
     321         fi
     322      fi
     323   fi
    129324   
    130    echo "converting $psfile to pdf..." >> $scriptlog 2>&1
    131    if ps2pdf $psfile >> $scriptlog 2>&1
    132    then
    133       rm -v $psfile  >> $scriptlog 2>&1
    134    else
    135       printprocesslog "WARN could not convert $psfile to pdf"
    136    fi
     325   setstatus "stop" >> $scriptlog 2>&1
     326   
     327   rm -v $lockfile >> $scriptlog 2>&1
     328
     329   sleep 2
    137330done
    138331
     332# rsync for complete directory delete removed sequences
     333echo "do rsync for callisto, star and ganymed files" >> $scriptlog 2>&1
     334rsync -aLvn --delete --exclude=*/*/20[01][0-9]*_[YI]_*[.]root --exclude=*/*.ps $datapath/{callisto,star,ganymed}/ $webpath/callisto >> $scriptlog 2>&1
     335
     336lockfile=$lockpath/${program}-${hour}h.txt
    139337finish >> $scriptlog 2>&1
    140338
     339
Note: See TracChangeset for help on using the changeset viewer.