Index: trunk/MagicSoft/TDAS-Extractor/Performance.tex =================================================================== --- trunk/MagicSoft/TDAS-Extractor/Performance.tex (revision 5631) +++ trunk/MagicSoft/TDAS-Extractor/Performance.tex (revision 5632) @@ -1,4 +1,3 @@ \section{Performance} -\ldots {\textit This section will be written after the previous one} \subsection{Calibration} @@ -44,7 +43,9 @@ \begin{enumerate} -\item Stability tests: These include numbers of excluded pixels by the calibration software, -numbers of reconstructed photo-electrons and counts of the numbers of outliers from the expected Gaussian -distributions of reconstructed charges. +\item Un-calibrated pixels and events: These tests measure the percentage of failures of the extractor +resulting either in a pixel declared as un-calibrated or in an event which produces a signal ouside +of the expected Gaussian distribution. +\item Number of photo-electrons: These tests measure the reconstructed numbers of photo-electrons, their +spread over the camera and the ratio of the obtained mean value for outer and inner pixels. \item Linearity tests: These test the linearity of the extractor with respect to pulses of different intensity and colour. @@ -54,9 +55,11 @@ We used data taken on the 7$^{th}$ of June, 2004 with different pulser LED combinations, each taken with -16384 events. The corresponding run numbers range from nr. 31741 to 31772. -\par -Although, we looked at and tested all colour and extractor combinations resulting from these data, -we will refrain ourselves to show here only exemplary behaviour and results of extractors. All taken -``control'' plots including those which are not displayed here, can be retrieved from the following +16384 events. The corresponding run numbers range from nr. 31741 to 31772. This data was taken before the +latest camera repair access which replaced about 2\% of the pixels known to be mal-functionning at that time. +Thus, there is a lower limit to the number of un-calibrated pixels of about 1.5--2\%. +\par +Although, we had looked at and tested all colour and extractor combinations resulting from these data, +we refrain ourselves to show here only exemplary behaviour and results of extractors. +All plots, including those which are not displayed in this TDAS, can be retrieved from the following locations: @@ -66,5 +69,51 @@ \end{verbatim} -\subsubsection{Stability tests} +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Un-calibrated pixels and events} + +The MAGIC calibration software incorporates a series of checks to sort out mal-functionning pixels. +Except for the software bug searching criteria, the following exclusion reasons can apply: + +\begin{enumerate} +\item The reconstructed mean signal is less than 2.5 times the extractor resolution $R$ from zero. +(2.5 Pedestal RMS in the case of the simple fixed window extractors). This criterium cuts out +dead pixels. +\item The reconstructed mean signal error is smaller than its value. This criterium cuts out +signal distributions which fluctuate so much that their RMS is bigger than its mean value. This +criterium cuts out ``ringing'' pixels or mal-functionning extractors. +\item The reconstructed mean number of photo-electrons lies 4.5 sigma outside +the distribution of photo-electrons obtained with the inner or outer pixels in the camera. +\item All reconstructed negative mean signal, signal sigma's and mean numbers of photo-electrons +smaller than one. +\end{enumerate} + +Moreover, the number of events are counted which have been reconstructed outside a 5 sigma region +from the mean signal. These events are called ``outliers''. Figure~\ref{fig:outlier} shows a typical +outlier obtained with the digital filter. + +\begin{figure}[htp] +\centering +\includegraphics[width=0.95\linewidth]{Outlier.eps} +\caption{Example of an event classified as ``un-calibrated''. The histogram has been obtained +using the digital filter (extractor \#32) applied to a high-intensity blue pulse (run 31772). +The event marked as ``outlier'' clearly has been mis-reconstructed. It lies outside the 5 sigma +region from the fitted mean.} +\label{fig:outlier} +\end{figure} + +The following figures~\ref{fig:unsuited:5ledsuv},~\ref{fig:unsuited:1leduv},~\ref{fig:unsuited:2ledsgreen} +and~\ref{fig:unsuited:23ledsblue} show the resulting numbers of un-calibrated pixels and events for +different colours and intensities. + +\par + +\begin{figure}[htp] +\centering +\includegraphics[height=0.95\textheight]{UnsuitVsExtractor-5LedsUV-Colour-13.eps} +\caption{Uncalibrated pixels and pixels outside of the Gaussian distribution for a typical calibration +pulse of UV-light which does not saturate the high-gain readout.} +\label{fig:unsuited:5ledsuv} +\end{figure} \begin{figure}[htp] @@ -73,7 +122,172 @@ \caption{Uncalibrated pixels and pixels outside of the Gaussian distribution for a very low intensity pulse.} -\end{figure} +\label{fig:unsuited:1leduv} +\end{figure} + +\begin{figure}[htp] +\centering +\includegraphics[height=0.95\textheight]{UnsuitVsExtractor-2LedsGreen-Colour-02.eps} +\caption{Uncalibrated pixels and pixels outside of the Gaussian distribution for a typical green pulse.} +\label{fig:unsuited:2ledsgreen} +\end{figure} + +One can see that in general, big extraction windows raise the +number of un-calibrated pixels and are thus less stable. Especially for the very low-intensity +$1LedUV$-pulse, the big extraction windows summing 8 or more slices, cannot calibrate more than 50\% +of the inner pixels (fig.~\ref{fig:unsuited:1leduv}). This is an expected behavior since big windows +add up more noise which in turn makes the for the small signal more difficult. +\par +In general, one can also say that all ``sliding window''-algorithms (extractors \#17-32) discard +less pixels than the ``fixed window''-ones (extractors \#1--16). The digital filter with +the correct weights (extractor \#32) discards the least number of pixels, but is also robust against +slight modifications of its weights (extractors \#28--31). Also the ``spline'' algorithms on small +windows (extractors \#23--25) discard less pixels than the previous extractors, although slightly more +then the digital filter. +\par +Concerning the numbers of outliers, one can conclude that in general, the numbers are very low never exceeding +0.25\%. There seems to be the opposite trend of larger windows producing less +outliers. However, one has to take into account that already more ``unsuited'' pixels have +been excluded thus cleaning up the sample somewhat. It seems that the ``digital filter'' and a +medium-sized ``spline'' (extractors \#25--26) yield the best result except for the outer pixels +in fig~\ref{fig:unsuited:5ledsuv} where the digital filter produces a worse result than the rest +of the extractors. +\par +In conclusion, one can say that this test excludes all extractors with too big window sizes because +they are not able to extract small signals produced by about 4 photo-electrons. The excluded extractors +are: +\begin{itemize} +\item: MExtractFixedWindow Nr. 3--5 +\item: MExtractFixedWindowSpline Nr. 6--11 +\item: MExtractFixedWindowPeakSearch Nr. 14--16 +\item: MExtractTimeAndChargeSlidingWindow Nr. 21--22 +\item: MExtractTimeAndChargeSpline Nr. 27 +\end{itemize} + +The best extractors after this test are: +\begin{itemize} +\item: MExtractFixedWindow Nr. 1--2 +\item: MExtractFixedWindowPeakSearch Nr. 13 +\item: MExtractTimeAndChargeSlidingWindow Nr. 17--19 +\item: MExtractTimeAndChargeSpline Nr. 24--25 +\item: MExtractTimeAndChargeDigitalFilter Nr. 28--32 +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Number of photo-electrons} + +Assuming that the readout chain is clean and adds only negligible noise with respect to the one +introduced by the photo-multiplier itself, one can make the assumption that variance of the +true (non-extracted) signal $ST$ is the amplified Poisson variance on the number of photo-electrons, +multiplied with the excess noise of the photo-multiplier, characterized by the excess-noise factor $F$. + +\begin{equation} +Var(ST) = F^2 \cdot Var(N_{phe}) \cdot \frac{^2}{^2} +\label{eq:excessnoise} +\end{equation} + +After introducing the effect of the night-sky background (eq.~\ref{eq:rmssubtraction}) +in formula~\ref{eq:excessnoise} and assuming that the number of photo-electrons per event follows a +Poisson distribution, one can +get an expression to retrieve the mean number of photo-electrons impinging on the pixel from the +mean extracted signal $$, its variance $Var(SE)$ and the RMS of the extracted signal obtained from +pure pedestal runs $R$ (see section~\ref{sec:determiner}): + +\begin{equation} + \approx F^2 \cdot \frac{Var(SE) - R^2}{^2} +\label{eq:pheffactor} +\end{equation} + +Equation~\ref{eq:pheffactor} must not depend on the extractor! Effectively, we will use it to test the +quality of our extractors by requiring that a valid extractor yields the same number of photo-electrons +for all pixels of a same type and does not deviate from the number obtained with other extractors. +As the camera is flat-fielded, but the number of photo-electrons impinging on an inner and an outer pixel is +different, we also use the ratio of the mean numbers of photo-electrons from the outer pixels to the one +obtained from the inner pixels as a test variable. In the ideal case, it should always yield its central +value of about 2.4--2.8. +\par +In our case, there is an additional complication due to the fact that the green and blue coloured pulses +show secondary pulses which destroy the Poisson behaviour of the number of photo-electrons. We will thus +have to split our sample of extractors into those being affected by the secondary pulses and those without +showing any effect. +\par +Figures~\ref{fig:phe:5ledsuv},~\ref{fig:phe:1leduv},~\ref{fig:phe:23ledsblue}~and~\ref{fig:phe:2ledsgreen} show +some of the obtained results. Although one can see an amazing stability for the standard 5Leds UV pulse, there +is a considerable difference for all shown non-standard pulses. Especially the pulses from green and blue LEDs +show a clear dependency on the extraction window of the number of photo-electrons. Only the largest +extraction windows seem to catch the entire range of (jittering) secondary pulses and get also the ratio +of outer vs. inner pixels right. + +\begin{figure}[htp] +\centering +\includegraphics[height=0.92\textheight]{PheVsExtractor-5LedsUV-Colour-13.eps} +\caption{Number of photo-electrons from a typical, not saturating calibration pulse of colour UV, +reconstructed with each of the tested signal extractors. +The first plots shows the number of photo-electrons obtained for the inner pixels, the second one +for the outer pixels and the third shows the ratio of the mean number of photo-electrons for the +outer pixels divided by the mean number of photo-electrons for the inner pixels. Points +denote the mean of all not-excluded pixels, the error bars their RMS.} +\label{fig:phe:5ledsuv} +\end{figure} + +\begin{figure}[htp] +\centering +\includegraphics[height=0.92\textheight]{PheVsExtractor-1LedUV-Colour-04.eps} +\caption{Number of photo-electrons from a typical, very low-intensity calibration pulse of colour UV, +reconstructed with each of the tested signal extractors. +The first plots shows the number of photo-electrons obtained for the inner pixels, the second one +for the outer pixels and the third shows the ratio of the mean number of photo-electrons for the +outer pixels divided by the mean number of photo-electrons for the inner pixels. Points +denote the mean of all not-excluded pixels, the error bars their RMS.} +\label{fig:phe:1leduv} +\end{figure} + +\begin{figure}[htp] +\centering +\includegraphics[height=0.92\textheight]{PheVsExtractor-2LedsGreen-Colour-02.eps} +\caption{Number of photo-electrons from a typical, not saturating calibration pulse of colour green, +reconstructed with each of the tested signal extractors. +The first plots shows the number of photo-electrons obtained for the inner pixels, the second one +for the outer pixels and the third shows the ratio of the mean number of photo-electrons for the +outer pixels divided by the mean number of photo-electrons for the inner pixels. Points +denote the mean of all not-excluded pixels, the error bars their RMS.} +\label{fig:phe:2ledsgreen} +\end{figure} + + +One can see that all extractor using a large window belong to the class of extractors being affected +by the secondary pulses. The only exception to this rule is the digital filter which - despite of its +6 slices extraction window - seems to filter out all the secondary pulses. +\par +Moreover, one can see in fig.~\ref{fig:phe:1leduv} that all peak searching extractors show the influence of +the bias at low numbers of photo-electrons. +\par +The extractor MExtractFixedWindowPeakSearch at low extraction windows apparently yields chronically low +numbers of photo-electrons. This is due to the fact that the decision to fix the extraction window is +made sometimes by an inner pixel and sometimes by an outer one since the camera is flat-fielded and the +pixel carrying the largest non-saturated peak-search window is more or found by a random signal +fluctuation. However, inner and outer pixels have a systematic offset of about 0.5 to 1 FADC slices. +Thus, the extraction fluctuates artificially for one given channel which results in a systematically +large variance and thus in a systematically low reconstructed number of photo-electrons. This test thus +excludes the extractors \#11--13. +\par +Moreover, one can see that the extractors applying a small fixed window do not get the ratio of +photo-electrons from outer to inner pixels correctly for the green and blue pulses. +\par +The extractor MExtractTimeAndChargeDigitalFilter seems to be veryu stable against modifications in the +exact form of the weights since all applied weights yield about the same number of photo-electrons and the +same ratio of outer vs. inner pixels. The last is also true for the extractor MExtractTimeAndChargeSpline, +although the number of photo-electrons depends on the extraction window for green and blue pulses, +(as with the other extractors). \subsubsection{Linearity tests} + +In this section, we test the lineary of the extractors. As the photo-multiplier is a linear device over a +wide dynamic range, the number of photo-electrons per charge has to remain constant over the tested +linearity region. We will show here only examples of extractors which were not already excluded in the +previous section. +\par +A first test concerns the stability of the conversion factor photo-electrons per FADC counts over the +tested intensity region.