source: trunk/MagicSoft/TDAS-Extractor/Pedestal.tex@ 6388

\section{Pedestal Extraction \label{sec:pedestals}}

\subsection{Pedestal RMS}

The background $BG$ (Pedestal) 
can be completely described by the noise-autocorrelation matrix $\boldsymbol{B}$ 
(eq.~\ref{eq:autocorr}), 
where the diagonal elements give what is usually denoted as the ``Pedestal RMS''. 
\par

By definition, the $\boldsymbol{B}$ and thus the ``pedestal RMS'' 
is independent from the signal extractor.

\subsection{Bias and Mean-squared Error}

Consider a large number of same signals $S$. By applying a signal extractor
we obtain a distribution of estimated signals $\widehat{S}$ (for fixed $S$ and
fixed background fluctuations $BG$). The distribution of the quantity

\begin{equation}
X = \widehat{S}-S 
\end{equation}

has the mean $B$ and the Variance $MSE$ defined as:

\begin{eqnarray}
   B   \ \ \ \  = \ \ \ \ \ \ <X> \ \ \ \ \  &=& \ \ <\widehat{S}> \ -\ S\\
   R   \ \ \ \  = \ <(X-B)^2> &=& \ Var[\widehat{S}]\\
   MSE \      = \ \ \ \ \ <X^2> \ \ \ \  &=& \ Var[\widehat{S}] +\ B^2
\end{eqnarray}

The parameter $B$ is also called the {\textit{\bf BIAS}} of the estimator and $MSE$ 
the {\textit{\bf MEAN-SQUARED ERROR}} which combines the variance of $\widehat{S}$ and 
the bias. Both depend generally on the size of $S$ and the background fluctuations $BG$, 
thus: $B = B(S,BG)$ and $MSE = MSE(S,BG)$.

\par
Usually, one measures easily the parameter $R$, but needs the $MSE$ for statistical analysis (e.g. 
in the image cleaning). 
However, only in case of a vanishing bias $B$, the two numbers are equal. Otherwise, 
the bias $B$ has to be known beforehand. Note that every sliding window extractor has a 
bias, especially at low or vanishing signals $S$.

\subsection{Pedestal Fluctuations as Contribution to the Signal Fluctuations}

A photo-multiplier signal yields, to a very good approximation, the 
following relation:

\begin{equation}
\frac{Var[Q]}{<Q>^2} = \frac{1}{<n_{phe}>} * F^2
\end{equation}

Here, $Q$ is the signal fluctuation due to the number of signal photo-electrons
(equiv. to the signal $S$), and $Var[Q]$ the fluctuations of the true signal $Q$ 
due to the Poisson fluctuations of the number of photo-electrons. Because of: 

\begin{eqnarray}
\widehat{Q} &=& Q + X \\
Var(\widehat{Q}) &=& Var(Q) + Var(X) \\
Var(Q) &=& Var(\widehat{Q}) - Var(X)
\end{eqnarray}

$Var[Q]$ can be obtained from: 

\begin{eqnarray}
Var(Q) &\approx& Var(\widehat{Q}) - Var(\widehat{Q}=0)
\label{eq:rmssubtraction}
\end{eqnarray}

In the last line of eq.~\ref{eq:rmssubtraction}, it is assumed that $R$ does not dependent 
on the signal height\footnote{%
A way to check whether the right RMS has been subtracted is to make the
``Razmick''-plot

\begin{equation}
    \frac{Var[\widehat{Q}]}{<\widehat{Q}>^2} \quad \textit{vs.} \quad \frac{1}{<\widehat{Q}>}
\end{equation}

This should give a straight line passing through the origin. The slope of
the line is equal to

\begin{equation}
    c * F^2
\end{equation}

where $c$ is the photon/ADC conversion factor  $<Q>/<m_{pe}>$.}
(as is the case 
for the digital filter, eq.~\ref{eq:of_noise}). One can then retrieve $R$ 
by applying the signal extractor to a {\textit{\bf fixed window}} of pedestal events, where the 
bias vanishes and measure $Var[\widehat{Q}=0]$.

\subsection{Methods to Retrieve Bias and Mean-Squared Error}

In general, the extracted signal variance $R$ is different from the pedestal RMS. 
It cannot be obtained by applying the signal extractor to pedestal events, because of the 
(unknown) bias. 
\par
In the case of the digital filter, $R$ is expected to be independent from the 
signal amplitude $S$ and depends only on the background $BG$ (eq.~\ref{eq:of_noise}).
It can then be obtained from the calculation of the variance $Var[\widehat{Q}]$ 
by applying the extractor to a fixed window of pure background events (``pedestal events'') 
and get rid of the bias in that way. 
\par

In order to calculate bias and Mean-squared error, we proceeded in the following ways:
\begin{enumerate}
\item Determine $R$ by applying the signal extractor to a fixed window
    of pedestal events. The background fluctuations can be simulated with different 
    levels of night sky background and the continuous light source, but no signal size 
    dependency can be retrieved with this method. 
\item Determine $B$ and $MSE$ from MC events with and without added noise. 
    Assuming that $MSE$ and $B$ are negligible for the events without noise, one can 
    get a dependency of both values from the size of the signal. 
\item Determine $MSE$ from the fitted error of $\widehat{S}$, which is possible for the 
    fit and the digital filter (eq.~\ref{eq:of_noise}). 
    In prinicple, all dependencies can be retrieved with this method.
\end{enumerate}


\begin{figure}[htp]
\centering
\includegraphics[width=0.3\linewidth]{MExtractTimeAndChargeSpline_Amplitude_Amplitude_Range_01_09_01_10_Run_38993_RelMean.eps}
\vspace{\floatsep}
\includegraphics[width=0.3\linewidth]{MExtractTimeAndChargeSpline_Amplitude_Amplitude_Range_01_09_01_10_Run_38995_RelMean.eps}
\vspace{\floatsep}
\includegraphics[width=0.3\linewidth]{MExtractTimeAndChargeSpline_Amplitude_Amplitude_Range_01_09_01_10_Run_38996_RelMean.eps}
\caption{MExtractTimeAndChargeSpline with amplitude extraction: 
Difference in mean pedestal (per FADC slice) between extraction algorithm
applied on a fixed window of 1 FADC slice (``extractor random'') and a simple addition of 
2 FADC slices (``fundamental''). On the left, a run with closed camera has been taken, in the center
 an opened camera observing an extra-galactic star field and on the right, an open camera being 
illuminated by the continuous light of the calibration (level: 100). Every entry corresponds to one 
pixel.}
\label{fig:amp:relmean}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics[width=0.3\linewidth]{MExtractTimeAndChargeSpline_Rise-and-Fall-Time_0.5_1.5_Range_01_10_02_12_Run_38993_RelMean.eps}
\vspace{\floatsep}
\includegraphics[width=0.3\linewidth]{MExtractTimeAndChargeSpline_Rise-and-Fall-Time_0.5_1.5_Range_01_10_02_12_Run_38995_RelMean.eps}
\vspace{\floatsep}
\includegraphics[width=0.3\linewidth]{MExtractTimeAndChargeSpline_Rise-and-Fall-Time_0.5_1.5_Range_01_10_02_12_Run_38996_RelMean.eps}
\caption{MExtractTimeAndChargeSpline with integral over 2 slices: 
Difference in mean pedestal (per FADC slice) between extraction algorithm
applied on a fixed window of 2 FADC slices (``extractor random'') and a simple addition of 
2 FADC slices (``fundamental''). On the left, a run with closed camera has been taken, in the center
 an opened camera observing an extra-galactic star field and on the right, an open camera being 
illuminated by the continuous light of the calibration (level: 100). Every entry corresponds to one 
pixel.}
\label{fig:int:relmean}
\end{figure}

\begin{figure}[htp]
\centering
\vspace{\floatsep}
\includegraphics[width=0.3\linewidth]{MExtractTimeAndChargeDigitalFilter_Weights_cosmics_weights.dat_Range_01_14_02_14_Run_38993_RelMean.eps}
\vspace{\floatsep}
\includegraphics[width=0.3\linewidth]{MExtractTimeAndChargeDigitalFilter_Weights_cosmics_weights.dat_Range_01_14_02_14_Run_38995_RelMean.eps}
\vspace{\floatsep}
\includegraphics[width=0.3\linewidth]{MExtractTimeAndChargeDigitalFilter_Weights_cosmics_weights.dat_Range_01_14_02_14_Run_38996_RelMean.eps}
\caption{MExtractTimeAndChargeDigitalFilter: 
Difference in mean pedestal (per FADC slice) between extraction algorithm
applied on a fixed window of 6 FADC slices and time-randomized weights (``extractor random'') 
and a simple addition of 
6 FADC slices (``fundamental''). On the left, a run with closed camera has been taken, in the center
 an opened camera observing an extra-galactic star field and on the right, an open camera being 
illuminated by the continuous light of the calibration (level: 100). Every entry corresponds to one 
pixel.}
\label{fig:df:relmean}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsubsection{ \label{sec:determiner} Application of the Signal Extractor to a Fixed Window
of Pedestal Events}

By applying the signal extractor to a fixed window of pedestal events, we 
determine the parameter $R$ for the case of no signal ($Q = 0$). In the case of 
extractors using a fixed window (extractors nr. \#1 to \#22 
in section~\ref{sec:algorithms}), the results are the same by construction 
as calculating the pedestal RMS.
\par
In MARS, this functionality is implemented with a function-call to: \\ 

{\textit{\bf MJPedestal::SetExtractionWithExtractorRndm()}} and/or \\
{\textit{\bf MExtractPedestal::SetRandomCalculation()}}\\

Besides fixing the global extraction window, additionally the following steps are undertaken 
in order to assure that the bias vanishes: 

\begin{description}
\item[\textit{MExtractTimeAndChargeSpline}:\xspace] The spline 
maximum position -- which determines the exact extraction window -- is placed arbitrarily 
at a random place within the digitizing binning resolution of one central FADC slice. 
\item[\textit{MExtractTimeAndChargeDigitalFilter}:\xspace] The second step timing 
offset $\tau$ (eq.~\ref{eq:offsettau}) gets randomized for each event. 
\end{description}

\par

The following figures~\ref{fig:amp:relmean} through~\ref{fig:df:relrms} show results 
obtained with the second method for three background intensities: 

\begin{enumerate}
\item Closed camera and no (Poissonian) fluctuation due to photons from the night sky background
\item The camera pointing to an extra-galactic region with stars in the field of view
\item The camera illuminated by a continuous light source of intensity 100. 
\end{enumerate}

Figures~\ref{fig:amp:relmean} through~\ref{fig:df:relmean} 
show the calculated biases obtained with this method for all pixels in the camera 
and for the different levels of (night-sky) background. 
One can see that the bias vanishes to an accuracy of better than 1\% 
for the extractors which are used in this TDAS. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%1

\begin{figure}[htp]
\centering
\includegraphics[width=0.47\linewidth]{MExtractTimeAndChargeSpline_Amplitude_Amplitude_Range_01_09_01_10_Run_38993_RMSDiff.eps}
\vspace{\floatsep}
\includegraphics[width=0.47\linewidth]{MExtractTimeAndChargeSpline_Amplitude_Amplitude_Range_01_09_01_10_Run_38995_RMSDiff.eps}
\vspace{\floatsep}
\includegraphics[width=0.47\linewidth]{MExtractTimeAndChargeSpline_Amplitude_Amplitude_Range_01_09_01_10_Run_38996_RMSDiff.eps}
\caption{MExtractTimeAndChargeSpline with amplitude:  
Difference in RMS (per FADC slice) between extraction algorithm
applied on a fixed window and the corresponding pedestal RMS. 
Closed camera (left), open camera observing extra-galactic star field (right) and 
camera being illuminated by the continuous light (bottom). 
Every entry corresponds to one pixel.}
\label{fig:amp:relrms}
\end{figure}


\begin{figure}[htp]
\centering
\includegraphics[width=0.47\linewidth]{MExtractTimeAndChargeSpline_Rise-and-Fall-Time_0.5_1.5_Range_01_10_02_12_Run_38993_RMSDiff.eps}
\vspace{\floatsep}
\includegraphics[width=0.47\linewidth]{MExtractTimeAndChargeSpline_Rise-and-Fall-Time_0.5_1.5_Range_01_10_02_12_Run_38995_RMSDiff.eps}
\vspace{\floatsep}
\includegraphics[width=0.47\linewidth]{MExtractTimeAndChargeSpline_Rise-and-Fall-Time_0.5_1.5_Range_01_10_02_12_Run_38996_RMSDiff.eps}
\caption{MExtractTimeAndChargeSpline with integral over 2 slices:  
Difference in RMS (per FADC slice) between extraction algorithm
applied on a fixed window and the corresponding pedestal RMS. 
Closed camera (left), open camera observing extra-galactic star field (right) and 
camera being illuminated by the continuous light (bottom). 
Every entry corresponds to one 
pixel.}
\label{fig:amp:relrms}
\end{figure}


\begin{figure}[htp]
\centering
\includegraphics[width=0.47\linewidth]{MExtractTimeAndChargeDigitalFilter_Weights_cosmics_weights.dat_Range_01_14_02_14_Run_38993_RMSDiff.eps}
\vspace{\floatsep}
\includegraphics[width=0.47\linewidth]{MExtractTimeAndChargeDigitalFilter_Weights_cosmics_weights.dat_Range_01_14_02_14_Run_38995_RMSDiff.eps}
\vspace{\floatsep}
\includegraphics[width=0.47\linewidth]{MExtractTimeAndChargeDigitalFilter_Weights_cosmics_weights.dat_Range_01_14_02_14_Run_38996_RMSDiff.eps}
\caption{MExtractTimeAndChargeDigitalFilter:  
Difference in RMS (per FADC slice) between extraction algorithm
applied on a fixed window and the corresponding pedestal RMS. 
Closed camera (left), open camera observing extra-galactic star field (right) and 
camera being illuminated by the continuous light (bottom). 
Every entry corresponds to one pixel.}
\label{fig:df:relrms}
\end{figure}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Figures~\ref{fig:amp:relrms} through~\ref{fig:amp:relrms} show the 
differences in $R$ between the calculated pedestal RMS and 
the one obtained by applying the extractor, converted to equivalent photo-electrons. One entry 
corresponds to one pixel of the camera.
The distributions have a negative mean in the case of the digital filter showing the 
``filter'' capacity of that algorithm. It ``filters out'' between 0.12 photo-electrons night sky 
background for the extra-galactic star-field until 0.2 photo-electrons for the continuous light.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\subsubsection{ \label{sec:determiner} Application of the Signal Extractor to a Sliding Window
of Pedestal Events}

By applying the signal extractor to a global extraction window of pedestal events, allowing 
it to ``slide'' and maximize the encountered signal, we 
determine the bias $B$ and the mean-squared error $MSE$ for the case of no signal ($S=0$). 
\par
In MARS, this functionality is implemented with a function-call to: \\ 

{\textit{\bf MJPedestal::SetExtractionWithExtractor()}} \\

\par

Figures~\ref{fig:amp:distped} through~\ref{fig:df:distped} show the 
extracted pedestal distributions for the digital filter with cosmics weights (extractor~\#28) and the 
spline amplitude (extractor~\#27), respectively for one examplary channel (corresponding to pixel 200). 
One can see the (asymmetric) Poisson behaviour of the 
night sky background photons for the distributions with open camera and the cutoff at the lower egde 
for the distribution with high-intensity continuous light due to a limited pedestal offset and the cutoff 
to negative fluctuations.
\par

\begin{figure}[htp]
\centering
\includegraphics[height=0.43\textheight]{PedestalSpectrum-18-Run38993.eps}
\vspace{\floatsep}
\includegraphics[height=0.43\textheight]{PedestalSpectrum-18-Run38995.eps}
\caption{MExtractTimeAndChargeSlidingWindow with extraction window of 4 FADC slices: 
Distribution of extracted "pedestals"  from pedestal run with 
closed camera (top) and open camera observing an extra-galactic star field (bottom) for one channel 
(pixel 100). The result obtained from a simple addition of 4 FADC 
slice contents (``fundamental'') is displayed as red histogram, the one obtained from the application of
the algorithm on 
a fixed window of 4 FADC slices as blue histogram (``extractor random'') and the one obtained from the
full algorithm allowed to slide within a global window of 12 slices. The obtained histogram means and 
RMSs have been converted to equiv. photo-electrons.}
\label{fig:sw:distped}
\end{figure}


\begin{figure}[htp]
\centering
\includegraphics[height=0.43\textheight]{PedestalSpectrum-23-Run38993.eps}
\vspace{\floatsep}
\includegraphics[height=0.43\textheight]{PedestalSpectrum-23-Run38995.eps}
\caption{MExtractTimeAndChargeSpline with amplitude extraction: 
Spectrum of extracted "pedestals"  from pedestal run with 
closed camera lids (top) and open lids observing an extra-galactic star field (bottom) for one channel 
(pixel 100). The result obtained from a simple addition of 2 FADC 
slice contents (``fundamental'') is displayed as red histogram, the one obtained from the application 
of the algorithm on a fixed window of 1 FADC slice as blue histogram  (``extractor random'')
and the one obtained from the
full algorithm allowed to slide within a global window of 12 slices. The obtained histogram means and 
RMSs have been converted to equiv. photo-electrons.}
\label{fig:amp:distped}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics[height=0.43\textheight]{PedestalSpectrum-25-Run38993.eps}
\vspace{\floatsep}
\includegraphics[height=0.43\textheight]{PedestalSpectrum-25-Run38995.eps}
\caption{MExtractTimeAndChargeSpline with integral extraction over 2 FADC slices: 
Distribution of extracted "pedestals"  from pedestal run with 
closed camera lids (top) and open lids observing an extra-galactic star field (bottom) for one channel 
(pixel 100). The result obtained from a simple addition of 2 FADC 
slice contents (``fundamental'') is displayed as red histogram, the one obtained from the application 
of time-randomized weigths on a fixed window of 2 FADC slices as blue histogram and the one obtained from the
full algorithm allowed to slide within a global window of 12 slices. The obtained histogram means and 
RMSs have been converted to equiv. photo-electrons.}
\label{fig:int:distped}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics[height=0.43\textheight]{PedestalSpectrum-28-Run38993.eps}
\vspace{\floatsep}
\includegraphics[height=0.43\textheight]{PedestalSpectrum-28-Run38995.eps}
\caption{MExtractTimeAndChargeDigitalFilter: Spectrum of extracted "pedestals"  from pedestal run with 
closed camera lids (top) and open lids observing an extra-galactic star field (bottom) for one channel 
(pixel 100). The result obtained from a simple addition of 6 FADC 
slice contents (``fundamental'') is displayed as red histogram, the one obtained from the application 
of time-randomized weigths on a fixed window of 6 slices as blue histogram and the one obtained from the
full algorithm allowed to slide within a global window of 12 slices. The obtained histogram means and 
RMSs have been converted to equiv. photo-electrons.}
\label{fig:df:distped}
\end{figure}

\par

Because the background is determined by the single photo-electrons from the night-sky background,
the following possibilities can occur:

\begin{enumerate}
\item There is no ``signal'' (photo-electron) in the extraction window and the extractor 
finds only electronic noise. 
Usually, the returned signal charge is then negative.
\item The extractor finds the signal from one photo-electron
\item The extractor finds an overlap of two or more photo-electrons.
\end{enumerate}

Although the probability to find a certain number of photo-electrons in a fixed window follows a 
Poisson distribution, the one for employing the sliding window is {\textit{not}} Poissonian. The extractor 
will usually find one photo-electron even if more are present in the global search window, i.e. the 
probability for two or more photo-electrons to occur in the global search window is much higher than 
the probability for these photo-electrons to overlap in time such as to be recognized as a double 
or triple photo-electron pulse by the extractor. This is especially true for small extraction windows
and for the digital filter.

\par

Given a global extraction window of size $WS$ and an average rate of photo-electrons from the night-sky 
background $R$, we will now calculate the probability for the extractor to find zero photo-electrons in the 
$WS$. The probability to find $k$ photo-electrons can be written as:

\begin{equation}
P(k) = \frac{e^{-R\cdot WS} (R \cdot WS)^k}{k!}
\end{equation}

and thus:

\begin{equation}
P(0) = e^{-R\cdot WS} 
\end{equation}

The probability to find more than one photo-electron is then:

\begin{equation}
P(>0) = 1 - e^{-R\cdot WS} 
\end{equation}

Figures~\ref{fig:sphe:sphespectrum} show spectra 
obtained with the digital filter applied on two different global search windows.
One can clearly distinguish a pedestal peak (fitted to Gaussian with index 0), 
corresponding to the case of  $P(0)$ and further 
contributions of $P(1)$ and $P(2)$ (fitted to Gaussians with index 1 and 2). 
One can also see that the contribution of $P(0)$ dimishes 
with increasing global search window size.

\begin{figure}
\centering
\includegraphics[height=0.3\textheight]{SinglePheSpectrum-28-Run38995-WS2.5.eps}
\vspace{\floatsep}
\includegraphics[height=0.3\textheight]{SinglePheSpectrum-28-Run38995-WS4.5.eps}
\vspace{\floatsep}
\includegraphics[height=0.3\textheight]{SinglePheSpectrum-28-Run38995-WS8.5.eps}
\caption{MExtractTimeAndChargeDigitalFilter: Spectrum obtained from the extraction 
of a pedestal run using a sliding window of 6 FADC slices allowed to move within a window of 
7 (top), 9 (center) and 13 slices.
A pedestal run with galactic star background has been taken and one exemplary pixel (Nr. 100).
One can clearly see the pedestal contribution and a further part corresponding to one or more 
photo-electrons.}
\label{fig:df:sphespectrum}
\end{figure}

In the following, we will make a short consistency test: Assuming that the spectral peaks are 
attributed correctly, one would expect the following relation:

\begin{equation}
P(0) / P(>0) = \frac{e^{-R\cdot WS}}{1-e^{-R\cdot WS}}
\end{equation}

We tested this relation assuming that the fitted area underneath the pedestal peak $Area_0$ is 
proportional to $P(0)$ and the sum of the fitted areas underneath the single photo-electron peak 
$Area_1$ and the double photo-electron peak $Area_2$ proportional to $P(>0)$. Thus, one expects:

\begin{equation}
Area_0 / (Area_1 + Area+2 ) = \frac{e^{-R\cdot WS}}{1-e^{-R\cdot WS}}
\end{equation}

We estimated the effective window size $WS$ as the sum of the range in which the digital filter 
amplitude weights are greater than 0.5 (1.6 FADC slices) and the global search window minus the 
size of the window size of the weights (which is 6 FADC slices). Figures~\ref{fig::df:ratiofit}
show the result for two different levels of night-sky background.

\par

\begin{figure}[htp]
\centering
\includegraphics[height=0.4\textheight]{SinglePheRatio-28-Run38995.eps}
\vspace{\floatsep}
\includegraphics[height=0.4\textheight]{SinglePheRatio-28-Run39258.eps}
\caption{MExtractTimeAndChargeDigitalFilter: Fit to the ratio of the area beneath the pedestal peak and 
the single and double photo-electron(s) peak(s) with the extraction algorithm
applied on a sliding window of different sizes.
In the top plot, a pedestal run with extra-galactic star background has been taken and in the bottom, 
a galatic star background. An exemplary pixel (Nr. 100) has been used.
Above, a rate of 0.8 phe/ns and below, a rate of 1.0 phe/ns has been obtained.}
\label{fig:df:ratiofit}
\end{figure}



%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "MAGIC_signal_reco"
%%% TeX-master: "MAGIC_signal_reco"
%%% End: