- This topic has 8 replies, 2 voices, and was last updated 9 years ago by
.
-
Posts
-
I am testing different settings for the integration step. Looking at the 2 different rejection maps for this image it looks clear the kappa value of 2 (on the right) results in a rejection level that is too high, but also the Kappa 3 shows a lot of rejects. But i can not go any higher than 3?
Hi @keesscherer,
Indeed, the kappa value of 2 is way too high if you look at the rejection map.
The kappa value of 3 is much beter. Very little removal of possible good data and the outliers are gone.
In the next version, after 1. 036.0, you’ll have the option to use kappa values larger than 3.0. I have extended the range to 5.0.
But some clarification is usefull here. From a technical statistics point of view, a kappa value larger than 3 will only be effective if you were to stack more than several 1000s of frames. See also
standard deviation, sigma, 68,95,99,7 rule
Let’s suppose the following, we have 500 light frames, that we are going to stack with an outlier rejection filter with a kappa value of 3. And that we have no outliers at all, because we are interested in how much an outlier rejection filter can harm the good data, which shouldn’t be rejected, in your stack.
The question now is, how much of the good data will be removed by this outlier rejection filter, which would be harmfull for the Signal to Noise ratio of your stack.
The calculation from the 68,95,99,7 rule is:
for kappa = 3, all data (good or bad), less than the median – 3 standard deviations and greater than the median + 3 standard deviatons will be removed. The median is the central value for each pixelstack.
for 3 standard deviations, this means that 99,7 % of all the frames are included and 0,3% are excluded and thus rejected.
0,3% of 500 frames = 1,5 frame. Let say 2 frames out of 500 frames.
So setting the kappa at 3 for a very big stack will hardly have a destructive influence on the SNR ratio of your stack. Which I have actually tested in the past. This testing is the reason why I set the kappa range to 1.0-3.0.
Using kappa values larger than 3.0 will have a higher quarantee that only real outliers are rejected, but also increases the chance that it will be less effective. And kappa 3 already has no sigificant influence on the SNR of your stack. This is less than 1%, not 10%, because the SNR has the following relation to the amount of subs, it scales as the square root of the number of subs. Twice as many subs only increases the SNR by the square root of 2.
Let’s say, the 500 frames integration gives an SNR of 20.
Then the 498 frames stack would give an SNR of square root of (0,997) * 20 = 0,9985 * 20 = 19,97
This is by no means significant. It’s 1 – (19,97/ 20) = 0.0015 or 0,15 %
So feel free to experiment with the different kappa settings in the next version and check the SNR and noise values of your stack (in the fits header). If you see that using higher kappa values than 3 is significantly beneficial for the combination of effective rejection and the SNR of you stack, I would be surprised actually. But that’s always a good thing I guess ?
Mabula, what am i looking at here, a master dark rejection map made from 65×600 seconds -20C darks. Cosmic ray hits?
Interesting to see ?
65 darks of 600 seconds integrated (average) and Winsorized sigma clip (WSC) 1 iteration kappa 3.
I suppose these could and should be cosminc rays indeed, they should be visible on some of the individual subs then. Interesting to see how bright they are in the frames that were hit.
Can you find a frame with such a streak?
And some statistics, the cosmic ray has 1758 ADU, average=828 and sdev=112, so 828+3×112=1164. The 1758 gets rejected alright……
Excellent @keesscherer,
You captured quite a few of those cosmic rays ;-)
Regarding the statistics and rejection of them:
The outlier rejection works a little bit differently. An outlier rejection filter while integrating several frames, is a filter the determines the central value (average/median for example) of a pixel stack. So all pixels of all frames that will make up 1 pixel in the integration are analysed. So the surrounding pixels don’t have any influence on the rejection for a pixel in the stack.
The principle is the same, like you mention. From that pixelstack, we calculate the central value (in APP this is the median, since this is way more robust than the average). And we calculate the standard deviation for the rejection boundaries relative to this central value.
For Winsorized clipping, a winsorized central value and standard deviation is calculated at each rejection iteration.
Regarding the statistics and rejection of them: The outlier rejection works a little bit differently.
Yes of course, but i will not do the calculation for the 65 frame stack by hand. The statistics for all the frames do not differ much from this “back of the envelope calculation”. The APP statistics for this master dark show a standard deviation of 125, close to the 112 that you see in one frame. (yes, that is sdev for all pixels in one frame versus one pixel in all frames so not correct either….)
FITS HDUs: 1
HDU1 – SIMPLE = T / Java FITS: Sat Apr 29 11:01:14 BST 2017
HDU1 – BITPIX = 16 / bits per data value
HDU1 – NAXIS = 2 / number of axes
HDU1 – NAXIS1 = 4520 / size of the n’th axis
HDU1 – NAXIS2 = 3616 / size of the n’th axis
HDU1 – EXTEND = T / Extensions are permitted
HDU1 – BSCALE = 1.0 / scaling to 16bit
HDU1 – BZERO = 32768.0 / offset data range to that of unsigned short
HDU1 – DATE = ‘2017-04-29T10:08:46’ / creation date of master dark
HDU1 – SOFTWARE= ‘ Astro Pixel Processor by Aries Productions’ / software
HDU1 – CALFRAME= ‘master dark’ / master dark frame
HDU1 – INSTRUME= ‘QHYCCD-Cameras-Capture’ / instrument name
HDU1 – CFAIMAGE= ‘no ‘ / Color Filter Array pattern
HDU1 – GAIN = 1.0 / gain or ISO depending on instrument
HDU1 – EXPTIME = 600.0 / exposure time (s)
HDU1 – MEAN = ‘ 7.79E+02’ / mean of channel
HDU1 – MED = ‘ 7.75E+02’ / median of channel
HDU1 – SIGMA = ‘ 1.25E+02’ / standard deviation of channel
HDU1 – NOISE = ‘ 2.50E+00’ / MRS gaussian noise estimate of channelRegarding the statistics and rejection of them: The outlier rejection works a little bit differently.
Yes of course, but i will not do the calculation for the 65 frame stack by hand. The statistics for all the frames do not differ much from this “back of the envelope calculation”.
Indeed, since this is a masterdark, you are totally right ;-)
- You must be logged in to reply to this topic.