Astro Pixel Processor – Calibration, integration and post-processing

This article in the Astro Pixel Processor series about working with APP gives an in-depth explanation about data-calibration with APP. The workflow uses data from a Astro Pixel Processor Beta group member in order to illustrate how to get correct data calibration using APP.

This workflow is using a small subset of M 51 data (The Whirlpool galaxy) in Canes Venatici from Hans van Overzee.

Data details:

  • Object: M51, the Whirlpool galaxy
  • OTA: Takahahsi TOA-130
  • Canon EOS 600D (mod)
  • Lights: 5 x 90 seconds, iso 1600
  • darks: 5x 90 seconds, iso 1600
  • bias: 5x 0 seconds iso1600

Step 1) Create a Bad Pixel Map

We first create a Bad Pixel Map. Since we only have darks, we can only correct hot pixels. Cold pixels can be corrected using flats but most modern camera’s hardly have any cold pixels at all.. so this is no significant problem.

Clicking on the images in this article is possible and recommended to view the images full screen.
Astro Pixel Processor Bad Pixel Map

Visual inspection of the Bad Pixel Map. strongly zoomed in, shows bad pixels evenly distributed over the sensor

Astro Pixel Processor BPM header

The fits header of the bad pixel map shows statistics on the amount of bad pixels. Since only darks are used, we only have hot pixels, 0,875% of the sensor is considered hottish in this case, which is pretty normal.

Be aware, put some effort into creating a good bad pixel map, since this Bad Pixel Map or BPM can be used for a very long time. You only need to make a good BPM once per year or even longer. I have been using a BPM for my Nikon D5100 BCF mod (firmware hacked: true dark current patch) for over 3 years now with high efficiency, still no need to make a new one… A good BPM can be created using about 20-50 darks with exposure times well over 5 minutes and the iso/gain of the camera that you are most likely to use in your imaging sessions. And also 20-50 flats with exposures of about 1 second or longer should be fine, iso value isn’t very important here. Remember, the more frames you use for the BPM, the better it will be for your bad pixel correction for a long time to come 😉

We first load the dark frames with 1) LOAD and push the dark button to select the darks. The we go to 2) Calibrate and scroll down to the Bad Pixel Map options. We select create bad pixel map and set hot pixel kappa at 1. If you have more darks ( >20), the kappa usually can be set at 2 for good results. Click on create calibration masters. After the creation of the Bad Pixel Map, we investigate the bad pixel map by zooming on it and we check the fits header for statistics. The amount of hot pixels, i.e. pixels that seem to show non-linear behaviour, is 0,825% with the hot kappa 1 setting. This is a very normal value and probably good for this small data set.

Step 2) Create a Master Bias

Astro Pixel Processor Master Bias

Visual inspection of the master bias

Astro Pixel Processor Master Bias statistics

Master Bias statistics in the FITS header, showing mean, medain, standard deviation and nosie values per channel of the CFA data of the Canon EOS 600D CR2 files( R, G1, G2, B)

We first deselect the create bad pixel map option. Then we go to 1) Load and push the clear button. The file list is now completely emptied. We proceed by loading the bias frames by cliking on the bias button. At 2) Calibrate, we use median integration for the bias master integration without outlier rejection. If you have more than 20 bias frames (which you’ll normally have), select average and use normal or winsorized sigma clipping for outlier rejection with a conservative setting, like 1 iteration kappa 3.(Don’t use Linear Fit clipping anywhere in the calibration process!) Click on create calibration masters.

After APP is finished, check the statistics of the master bias. You’ll notice, that the more bias frames that you’ll use, the lower the Multi Resolution Support Gaussian Noise (MRS noise) estimate becomes. So with more frames, the master bias will become less noisy. Your aim should be to to create a master bias with very little noise, so that means, use plenty of bias frames. I usually use 100-250 frames. The explanation being: the noise in the master bias will always be injected in your light frames in calibration, so try to reduce this injection to a minimum by using a lot of bias frames.

Step 3) Create a master Dark

Astro Pixel Processor Master Dark

Visual Master Dark inspection

Astro Pixel Processor Master Dark statistics

Master Dark statistics in the FITS header

We can now choose 2 ways to properly calibrate the lights frames with a master dark:

  1. calibrate all darks with the master bias, so we subtract the bias pedestal from the darks. This will mean that both the master dark and master bias should be used in the calibration of the lights
  2. don’t calibrate the darks with the master bias, since the bias is already part of your darks. This will mean we’ll only use the master dark in the the calirbation of the light frames. So the master bias actually becomes redundant. (This is actually the recommended method for camera sensors that have significant fixed pattern noise in the dark current, so dark current calibration is really needed)

I’ll choose method 2). So I remove the bias frames and master bias from the file list by clicking on clean in 1)LOAD. Then I’ll load the darks by clicking on the dark button. At 2) Calibrate, I choose median integration and no outlier rejection for the same reasons as in the previous bias integration. Click on create calibration masters. And check the master dark visually and the statistics in the FITS header.

Step 4) Calibrate the light frames and remove possible chromatic aberration

Astro Pixel Processor uncalibrated light frame

uncalibrated light frame zoomed in on M51. We see plenty of hot pixels. Image is shown with boosted saturation in the preview filter.

Astro Pixel Processor calibrated frame

the same light frame, but calibrated with the master dark and bad pixel map. Clearly, the BPM has corrected a lot of bad pixels. Image is shown calibrated using the l-calibrated image viewer option.

in step 3) I chose to do dark calibration without bias calibration of the darks frames, so I only need the master dark (which contains the Bias) for dark and bias calibration at the same time.

I clean the file list again and I will load the lights with the light button. Then I load the masterdark and our Bad Pixel Map as well using any button to load frames (APP will detect these master frames). And I visually inspect the uncalibrated and calibrated images using the linear and l-calibrated image viewer options. Clearly, the bad pixel map is doing a very good job, since the calibrated frame hardly shows any hot pixels.

The data of Hans has no visible chromatic aberration due to his fantastic refractor, the Takahahsi TOA-130. If your data does have colored cyan/magenta/red/blue edges on the stars indicating chromatic aberration, APP can remove this to a large degree (both lateral and tranversal CA thanks to a dedicated registration model for CA correction) using the align channels option, part of the options to save the calibrated frames. In this case I don’t use this option since I can’t visually detect any chromatic aberration.

Data calibration is now finished.

Step 5) analyse stars & registration

So wtih the 5 lights loaded together with the master dark frame and the bad pixel map, we proceed with 3) star analysis. We use default settings so we change nothing. And at 4) register we also use the default settings. We then click on start registration. Now both star analysis and registration are performed. Let’s check the results after registration in the file list panel, since we encounter some output that needs our attention.

Astro Pixel Processor registration results

Analytical results of star analysis and registration of the frames

First look at the #stars & star density colomn. In frame 2 and 3, the amount of stars that are detected is much lower than the other frames. This is a direct and clear indication that these frames or bad compared to the other three. The star density value, is a value that corrects the #stars value to make the stars value image scale and Field of View independent, hence the name star density. This is most useful in case you want to integrate frames with different imaging scales and/or Field of Views. So the star density value is an image scale & Field of View independent metric and much better suited for comparison of quality of your frames when you have frames of different scales and/or Field of Views.

Secondly, look at the Full Width Half Maximum, or FWHM, value of the stars in the frames. We see a similar pattern. Frame 2 & 3 have strongly elongated stars when compared to the other frames. The reported FWHM is the average FWHM of all of the detected stars. The min and max FWHM value is the minimum FWHM cross section of this average star profile in the frame, where as the max FWHM, is the maximum cross section of this average star profile in the frame. If the image has perfect round stars, then the min and max FWHM value should be the same. If the max value is twice the min value, it indicates that the star, in the direction of the maximum cross section, is twice as big as in the direction of the minimum cross section.

FHWM is als reported relatively instead of absolute, so image scale independently, like the star density. So APP can actually compare the quality of frames of different image scales and Field of VIews directly.

Thirdly, the registration RMS of the frames that were registered to the reference frame, frame 5, also show a problem which simply can’t be corrected due to the strongly elongated stars. Frame 2 and 3 are hard to register since they contain elongated stars. Possibly the result of bad tracking during the exposure. Frame 2 has a registration error of 0,58 pixel for 120 star pairs between frame 2 and it’s reference frame. This is not good, but in this case acceptable. Frame 3 has an error of 2,38 pixel. We can’t accept this so we will not use this frame later on in integration.

Finally, the quality of the frames is expresses as a number, which is the result of a formula containing, star density, star shape, star size and noise which will be calculated in the next step. Frame 2 and 3 are worse than the other frames again.

Step 6) image normalization

We use default settings, we normalize with add + scale and biweight midvariance as scale estimator. And we enable background neutralization. We click on normalize lights and check the analytical results in the file list window. Basically, what we find is that the 5 frames are almost identical for background, dispersion, SNR and noise. The background, dispersion and noise values are always reported normalized in the range of 0 to 1. So if you were planning to integrate frames of both 16bits and 32bits, the data normalisation is always done in the 0-1 data range (32bits floats) so this is no problem for APP.

Notice that the quality score is adjusted now, since the MRS gaussian noise in the frames is taken into account as well in the quality calculation. Picture stays the same however. Frame 2 and 3 are not very good when compared to the other frames.

Astro Pixel Processor normalization analytical results

Image normalization analytical results. We find the background (location), scale/dispersion, SNR and noise of the frames.

Step 7) image integration

We decided earlier we definitely don’t want to use frame 3 in the data integration. I use the “lights to stack” slider to deselect this frame from stacking.

Astro Pixel Processor lights to stack selection

We deselect the worst frame(s) by using the lights to stack slider. automatically, the worst frames are deselected. You can verify this, since the word STACK is not present anymore in the quality score column for the frame(s) we wish to exclude from stacking.

So we are only integrating 4 frames, so this time again, we choose median integration without outlier rejection. I set the composition mode to full to have a field of view for the stack that includes all pixels of all frames. I enable MBB at 5%, no LNC. I don’t create output maps. Data interpolation is set at Lanczos-3 with NUOS, since we have very little frames. Other settings are left at default. We click on stack  When APP is finished, we inspect the the stack/data integration and statistics in the fits header. No doubt, 4 x 90 seconds is a very short integration time for this object. If we were to combine a lot more light frames, this should prove to be a very nice image 😉 No doubt as well. Colors are very good already.

Astro Pixel Processor M51 4x90sec

M51 stack of 4×90 seconds.

Astro Pixel Processor M51 4x90sec stack statistics

M51 stack statistics of 4×90 seconds.

Step 8) post processing of the stack

I’ll now do a short postprocessing of  the stack of Hans’ data. First I do a light pollution correction, then background calirbation and in the end a final crop of the field of view. And, in the end, I stretch the data with some  added saturation. I now start to notice that flat calibration is really recommended. I do see some dust donuts appearing when I stretch the data a bit more aggresive.

I think, that if we were to combine more data and we would add good flat calibration into the mix, this will become a very nice picture sooner than later! For 4×90 seconds this is already very nice, especially considering the colors in M51, the Whirlpool galaxy.

Astro Pixel Processor, M51 by Hans van Overzee

M51, 4×90 seconds © Hans van Overzee

Please feel free to comment/or make remarks to this article or ask your question in the APP Group.

Copyright © Mabula Haverkamp (author & developer APP) & © 2017

3 comments on “Astro Pixel Processor – Calibration, integration and post-processing

  1. Profile photo of HaverkampHaverkamp Post author

    Hi Rob, thank you for your question.

    It’s a Bad Pixel Map, not a Dark Pixel Map.

    The purpose of a bad pixel map is to correct all bad pixels of your sensor,

    hot pixels, which are pixels that don’t behave linearly and react more strongly than the surrounding linear pixels to incident light. These are found from your dark frames.
    cold pixels, these could be defect pixels or pixels that don’t behave linearly to light and react very little to incident light. Thes are found from your flat frames.

    So both dark and flat frames can be used to create a bad pixel map.

    Unlike dark and flat calibration, BPM calibration will never inject noise. There is no add/subtract/multiply/divide involved in the calibration process like in dark and flat calibration. The bad pixels which are lokated in the Bad Pixel Map are simply corrected using data interpolation from linear pixels in the calibration proces.

    So for the purpose of removal of just hot pixels, you should always use a Bad Pixel Map instead of dark calibration, since dark calirbation will always inject noise in your light frame where the BPM will not.


Leave a Reply