Calibration questions/suggestions

[Rosen]

Hi Mabula – I’m still getting my feet wet with using APP. I have a few things that are a little confusing to me, so I’m hoping you can clarify. Also, I have some suggestions that might help the slow folk like me figure out how to use things!

1) Following your “Data calibration workflow and processing in APP” thread, I created a Bad Pixel Map. It created 2 files – Bad Pixel Map 1 (file name “BPM-QSICCDCamera-1694×1356.fits”) and Bad Pixel Map 2 (file name “BPM-QSI690wsHW12.12.00FW07.01.01S_N00604166-1694×1356.fits”. When looking at these two files, BPM 1 looks like a regular bad pixel map, whereas BPM 2 looks like an all grey file (100% of the pixels have a value of 127). I presume that I should only be checking BPM 1 when calibrating my flats and darks.

Is that correct? If so, what is the reason for generating Bad Pixel Map 2 – perhaps it would be better to not create it at all?

2) If I previously have created a Bad Pixel Map, I would presume that I could reuse this file for many months. However, I couldn’t find any way to load a previously generated BPM. Perhaps this should be included on the “1) Load” page?

3) If I select “remove light pollution” when calibrating my lights, it asks me to select 5 areas. However, I have no idea what kind of areas it wants me to select? Is it areas where there aren’t any stars, background nebulosity, or ???? It might be helpful to include a description in that message box so the user isn’t wondering what to do.

By the same token, there’s a calculate box and an OK box. It’s not clear that you’re supposed to do the calculate first (I think that’s correct), as it identified some yellow boxes, I removed them, and then kept selecting new boxes until I had 5 green ones.

So, I think I have it working correctly, but I think some more complete directions in the remove light pollution dialog could help make it much easier to use.

4) After I had found good boxes for the light pollution model and clicked “Ok”, it asked me again to find 5 boxes. I’m guessing that it wants me to pick boxes for each subexposure. If that’s the case, it would make using the remove light pollution function pretty much unusable. I often do hundreds of subs. Even if I only had 15 or 20, that’s a lot of box picking! Would it be possible to apply the same boxes to each subexposure after the first? Or, perhaps I’m just not using this capability correctly.

I REALLY like the idea of removing the LP from each sub before stacking. That seems like an excellent idea versus the traditional stacking and them remove some gradient that’s traveled across the frame all night long. I’m just hoping there’s an easier way to implement it (or easier way for me to use it!).

Thanks,

Scott

[Haverkamp]

Hi @srosenfraz,

Let me try…

1) The creation of 2 bad pixel maps would happen if the instrument name of the darks and or flats are not equal. The Bad Pixel Map will be matched with lights or flats of the same instrument.

Sometimes, different capture programs will output different isntrument names, so be aware of this.

And there is a good and simple solution to this:

in 9) tools, I have made the batch modify tool (made especially for the conversion of your old IP data ;-) ). You can adjust the instrument name in the fits header of a batch of files. Make sure that you modify it exactly to the instrument name of your lights. And when you create a bad pixel map, check that all files have the correct instrument name.

The batch modify tool will output new files with the proposed changes.

Regarding the BPM, in the fits header you’ll find info about the hot/cold and linear pixels that were detected.

pixels with a value of 0 are cold

pixels with a value of 127 are linear

pixels with a value of 255 are hot

In my experience, the amount of pixels that are hottish, can vary from 0.01 to as much as 1% and more with older sensors.

[Haverkamp]

2) Yes, I have been using my BPM of my Nikon D5100 for more than three years with excellent results.

APP wil detect whether a file that is being loaded is a master calibration frame or not. So you could use any load button to load the BPM. It will recognize it’s a BPM and use it as such. Same holds for the other master calibration files. The fits header information is used for this, which you can check with the details selectbox above the image viewer.( You can also see all meta data of DSLR RAW frames.)

[Haverkamp]

3) Good questions

You need to select areas where there is only sky background with stars. So no nebulosity should be there. Dark nebula shouldn’t be used as well. APP uses very robust statistics for the tools where you need to create area select boxes. So it’s no problem if you select area’s with a lot of stars, as long as there is no nebulosity. I’ll make a short movie of this, correcting one of your frames. I will give you a signal when it’s uploaded.

The flexibility slider plays a very important role here. It damps the correction and this will help you if you place your area select boxes on wrong spots. Due to the model damping, a misplaced select box will have less influence on the model and it could also turn red, indicating it’s a possible misplaced select box.

1 is least flexibility, largest damping

16 is no damping at all

Indeed, calculate will create a model and show you the image corrected for the model. Note that APP immediately shows your image background calibrated using the supplied are select boxes. The tool starts off with showing your data background neutralised. The background calibration tool should always be used after the remove LP tool.

And finally, clicking OK, will apply the current model to your lights and save the result with a -lpc postfix.

About the colors of the select boxes.

Green are all select boxes with a deviation to the model within 1 standard deviation.

Yellow is everything between 1 and 2 standard deviations

Red is larger than 2 standard deviations.

They way to use these colors is the same as the 123 sigma rule or

https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule

The select boxes are samples of a distribution.

68% should be within 1 standard deviations so roughly 2/3s of the select boxes should be green at least.

95% shoudl lie within 2 standard deviations, so roughly 19/20 of the select boxes should be green or yellow.

The remaining 5% can be red. So 1 in 20 select boxes can be Red and not pose a problem.

So yellow select boxes are in most case fine select boxes, and don’t need to be removed.

If a red select box pops up, double check it, if you think you really sampled onlu background and stars without nebulosity, keep it. If in doubt, remove it, and place a new select box somewhere else.

In really difficult cases, you can use this tool iteratively. But in most cases this is not necessary.

I will explain this in the manual as good as I can ?

[Haverkamp]

4) The 5 boxes are a minimum, the more you’ll select the better. It also helps to place them evenly spread across your image. If you put 20 boxes in one corner and only 4 in the other corner, that corner could have less correction as you would like. So each box has a weight in the results.

The LP correction for calibration is indeed way to cumbersome. It takes forever with a lot of frames.

In the RFC list, is the proposal to make this a batch tool for calibration. For the first frame you select the boxes and then the tool would apply those boxes to all the frames that you are calibrating. It should work really well after you have registered those frames first. The registration will ensure that good areas are used automatically in the gradient/LP correction of all frames ;-) so this feauture will come soon. It will be really helpfull. (So your proposal in already on the RFC list)

The Local Normalization Correction during stacking will also work pretty well, it causes all layers in the stack to become evenly illuminated over the entire field of view, giving you better data normalization and much less stack artefacts. But the LNC won’t ensure a completely flat stack, whereas this proposed LP calibration batch tool would ensure this. So it will be very usefull, no doubt ;-)

[Haverkamp]

Probably the batch LP correction tool + LNC (1st degree 3 iterations should do it then )during stacking will give you results that will look almost perfect straight after stacking… I’ll try to implement this batch LP tool in the next version ;-)

Let me know if all of this is clear or if you have additional questions ;-)

Mabula

[Author]

Posts