Astro Pixel Processor

APP stacking process slowing down with 157 Lights

Viewing 15 posts - 1 through 15 (of 23 total)
  • Author
    Posts
  • #13344

    KeesScherer
    Participant
    posts: 1149

    (Version 1.034 8GB DEB package on Panasonic Toughbook Core i5 16 GB RAM)

    I started a stack from scratch of 157 Lights (CR2 Canon 6D)/ 34 Flats and 150 Bias. All files and APP work directory on external USB3 HDD. After 5 hours the “Creating Lights stack” is at 36%. APP says: ” Stack task requires 35 GB of free harddisk space but the Work folder has 35.0+17.5 GB in 2 dat files at that point. See screenprint for status) The stacking process is slowing down more and more. The 40 Lights stack yesterday finished within 2 hours. Progress is now 8%-9% per hour and at this pace the stack would need over 9 hours to complete. The external hard drive speed is not much of an issue here, each 6 seconds of disk activity is followed by 60 seconds of inactivity.

    #13357

    Haverkamp
    Participant
    posts: 640

    Thank you Kees for your feedback.

    I forget to include the weight map in the required HDD space, I will put this on the RFC list.

    “The 40 Lights stack yesterday finished within 2 hours. Progress is now 8%-9% per hour and at this pace the stack would need over 9 hours to complete. The external hard drive speed is not much of an issue here, each 6 seconds of disk activity is followed by 60 seconds of inactivity.”

    Okay, this is really slow… ;-( and I think I know why. It probably has to do with memory management. I need to investigate the stack engine and make a big stack myself while monitoring the application for CPU, MEM and HDD activity.

    Speed is very important so I will make this a priority and put it at the top of the RFC.

    • 1 person likes this.
    #13478

    Haverkamp
    Participant
    posts: 640

    Hi @keesscherer,

    I have uploaded new versions 1.034.2 , can you please try another similar stack, to check if the speed has improved now?

    I have fixed some problems, and I think memory management is now much better. APP shouldn’t slow down anymore like you mentioned.

    In my tests over here, APP is significantly faster and doesn’t slow down. So if it still does on your system, let me know 😉

    Mabula

    • 1 person likes this.
    #13516

    KeesScherer
    Participant
    posts: 1149

    Thanks Mabula, I will do that tomorrow (31/3).

    • 1 person likes this.
    #13736

    KeesScherer
    Participant
    posts: 1149

    The same process is running now, i have added the iso200 bias frames also and total time will be 9 hours for this stack versus the 12 hours for the previous trial run.

    #13738

    Haverkamp
    Participant
    posts: 640

    Hi @keesscherer, thank you,

    So with more total frames, but still 9 hours… ;-( (but at least some improvement)

    Were you using distortion correction in the registration proces or not?

    (I noticed the dynamic distortion correction could take a very long time with more than 100 lights as well.)

    #13743

    KeesScherer
    Participant
    posts: 1149

    After 7 hrs still at 49% so it will take 14+ hrs. No distortion correction.

    #13753

    Haverkamp
    Participant
    posts: 640

    Amazing… I don’t believe I experience this slow behaviour on Windows or Mac. So something fishy must be going on which causes this slow speed ;-(

    Stacking 40 Canon 6D frames only takes about 21 minutes on my Windows 7 i7 930 quad core, (which is almost 7 years old).

    Are you using the 4GB version on the machine with 8GB mem?

    Are you running the stack with or without oulier rejection? If you are, turn it off and check if that has some influence? Do you know which part of 3)Calibrate to 6)Integrate has the longest duration?

    Possibly the platform in which APP runs has a problem on your Linux Mint system, which I am unaware of at the moment. Maybe we could schedule a remote desktop/teamviewer session next week so I can have a look at how APP behaves on your system(s)?

    Mabula

    #13755

    KeesScherer
    Participant
    posts: 1149

    I am running the 8GB version on the 16GB machine. With sigma clipping. The time consuming step is 6, all the others were done in 3 hours. I still don’t understand how i should handle the 200 iso flats, i loaded the lights 1600 iso, bias 1600 iso and 200 iso bias frames i selected sigma clippng and clicked stak on tab 6. But the flats still do not work ( i saw this :

    ” So your masterbias of iso200 for your flats and your masterbias of iso1600 for your lights will automatically be applied to the correct frames (that have the same ISO), since APP has the strict rule that master bias frames must match the iso of the frame that needs calibration.”)

    so i must assume APP needs some other manual setting to work with the 2 sets of flats?

    EDIT: The stack looks better than the first stack after all…

    All my systems work with a LAN only Teamviewer and i don’t like to change that setting.

    #13764

    KeesScherer
    Participant
    posts: 1149

    I am uploading the finished stack, and master bias/flat frames to our shared Dropboxfolder. (the timestamps show that the first master was made at 14:03 yesterday and the final stack at 01:42 this morning so about 12 hrs difference.

    #13767

    KeesScherer
    Participant
    posts: 1149

    Last night’s APP stack (right) looks better than the first APP stack when given the same autostretch in PI, see screenprint. By the way, feel free to use this Monkeyhead Nebula stack for promotional purposes. (All my material is Public Domain b.t.w.)

    #13772

    KeesScherer
    Participant
    posts: 1149

    Same stack in DSS, PI and APP (Left as opened in PI, right with same autostretch applied [target background 0.12])

    #13773

    KeesScherer
    Participant
    posts: 1149

    The FWHMEccentricity script results:

    —————–FWHM             Ecc

    DSS                2.580              0.4261

    PI                    1.998              0.4610

    APP                1.989               0.4599

    #13778

    Haverkamp
    Participant
    posts: 640

    Thank you @keesscherer,

    “I am running the 8GB version on the 16GB machine. With sigma clipping. The time consuming step is 6, all the others were done in 3 hours.”

    Okay, so the major bottleneck is the stacker currently… looking at the name of your stack, I think I’ll investigate the Linear Fit outlier rejection filter in APP. Possibly this becomes really slow in APP due to the regression that is done in each pixel stack to calculate linear fit parameters. The more frames you use the slower it becomes probably. So I’ll run some tests with that filter and compare it to the sigma and winsor filters. If this is the culprit, I’ll need to think of a smarter way to use this filter.

    Having said this, I also notice you are using 3 iterations with kappa 2 (the default values currently). For 150 lights this would normally be way too aggresive for outlier rejection. I will set the default values to much more conservative settings in the next version ( 1 iteration kappa 3), since with stacks of 50 frames, usually in APP, this is all you need to reject all outliers. Probably, depending on implementation, the setttings that you need to use in different programs can be quite different due to different implementations of the calculation of statistical properties of the pixel stacks.

    And this is probably good news for your Monkey Head stack, I am quite certain that with less clipping the stack will be much better when judged on noise and SNR.

    In the FITS header of the stack (to be seen with details selectbox above image viewer), you’ll find stack statistics. the median and reference noise reduction are only 6 for a stack of 150 lights. If you have dithered your data, this is really low. Ideally, the reduction is the square of the number of lights. So for 100 frames, the best possible noise reduction is 10. So for 150 lights, the noise reduction could possible be doubled to 12 ;-). This is a clear sign to me that possibly the outlier rejection is way too aggresive.

    I still don’t understand how i should handle the 200 iso flats, i loaded the lights 1600 iso, bias 1600 iso and 200 iso bias frames i selected sigma clippng and clicked stak on tab 6. But the flats still do not work ( i saw this :

    ” So your masterbias of iso200 for your flats and your masterbias of iso1600 for your lights will automatically be applied to the correct frames (that have the same ISO), since APP has the strict rule that master bias frames must match the iso of the frame that needs calibration.”)

    so i must assume APP needs some other manual setting to work with the 2 sets of flats?”

    I guess 2 sets of flats, would be 2 sets of bias?

    I’ll make a calibration video with sound to show what should happen. There is no manual setting. APP should apply a master bias automatically to the frame needs calibration with the same ISO.

    “All my systems work with a LAN only Teamviewer and i don’t like to change that setting.”

    No problem, I’ll investigate the Linear Fit outlier rejection filter. I strongly suspect this is the problem.

    Adding to this, I actually never use the Linear Fit filter. I consider it deprecated due to the implementation of LNC. A linear fit filter is a solution for conventional normalizaion of data where the gradients in the stack change clearly between the frames. Linear Fit can account for this and help with outlier rejction. But LNC will solve this automatically by correcting all subs in your stack for gradients before data integration. The concept of a Linear Fit filter then becomes deprecated and unadvisable. I’ll make work of this to explain this more thoroughly in the manual, since LNC is a new concept.

    So my advise would be, try to stack your 150 lights with normal sigma clip 1 iteration, kappa 3 and LNC on 1 iteration first degree. I wouldn’t be surprised if the stack comes out much better ?

    (I can also put it differenly: a linear fit filter is a solution for bad data normalisation, LNC solves this at it’s root by strongly improving data normalisation in your stack in the entire field of view and therefore a linear fit filter becomes useless and a normal sigma/winsor filter will suddenly work much better due to much better data normalisation.)

    #13779

    Haverkamp
    Participant
    posts: 640

    Same stack in DSS, PI and APP (Left as opened in PI, right with same autostretch applied [target background 0.12])

    Hi Kees, please note, that these stacks possibly have different lokations and dispersions, so using a similar stretch with a target background of 0.12 can’t be considered good comparisons if you don’t normalize the data first.

Viewing 15 posts - 1 through 15 (of 23 total)

You need to log in or to reply to this topic.