Thank you Kees for your feedback.
I forget to include the weight map in the required HDD space, I will put this on the RFC list.
“The 40 Lights stack yesterday finished within 2 hours. Progress is now 8%-9% per hour and at this pace the stack would need over 9 hours to complete. The external hard drive speed is not much of an issue here, each 6 seconds of disk activity is followed by 60 seconds of inactivity.”
Okay, this is really slow… ;-( and I think I know why. It probably has to do with memory management. I need to investigate the stack engine and make a big stack myself while monitoring the application for CPU, MEM and HDD activity.
Speed is very important so I will make this a priority and put it at the top of the RFC.