I was wondering if anyone else might have noticed some unusual behavior when running the Finantic Optimizers (SMAC and Bayesian) in parallel mode. Initially they start out with 40%-60% CPU utilization but as the optimization proceeds the CPU utilization decreases to ~10%. Furthermore, when I cancel the optimization I get "Completing Last Running Threads..." which seems to run indefinitely (I actually let it run about an hour before shutting it down the "hard way"!) 😉
I am coming to the conclusion that, for whatever reason, the optimizer has threads which never complete, stop running (or barely run) and eventually become zombies. Does this make sense?
I am coming to the conclusion that, for whatever reason, the optimizer has threads which never complete, stop running (or barely run) and eventually become zombies. Does this make sense?
Rename
QUOTE:
... in parallel mode. Initially they start out with 40%-60% CPU utilization but as the optimization proceeds the CPU utilization decreases to ~10%.
And why is that? You need to do more troubleshooting to determine why that is, then we can respond with our thoughts.
One possibility is that the process(es) are running out of resources to solve the problem. Let's take memory management as an example. Perhaps the problem is growing in size such that it no longer fits within the "working set" of the WL application. When that happens, the OS page swapper will start paging like crazy. Fire up Process Explorer and see if the paging count is rising fast for the WL process.
QUOTE:
Furthermore, when I cancel the optimization I get "Completing Last Running Threads..." which seems to run indefinitely
I get that informational message too. But the holdout thread eventually does complete. Perhaps there's some stock in your dataset that's causing your strategy to behave badly. Try using a much smaller dataset (that may exclude the problematic stock) to see if the problem goes away.
Are you using PreExecute? If so, does this problem go away with strategies that don't use PreExecute? It might be an indicator colliding problem with different threads in PreExecute.
superticker,
Thanks for the suggestions!
I doubt it is memory. I have 192GB (128GB available) and I am only using ~30GB for the optimization. Is there some other "resource" that I need to monitor or adjust?
I tried this on a portion of the current NASDAQ 100 (84 stocks), all VERY liquid and all trading. Same result. Never shuts down, even after over an hour.
No. very plain vanilla (I am not sophisticated enough to to anything fancy, yet! 😉)
Thanks for the suggestions!
QUOTE:
One possibility is that the process(es) are running out of resources to solve the problem.
I doubt it is memory. I have 192GB (128GB available) and I am only using ~30GB for the optimization. Is there some other "resource" that I need to monitor or adjust?
QUOTE:
I get that informational message too. But the holdout thread eventually does complete. Perhaps there's some stock in your dataset that's causing your strategy to behave badly. Try using a much smaller dataset (that may exclude the problematic stock) to see if the problem goes away.
I tried this on a portion of the current NASDAQ 100 (84 stocks), all VERY liquid and all trading. Same result. Never shuts down, even after over an hour.
QUOTE:
Are you using PreExecute?
No. very plain vanilla (I am not sophisticated enough to to anything fancy, yet! 😉)
QUOTE:
I doubt it is memory. I have 192GB (128GB available) and I am only using ~30GB for the optimization.
The more memory you have the larger the OS page file. And the more page swapping may be a problem. It's not a problem with the memory itself; it's a problem with page swapping. Tell us what Process Explorer is telling you about the paging frequency.
If it is a page swapping problem, then it's telling us you are running too many processor cores at once to fit into the working set of the WL application. The solution there is to tell Windows to allocate less processor cores to the problem, which you can control with Windows 11 (but not Windows 10). Honestly, I wouldn't allocate more than 3 or 4 cores to a large WL optimization problem. You may run out of L3 cache at 4 cores depending on how big your L3 cache is on the processor chip you are using.
QUOTE:
Is there some other "resource" that I need to monitor or adjust?
That's my point. You need to put your system administrator hat on and look at the OS performance metrics to find the resource that's choking your optimization process. Others haven't responded to your question as yet because they are waiting for more system-level performance metrics before rendering an opinion.
QUOTE:
I was wondering if anyone else might have noticed some unusual behavior
I don't see this. To hunt this problem more info and/or experimentation is needed (like superticker said)
* What happens if you use a very simple strategy, or even a (nearly) empty strategy?
* What happens if you use a much smaller DataSet or even a single symbol backtest.
* What happen if you run it on a different PC?
* what happens if you run it of a fresh install?
For a "fresh install" you'd rename your WL installation folder and your user data folder, then download WL (and the optimizers) from wealth-lab.com again.
If there is no new insight from the experiments above,
It could help tremendously if you make the problem causing strategy as small as possible and send it over: rene dot koch at finantic dot de. This would enable me to
a) chsek if it happens here again
b) find out what might be the root cause
Your Response
Post
Edit Post
Login is required