I just finished an Walk Forward Optimization run with the latest version of the SMAC optimizer.
I added a "runParallel" option to the SMAC optimizer (and the necessary code changes).
Now the WFO came down form the usual 6 hours to just 3 hours on my good old four-core PC - quite an improvement.
This will be available with the next build (Build 2) of finantic.Optimizers.
Here is the link: https://wealth-lab.com/extension/detail/finantic.Optimizer
I added a "runParallel" option to the SMAC optimizer (and the necessary code changes).
Now the WFO came down form the usual 6 hours to just 3 hours on my good old four-core PC - quite an improvement.
This will be available with the next build (Build 2) of finantic.Optimizers.
Here is the link: https://wealth-lab.com/extension/detail/finantic.Optimizer
Rename
Awesome Ive been waiting for a faster option for WFO!!
Is there a way to optimize this Extension for SMAC Optimizer?
Check out the difference between SMAC and Shrinking Window
Can you tell me why the SMAC Optimizer isn't using all of my CPUs to the fullest? It takes forever for it to run
Shrinking Window
SMAC Optimizer
Check out the difference between SMAC and Shrinking Window
Can you tell me why the SMAC Optimizer isn't using all of my CPUs to the fullest? It takes forever for it to run
Shrinking Window
SMAC Optimizer
There's been some WL8 Build 19 code changes that have affected the SMAC optimizer. See the thread https://www.wealth-lab.com/Discussion/WL8-Build-19-Optimizations-running-slow-8752
Now the next question is whether or not the SMAC optimizer will need to be updated after these code changes? I don't know. But if the shrinking window optimizer is using all the cores after Build 19 and the SMAC optimizer isn't, then it sounds like the SMAC optimizer needs an update. But this is only my "guess".
One can optionally downgrade to WL8 Build 18 to get finantic.Optimizers Build 1 (SMAC optimizer) working again.
Now the next question is whether or not the SMAC optimizer will need to be updated after these code changes? I don't know. But if the shrinking window optimizer is using all the cores after Build 19 and the SMAC optimizer isn't, then it sounds like the SMAC optimizer needs an update. But this is only my "guess".
One can optionally downgrade to WL8 Build 18 to get finantic.Optimizers Build 1 (SMAC optimizer) working again.
Exactly this is my concern
SMAC has not been update to the latest version
SMAC has not been update to the latest version
There's essentially no change that should impact the SMAC optimizer as of Build 20. Are you sure it was any faster in Build 18? It's confirmed that Exhaustive and Shrinking Window are maxing out all CPUs so I can't understand why the SMAC optimizer would be suffering.
I'm not sure of the SMAC optimizer is even parallel enabled, perhaps it needs to use the feedback from one run to determine how to adjust the parameters of the next run?
I'm not sure of the SMAC optimizer is even parallel enabled, perhaps it needs to use the feedback from one run to determine how to adjust the parameters of the next run?
Basically, I'm asking the developer (DrKoch) to make this extension parallel and use all CPUs
Currently, it takes forever & I started using it after seeing the article in S&C Magazine.
Currently, it takes forever & I started using it after seeing the article in S&C Magazine.
QUOTE:
I'm not sure of the SMAC optimizer is even parallel enabled,
You may be right. I'm running WL8 Build 18 with finantic.Optimizers Build 1, and I see one core being primarily utilized with the other cores barely working. So it looks like Build 1 isn't doing parallel execution, and that's normal.
For parallel execution, you'll have to wait for finantic.Optimizers Build 2.
If you need parallel execution, try the Shirking Window optimizer instead.
Depending on how it’s designed parallel might not even be possible.
What if it requires the results of the first run in order to adjust the parameters of the second? Can’t do that in parallel!
What if it requires the results of the first run in order to adjust the parameters of the second? Can’t do that in parallel!
QUOTE:
What if it requires the results of the first run in order to adjust the parameters of the second?
If the calculation process can be broken up into stages (steps), you can do pipelining with each core responsible for a different stage of the pipelined calculation. That's how floating-point, DSP, and especially vector processors work.
And if you're optimizing an entire dataset, you could assign one symbol to each core until you run out of cores (or cache memory) to optimize multiple symbols (in parallel) at a time.
Is there a way we can get in touch with the extension's developer?
He's away travelling. He'll respond when he's back from his trip.
Build 2 of finantic.Optimizers is now available:
https://www.wealth-lab.com/extension/detail/finantic.Optimizer
This build enables multithreading for the SMAC Optimizer.
The SMAC Optimizer runs a few backtests (in parallel) then builds a model based on all available backtest results. Then it uses the model to find the next generation of candidates. Then the cycle starts again.
Because of this sequential behavior it is:
1. a good idea to set the parameter "Function Evaluations per Iteration" to the number of available cores (or slightly higher). (This is the new default value)
2. In every cycle CPU usage will start close to 100% but will get lower towards the end of the cycle when SMAC waits for all parallel backtests to finish.
So please don't expect a 100% CPU usage all the time. This is simply not possible because of the nature of the SMAC algorithm.
After all the SMAC will find good parameter combinations quite quick (in number of backtest executions AND in wall-clock time). If there are enough parameters to optimize it will outperform all other optimizer algorithms available for WL.
https://www.wealth-lab.com/extension/detail/finantic.Optimizer
This build enables multithreading for the SMAC Optimizer.
The SMAC Optimizer runs a few backtests (in parallel) then builds a model based on all available backtest results. Then it uses the model to find the next generation of candidates. Then the cycle starts again.
Because of this sequential behavior it is:
1. a good idea to set the parameter "Function Evaluations per Iteration" to the number of available cores (or slightly higher). (This is the new default value)
2. In every cycle CPU usage will start close to 100% but will get lower towards the end of the cycle when SMAC waits for all parallel backtests to finish.
So please don't expect a 100% CPU usage all the time. This is simply not possible because of the nature of the SMAC algorithm.
After all the SMAC will find good parameter combinations quite quick (in number of backtest executions AND in wall-clock time). If there are enough parameters to optimize it will outperform all other optimizer algorithms available for WL.
WOW I will give it a try now on a strategy with a lot of Parameters
This looks more or less like expected. You may experiment with the "Evaluations per Iteration" parameter. What happens if you set this to 32 (twice the number of cores of your monster PC)?
Of course then you should set the number (overall) Iterations to 20*32 or more to give SMAC enough cycles to find its way...
And don't forget to observe the "quality of optimization results"
best seen in the "Parameters and Metrics" graph.
Of course then you should set the number (overall) Iterations to 20*32 or more to give SMAC enough cycles to find its way...
And don't forget to observe the "quality of optimization results"
best seen in the "Parameters and Metrics" graph.
This is my current setting as I play around with it - I hope it make sense
I'd set "Evaluations per Iteration" to 16 or 20. (The number of cores in your PC or slightly more)
Are you sure you need 90000 Iterations?
You should get similar results with just 1000!
Are you sure you need 90000 Iterations?
You should get similar results with just 1000!
I was familiarizing/playing with evolvers and optimizers last night. @kls06541
Couldn't believe how fast it processed, including the shrinking window.
I may have to purchase the Finantic Optimizer extension. Far too many metrics that I am not familiar with however. Would welcome a webinar or paper from @DrKoch on the subject. Particularily what a good strategy should look like when optimized.
Couldn't believe how fast it processed, including the shrinking window.
I may have to purchase the Finantic Optimizer extension. Far too many metrics that I am not familiar with however. Would welcome a webinar or paper from @DrKoch on the subject. Particularily what a good strategy should look like when optimized.
What is your computer specifications?
Yes I wish there should be something more to read about it
Yes I wish there should be something more to read about it
QUOTE:
Shrinking Window is finding better results than SMAC Optimizer, but it takes forever to run.
There will always be a trade-off between speed and quality with any optimizing algorithm. That's to be expected.
For stock screening, it's normal to go with a faster optimizer (and settings) to get a quick answer if a given stock is a good fit for a given strategy.
For production runs, it's normal to go with a slower optimizer (and settings) to find more exacting parameters for a stock and its strategy.
Bottom line, there's a reason why several different optimizers (and setting for them) are offered in WL. Also, the fewer parameters to optimize the faster and more reproducible the results will be.
Process Explorer from Microsoft
https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer
https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer
@Dr Koch
I've been wondering why Ryzen 9, 16 core processor takes 5 minutes or more for one parameter with current build 2.
I've been wondering why Ryzen 9, 16 core processor takes 5 minutes or more for one parameter with current build 2.
There is not enough information.
* How long takes a single backtest?
* How many iterations did you request from SMAC optimizer?
Please run:
* a known strategy
* a reasonable number of iterations (200-1000)
* more than two parameters
* report dataset properties (how many symbols, how many bars, Daily?)
* add snapshot of task manager performance view (like above)
Report the time it needs to complete, compare to Exhaustive and Shrinking window.
* How long takes a single backtest?
* How many iterations did you request from SMAC optimizer?
Please run:
* a known strategy
* a reasonable number of iterations (200-1000)
* more than two parameters
* report dataset properties (how many symbols, how many bars, Daily?)
* add snapshot of task manager performance view (like above)
Report the time it needs to complete, compare to Exhaustive and Shrinking window.
The 2nd part article is missing in the Current December Edition of S&C Magazine
QUOTE:
2nd part article is missing
This part is still in the review process.
Thanks @Cone
--------------------------------------------------------
@DrKoch
As far as my testing goes, Shrinking Window is finding better results than SMAC Optimizer, but it takes forever to run.
Would you be so kind as to tell me why you think that SMAC Optimizer is better?
Maybe we can do a Zoom call?
--------------------------------------------------------
@DrKoch
As far as my testing goes, Shrinking Window is finding better results than SMAC Optimizer, but it takes forever to run.
Would you be so kind as to tell me why you think that SMAC Optimizer is better?
Maybe we can do a Zoom call?
You’re giving the Shrinking Window like 40,000 permutations, right? Are you giving SMAC something similar?
Of course it's the same
When running an optimization such as SMAC, as long as the number of iterations are the same, should the "RunParallel" checkbox make any difference in the overall end results? A quick test I just did showed a small difference. I imagine this might become increasingly pronounced with larger parameter sets and iterations.
If this is a known and intended outcome, I'm curious how much better parallel optimizations could potentially be than non-parallel optimizations. I had been avoiding "RunParallel" to try and not overload my system's RAM, but now I'm worried I've been getting suboptimal results from doing this.
If this is a known and intended outcome, I'm curious how much better parallel optimizations could potentially be than non-parallel optimizations. I had been avoiding "RunParallel" to try and not overload my system's RAM, but now I'm worried I've been getting suboptimal results from doing this.
If you're not getting the same results with parallel runs verses single-symbol runs, then your C# code is messed up somehow. When you run parallel, WL forks off (spawns) a separate execution thread for each symbol. If those symbols are sharing some resource (say you declared a static variable that all the threads are reading and writing to), then one thread will clobber that resource when other ones are using it and you'll have a mess.
For this above example, you want to avoid all static variable declarations that more than one symbol might be using (sharing) at the same time during a multi-threaded optimization.
Setup your declarations and C# design so all resources are exclusively allocated to each strategy thread (per symbol) and not shared.
There are special exceptions. For example, perhaps your strategy needs to initialize some data structure (say read from disk) for all the other symbols to use. In this case, have the first symbol initialize that structure as the "writer" and have all the other symbols reference that structure as "readers". It's okay to have many readers share a resource provided there's only one initializing writer.
For this above example, you want to avoid all static variable declarations that more than one symbol might be using (sharing) at the same time during a multi-threaded optimization.
Setup your declarations and C# design so all resources are exclusively allocated to each strategy thread (per symbol) and not shared.
There are special exceptions. For example, perhaps your strategy needs to initialize some data structure (say read from disk) for all the other symbols to use. In this case, have the first symbol initialize that structure as the "writer" and have all the other symbols reference that structure as "readers". It's okay to have many readers share a resource provided there's only one initializing writer.
The SMAC optimizer has some random components, for example it selects the first N (usually 30) parameter sets using a random number generator.
It uses the same seed every time, so the sequence of random numbers generated should be the same as long as you use the same (selectable) seed.
You might run the same optimization with different seeds to see what differences this randomness produces.
Now, when we switch from sequential to parallel, it is possible that the sequence of random numbers becomes different because these threads start and an end in a different order than with non-parallel execution.
To make a long story short: Parallel execution might introduce small variations in results, similar to choosing another seed for the random number generator.
And, as superticker mentioned above, if there are large differences, look for global, static and shared variables in your coded strategy.
You could do a cross check with Exhaustive and Exhaustive non parallel. Your strategy should produce exactly the same results with both versions of exhaustive. If not you should revise your code.
See BLOG article "Anatomy of a WL Strategy": https://www.wealth-lab.com/blog/anatomy-of-a-wl7-strategy
It uses the same seed every time, so the sequence of random numbers generated should be the same as long as you use the same (selectable) seed.
You might run the same optimization with different seeds to see what differences this randomness produces.
Now, when we switch from sequential to parallel, it is possible that the sequence of random numbers becomes different because these threads start and an end in a different order than with non-parallel execution.
To make a long story short: Parallel execution might introduce small variations in results, similar to choosing another seed for the random number generator.
And, as superticker mentioned above, if there are large differences, look for global, static and shared variables in your coded strategy.
You could do a cross check with Exhaustive and Exhaustive non parallel. Your strategy should produce exactly the same results with both versions of exhaustive. If not you should revise your code.
See BLOG article "Anatomy of a WL Strategy": https://www.wealth-lab.com/blog/anatomy-of-a-wl7-strategy
Your Response
Post
Edit Post
Login is required