Does anyone have any tips on how to select the best optimizer for a strategy, in general? Say you have a strategy that examines 4 or 5 different indicators, each of which have 2 or 3 Parameters. Is there a way to gauge when you should use Shrinking Window, or Bayesian, or Particle Swarm, etc?
Any general advice would be appreciated. Thanks!
Any general advice would be appreciated. Thanks!
Rename
In order to answer such a question I recently conducted an interesting experiment:
I took the strategy "A Seventeen Liner" by WL veteran gbeltrame and replaced all constants in the strategies code with optimizable parameters:
This are 21 Parameters and an exhaustive optimization would take an impractical number of runs:
I ran various optimizations on 10 years of Dow30 Data. I requested 900 permutations from each optimizer and chose "APR" as Target metric.
Then I copied the "Tabular Results" to Excel, sorted for APR and put all lines in a single graph:
The Y-Axis shows APR in percent, the optimization goal.
The X-Axis shows number of Runs/Permutations.
Under the assumption that the optimzer algorithms try to achieve better and better APR values the curves from left to right show approximately the
path to success from run to run.
I took the strategy "A Seventeen Liner" by WL veteran gbeltrame and replaced all constants in the strategies code with optimizable parameters:
This are 21 Parameters and an exhaustive optimization would take an impractical number of runs:
I ran various optimizations on 10 years of Dow30 Data. I requested 900 permutations from each optimizer and chose "APR" as Target metric.
Then I copied the "Tabular Results" to Excel, sorted for APR and put all lines in a single graph:
The Y-Axis shows APR in percent, the optimization goal.
The X-Axis shows number of Runs/Permutations.
Under the assumption that the optimzer algorithms try to achieve better and better APR values the curves from left to right show approximately the
path to success from run to run.
The clear winner in such a scenario is the "SMAC" optimizer which reaches good results with 400 to 500 permutations, which is quite remarkable under these circumstances.
Second is "Shrinking Window" closely followed by "Particle Swarm" and "Bayesian".
All three need a significant higher number of permutatiosn to arrive at a parameter combination with results comparable to the "SMAC" optimizer.
Finally, the "Nelder-Mead" optimizer is not very useful in this scenario.
Second is "Shrinking Window" closely followed by "Particle Swarm" and "Bayesian".
All three need a significant higher number of permutatiosn to arrive at a parameter combination with results comparable to the "SMAC" optimizer.
Finally, the "Nelder-Mead" optimizer is not very useful in this scenario.
I would choose x100 larger runs number if it helped to avoid curve-fitting.
Do you have OOS results for all these 5 options?
Do you have OOS results for all these 5 options?
Thank you very very much for this workup. This should definitely be published. And I'm going to have to look deeper into the SMAC algorithm and why it works so much better.
The only question remains, how would these optimizers fair if the number of parameters being optimized was cut in half or something? Would the results be similar or somewhat different?
The only question remains, how would these optimizers fair if the number of parameters being optimized was cut in half or something? Would the results be similar or somewhat different?
Another result that is interesting to look at, is what parameters' values do you have as a result of this different optimizers. It may happen they can be very-very far away from each other.
What I want to say is all of this has nothing close to strategy OOS robustness).
Still nice experiment).
What I want to say is all of this has nothing close to strategy OOS robustness).
Still nice experiment).
Here the same Information as above but this time orderd by "Run Number":
Y-Axis: APR
X-Axis: True Run Number
The SMAC Optimizer needs less than 100 Runs to find good Parameter sets.
The Shrinking Window Optimizer needs about 600 Runs to reach a similar region of good APR values.
Y-Axis: APR
X-Axis: True Run Number
The SMAC Optimizer needs less than 100 Runs to find good Parameter sets.
The Shrinking Window Optimizer needs about 600 Runs to reach a similar region of good APR values.
Thank you very much Dr. Koch for this study! Very interesting findings that I will now take into account.
Where do I find SMAC and particle swarm optimizers?
It's part of finantic.Optimizers:
https://www.wealth-lab.com/extension/detail/finantic.Optimizer
You can find more extensions developed by third parties in the footer:
https://www.wealth-lab.com/Extension
But look forward to build 4 finantic.Optimizers (to be released together with WL build 18 soon) which fixes a bug.
https://www.wealth-lab.com/extension/detail/finantic.Optimizer
You can find more extensions developed by third parties in the footer:
https://www.wealth-lab.com/Extension
But look forward to build 4 finantic.Optimizers (to be released together with WL build 18 soon) which fixes a bug.
I just ran SMAC vs shrinking window resulting in even more thoroughly detailed results (down to 6 decimals) and increased my win/loss percentage by 2% along with a quicker optimization time than shrinking window. Thank you for your info above as I plan on using SMAC.
+2% OOS winrate or IS?)
2% better win rate IS based on best OOS walk forward optimized parameters found. Then I also run a monte carlo simulation on trade order scramble to verify timing of the IS before I started live trading the strategy.
DrKoch, would you be able to share with us the Configuration details you used for SMAC in these tests? When increasing the number of iterations, I'm never sure if I should be adjusting the other settings as well, or if it's fine to leave everything at default settings.
Your Response
Post
Edit Post
Login is required