Hello, DrKoch
As I have seen on forum SMAC has a lot of parametes
Having seen this screenshot I have some questions regarding SMAC:
1. Is there any article available describing these parameters?
2. Should I live them default?
3. Are there some "best practices" on using these parameters from your own experience?
As I have seen on forum SMAC has a lot of parametes
Having seen this screenshot I have some questions regarding SMAC:
1. Is there any article available describing these parameters?
2. Should I live them default?
3. Are there some "best practices" on using these parameters from your own experience?
Rename
Short answer: Choose 2.) just leave the settings at their default value. I never changed these settings and always got good results.
You'll use Iterations and Metric Name of course.
You'll use Iterations and Metric Name of course.
Long answer: Setting these parameters in a sensible way requires deep understanding of the SMAC algorithm.
The implementation of SMAC in the finantic.Optimizer extension is based on this paper:
7268-11-LION5-SMAC-pdf.
It covers all the details.
Two parameters come from the WL implementation and are not mentioned in the paper:
Seed - Seed of random number generator. Use this to get various versions of "repeatable randomness", i.e. an optimization with the same seed (and no NSF positions) will always yield the same results.
Run Parallel - use as many threads as meaningful with the current CPU configuration, for example eight threads with eight cores. This applies to the individual stages of the SMAC algorithm only however, because all results from one stage are required for the next stage. So you'll not see a 100% CPU utilization for the whole optimization run.
The implementation of SMAC in the finantic.Optimizer extension is based on this paper:
7268-11-LION5-SMAC-pdf.It covers all the details.
Two parameters come from the WL implementation and are not mentioned in the paper:
Seed - Seed of random number generator. Use this to get various versions of "repeatable randomness", i.e. an optimization with the same seed (and no NSF positions) will always yield the same results.
Run Parallel - use as many threads as meaningful with the current CPU configuration, for example eight threads with eight cores. This applies to the individual stages of the SMAC algorithm only however, because all results from one stage are required for the next stage. So you'll not see a 100% CPU utilization for the whole optimization run.
Thank you, DrKoch
And is there any result metric of the SMAC to decide if I used enough iterations or should I use more and run it "additionally" or run it one more time from the start?
And is there any result metric of the SMAC to decide if I used enough iterations or should I use more and run it "additionally" or run it one more time from the start?
QUOTE:
to decide if I used enough iterations
Use the "Parameters and Metrics" Optimization Results Visualizer.
Choose your Target Metric in the lower "Metric" panel and Order By "Metric1"
This will produce a plot where your target metric (example: APR) gets better from left to right.
If this curve looks like it could rise further, you need more iterations.
If this curve reaches a flat top, it is unlikely that more iterations will produce some better result.
Also: It makes not much sense to use too may iterations (like more than 2000) because the results will be unstable, i.e. these "peaks" in parameter space will not repeat in the future.
SMAC will find "stable" parameter regions with 500 to 1000 iterations.
Hello, Dr Koch
Finally bought the plugin and starting using it. I have one more question. With SMAC I made 500 iterations and on ~450 iteration APR started to rise drastically. Is there any way I can resume the optimization from 500 iterations point? Or I just need to increase the iterations count and start from the beginning every time?
*** it’s not very nice to start over when there weren’t enough iterations
Finally bought the plugin and starting using it. I have one more question. With SMAC I made 500 iterations and on ~450 iteration APR started to rise drastically. Is there any way I can resume the optimization from 500 iterations point? Or I just need to increase the iterations count and start from the beginning every time?
*** it’s not very nice to start over when there weren’t enough iterations
Some kind of intelligence would be great to allow it to continue until it stops improving!
Actually not kind of intelligence, just a way to continue optimization by hand - add 200 optimizations more not to start from the very beginning every time...
And SMAC is really fantastic - saves lots of time. Thank you, DrKoch! Enormous value!
QUOTE:
SMAC is really fantastic
Thanks for the kind words!
QUOTE:
Thank you, DrKoch! Enormous value!
Please keep in mind: SMAC was invented by Frank Hutter et.al. (see PDF above)
I just made it available/usable within WL.
QUOTE:
Is there any way I can resume the optimization from 500 iterations point?
Yes, WL already has a builtin stop/resume logic.
Just think the other way round: Start the optimizer with a large number of iterations. Then after some time click "Interim Update" and open the Tabular results.
Sort your target metric column and see if the values reach some saturation.
If you see no strong improvements anymore you might stop the optimization and inspect all results more closely.
Alternatively you might click "Cancel" check results and click "Resume" if you feel you need more iterations.
You'll find yourself starting several optimizer runs for the same problem/strategy/project, so after a few iterations you'll be pretty sure what a good number of iterations will yield (very) good results with the lowest possible waiting time.
We are talking about minutes here, not hours... ;)
Your Response
Post
Edit Post
Login is required