Hello, DrKoch!
Made a lot of backtests with help of SMAC recently and it works really great BUT I have noticed one thing ...
1. Backtest machine Asus Nitro 5 - AMD Ryzen 9 5900HX 8 Cores 16 Threads
2. Same backtest data
3. Exhaustive vs SMAC
Exhaustive
Total permutations: 7560
CPU Utilization
(60%-70% CPU utilization)
Completition time
SMAC
Total permutations: 1500
CPU Utilization
(20 % CPU utilization)
Completition time
So SMAC has total 5-7 times less permutations but consumes same time to complete...
Made a lot of backtests with help of SMAC recently and it works really great BUT I have noticed one thing ...
1. Backtest machine Asus Nitro 5 - AMD Ryzen 9 5900HX 8 Cores 16 Threads
2. Same backtest data
3. Exhaustive vs SMAC
Exhaustive
Total permutations: 7560
CPU Utilization
(60%-70% CPU utilization)
Completition time
SMAC
Total permutations: 1500
CPU Utilization
(20 % CPU utilization)
Completition time
So SMAC has total 5-7 times less permutations but consumes same time to complete...
Rename
Completition time with SMAC
Your findings are correct.
Let me add a few explanations:
1. Parallel algorithms
The exhaustive optimizer is an algorithm that can run in parallel. Completely. All parameter combinations are known in advance so each run can start on any core.
The SMAC optimizer runs in batches (or phases). For example the first phase runs a number of random parameter combinations. (Controlled by the "Number of Starting Points" meta parameter). Then the algorithm must wait for all these runs to finish before it can enter the next phase. This means if you set "Number of Starting Points" to a number larger than the number of (logical) cores, you'll see a 100% CPU utilization for a few moments followed by lower and lower CPU utilization until all runs finished.
Then SMAC enters the next batch: It choose a few variations of the best combinations found so far (controlled by the "Local Searchpoint Count" meta parameter) and adds some more random combinations (controlled by the "Random Searchpoint Count" meta parameter"). If you choose these numbers in a way that the sum of both is the number of available cores (or more) you'll again see 100% utilization for a few moments and then lower utilization until all runs finished.
This means: With matching meta parameters you can drive your CPU utilization higher. You get more overall iterations without more waiting time. But does it improve the overall result?
In my experience it is more important to let SMAC go through severeal cycles as described above and less important to have gigantic startpoint counts.
2. Algorithms vs. Computing Power
It is old wisdom in computer science that a clecver algorithm outperforms (any amount of) computing power. Same here. If you have more than three parameters to optimize the number of required runs for exhaustive optimization will grow rapidly. And because there are only a few "Steps" for each parameter the results are not very precise. A large effort for mediocre results.
With SMAC you can have a large number of parameters (say 10 to 20) and still only need a medium number of runs (lower than 2000) and it will zero in on some "exact" parameter values. (It ignores the "Step" values.
Conclusion: I'd live with a somewhat lower CPU utilization and enjoy the good results of SMAC.
Let me add a few explanations:
1. Parallel algorithms
The exhaustive optimizer is an algorithm that can run in parallel. Completely. All parameter combinations are known in advance so each run can start on any core.
The SMAC optimizer runs in batches (or phases). For example the first phase runs a number of random parameter combinations. (Controlled by the "Number of Starting Points" meta parameter). Then the algorithm must wait for all these runs to finish before it can enter the next phase. This means if you set "Number of Starting Points" to a number larger than the number of (logical) cores, you'll see a 100% CPU utilization for a few moments followed by lower and lower CPU utilization until all runs finished.
Then SMAC enters the next batch: It choose a few variations of the best combinations found so far (controlled by the "Local Searchpoint Count" meta parameter) and adds some more random combinations (controlled by the "Random Searchpoint Count" meta parameter"). If you choose these numbers in a way that the sum of both is the number of available cores (or more) you'll again see 100% utilization for a few moments and then lower utilization until all runs finished.
This means: With matching meta parameters you can drive your CPU utilization higher. You get more overall iterations without more waiting time. But does it improve the overall result?
In my experience it is more important to let SMAC go through severeal cycles as described above and less important to have gigantic startpoint counts.
2. Algorithms vs. Computing Power
It is old wisdom in computer science that a clecver algorithm outperforms (any amount of) computing power. Same here. If you have more than three parameters to optimize the number of required runs for exhaustive optimization will grow rapidly. And because there are only a few "Steps" for each parameter the results are not very precise. A large effort for mediocre results.
With SMAC you can have a large number of parameters (say 10 to 20) and still only need a medium number of runs (lower than 2000) and it will zero in on some "exact" parameter values. (It ignores the "Step" values.
Conclusion: I'd live with a somewhat lower CPU utilization and enjoy the good results of SMAC.
QUOTE:
Conclusion: I'd live with a somewhat lower CPU utilization and enjoy the good results of SMAC
You are 100% percents right! But maybe there is a way to make some params of SMAC to be autocalculated depending the number of CPU and maybe it will lead to better CPU utilization and there will be less iterations needed for SMAC to find great results. I'm not a specialist in SMAC algorithm, but as I have red from your description there is a way to make it automatical and use resources of PC with better time efficiency.
Your Response
Post
Edit Post
Login is required