Is there any way to configure Exhaustive Optimizer on how many parallel optimizations it would run? Just to avoid out of memory errors
Rename
No it's programmed to use a max degree of parallelism one less than the number of CPUs detected on your computer. Are you running into issues? You can always use the Exhaustive Non-Parallel.
QUOTE:
No it's programmed to use a max degree of parallelism one less than the number of CPUs detected on your computer. Are you running into issues? You can always use the Exhaustive Non-Parallel.
It would be nice to make it configurable
But at the moment using Non-Parallel
Thank you
Memory errors, what are they? An OutOfMemory exception may suggest that you're pushing the limits. And rather than doing that you could simply choose a better tool for your task armed with DrKoch's excellent analysis:
KnowHow: Optimizers
KnowHow: Optimizers
It could also signal a flaw in the strategy.
The problem here that modern desktop CPU are usually limited to 128 Gb of RAM and number of cores and threads on CPU grows drastically ower time. So the memory usage per thread drops over time. That's why it would be great to have a configation variable with default value set to number of threads on CPU and an ability to set it by hand for Exhaustive Optimizer.
QUOTE:
and an ability to set it by hand for Exhaustive Optimizer.
Don't use "Exhaustive" for any serious work. It is a waste of life-time, Memory and CPU cycles.
Use "SMAC" instead. Much faster. See link above (KnowHow: Optimizers) for details.
QUOTE:
Don't use "Exhaustive" for any serious work. It is a waste of life-time, Memory and CPU cycles.
I have only 700 permutations with rather big dataset and 1 minute scale.
I have found a way to solve this issue setting environment variable DOTNET_PROCESSOR_COUNT to the desired number of virtual processors. Seams to work fine and limits the count of threads running.
Great solution, and it helps save WL from featuritis 👍
QUOTE:
The problem here that modern desktop CPU are usually limited to 128 Gb of RAM and number of cores and threads on CPU grows drastically over time.
I would pick a processor chip that (1) maximizes the on-chip cache on the processor and (2) minimizes the number of fast cores on the processor to minimize this problem. For WL, the bottleneck is the on-chip cache size of the processor chip.
There's a reason why a processor chip is not included when you buy a motherboard for your workstation. You need to select the processor chip (and its hardware configuration) that best fits your particular need.
Your Response
Post
Edit Post
Login is required