I am trying to optimize 5 parameters of a strategy using the SMAC with the data of the Russel 3000 (current & past). I have the problem that already in the initialization phase, i.e. before the expected optimization time is displayed, the system runs out of memory (256 GB!).
Details:
1) Optimization period 95 months (5410 shares), use of 8 cores at 100%, optimization with regard to APR_IS, 512 optimization steps) -> memory consumption around 22 GB, optimization runs through
2) Optimization period 105 months (5613 shares), use of fewer cores with very low utilization, optimization with regard to APR_IS, 512 optimization steps) -> memory consumption > 256 GB, optimization does not come out of the initialization phase
It looks to me as if a variable, an array or similar is dimensioned too small and thus more and more memory is allocated until the memory runs out. Even after a long wait, the system does not manage to end the initialization phase and start the actual calculation runs -> kill
The Optimizer Random with 105 months (5613 shares), 5 variables, 512 calculation runs, needs approx. 42 GB, unfortunately only 8 cores are used, optimization runs through.
The Optimizer Random with 120 months (5891 shares), 5 variables, 512 calculation runs, needs more than 256 GB already in the initialization phase and does not get out of the initialization phase -> kill
The problem is apparently not strategy-dependent
Details:
1) Optimization period 95 months (5410 shares), use of 8 cores at 100%, optimization with regard to APR_IS, 512 optimization steps) -> memory consumption around 22 GB, optimization runs through
2) Optimization period 105 months (5613 shares), use of fewer cores with very low utilization, optimization with regard to APR_IS, 512 optimization steps) -> memory consumption > 256 GB, optimization does not come out of the initialization phase
It looks to me as if a variable, an array or similar is dimensioned too small and thus more and more memory is allocated until the memory runs out. Even after a long wait, the system does not manage to end the initialization phase and start the actual calculation runs -> kill
The Optimizer Random with 105 months (5613 shares), 5 variables, 512 calculation runs, needs approx. 42 GB, unfortunately only 8 cores are used, optimization runs through.
The Optimizer Random with 120 months (5891 shares), 5 variables, 512 calculation runs, needs more than 256 GB already in the initialization phase and does not get out of the initialization phase -> kill
The problem is apparently not strategy-dependent
Rename
QUOTE:
The problem is apparently not strategy-dependent
If it's not strategy dependent, then tell us what it does depend on. Sounds like you need to do more experimenting. None of the rest of us are seeing this.
Do you have static variables declared? It could be a memory management problem where one execution thread is clobbering the variables of another. Try running the optimization in non-parallel mode to see if that fixes it.
Try running a different optimizer if you think it's the optimizer.
I didn't use any c# code, so I think there should be no static variables.
I alredy used 2 optimizers (SMAC and Random).
I alredy used 2 optimizers (SMAC and Random).
What happens if you use the "Exhaustive" Optimizer?
(this one serves as the "Baseline" implementation)
(You'll need to restrict the parameter ranges to make this work)
(this one serves as the "Baseline" implementation)
(You'll need to restrict the parameter ranges to make this work)
It'll take some time to investigate
Your Response
Post
Edit Post
Login is required