Symbol-by-Symbol-Optimization-improve-speed

It's just literally running a standard optimization one at a time for each symbol, nothing more. So maybe it's performing more poorly because it's consuming more resources as the optimizations progress?

It's possible that's all it is, but I feel like I'm seeing better results when I simply open multiple windows within WL7, one for each symbol, and then just run all of the optimizations simultaneously that way. I would need to do more precise testing to confirm, but it tends to make me suspect two possibilities:

1) Concurrency via multiple threads/cores might be performing non-optimally in SBS
2) Memory might be consumed in much larger amounts during SBS, perhaps as a result of a memory leak or by some difference in the way that results are stored in SBS when compared to a standard optimization

QUOTE:
It's just literally running a standard optimization one at a time for each symbol

Any reason why the optimization doesn't run each symbol in parallel?
If the DataSets have 10 symbols and there are 10 CPU cores, I would like the optimization to utilize the CPU to maximize the optimization performance.
Yes, it consumes 10 times more memory than running one symbol at a time in which case you may introduce an option to specify the maximum number of symbols to optimize at a time.

They do run in parallel.

QUOTE:
Yes, it consumes 10 times more memory than running one symbol at a time in which case you may introduce an option to specify the maximum number of symbols to optimize at a time.

Exactly. You're running out of on-chip processor cache memory. So to speed things up, reduce your memory footprint by scaling back the maximum number of symbols processed at a time.

You should also be running an Intel processor that's at least a 7i so you have as much on-chip cache as possible. You do not need more processor cores or processor speed. You need more on-chip processor cache. Keep that in mind when purchasing your CPU chip.

QUOTE:
They do run in parallel.

Really?

From what I see the debug print, I see StrategyExecutionModes.Strategy running at the beginning in parallel but followed by each series of StrategyExecutionModes.Optimization per symbol at a time.
It looks to me, if the initial execution is running in parallel but the actual optimization seems running one symbol at a time.
The optimization progress bar also shows one symbol at a time.

Yes. I meant the optimizations themselves are running in parallel. Not the symbol by symbol processing. Since the optimizations are already parallel and consuming the cpu cores do you really think there is much benefit to be had by making the outer loop parallel as well? It strikes me as overkill.

I guess it's worth to try.

As far as I tested, one instance of symbol-by-symbol optimization can only reach up to 30%+ CPU usage. Whereas running two instances and three instances can reach 60% and 80% CPU usage respectively.

Also running one instance of symbol-by-symbol optimization three times took 4:09 (1:23 * 3) whereas running three instances of symbol-by-symbol optimization took 2:48.

There is definitely a huge room to improve the parallelism.

What Optimizer are you using and what is the range of parameters? With a smaller parameter range, than yes it would probably help. But my CPU maxes out with typical optimziations. I tagged this topic as a #FeatureRequest so we can tackle the change in due course.

QUOTE:
As far as I tested, one instance of symbol-by-symbol optimization can only reach up to 30%+ CPU usage. Whereas running two instances and three instances can reach 60% and 80% CPU usage respectively.

I think running one instance of symbol-by-symbol optimization manager should be enough (since it should not be the rate determining step). However, one has to ask why that one instance is a bottleneck in the first place? The reason for that bottleneck needs to be investigated (via the Visual Studio Profiler) and fixed.

If it's working correctly, the optimization itself should be the rate determining step and that should be multi-threaded, which it is. But it sounds like the system-by-symbol management is adding an unnecessary delay in how it's dispatching new optimization threads. Perhaps if one preallocates a fixed number of threads (adjustable by the user) at the beginning, and reuses those threads after their previous optimization completes, things would move much faster.

My point is the bottleneck is in the creation of a new thread. The Windows scheduler has major overhead each time a new thread is created, so you want to avoid that. Reuse the old (idling) threads instead.