WL8-memory-management

The release of memory is under the control of the .NET garbage collector. Just closing the window won’t cause .NET to perform a garbage collection. It does it when it decides it’s necessary.

And yes sticking to non parallel optimizations may help decrease memory consumption, how much depends on how many unique indicators might be created in each strategy run.

QUOTE:
The release of memory is under the control of the .NET garbage collector. Just closing the window won’t cause .NET to perform a garbage collection. It does it when it decides it’s necessary.

I can't speak specifically about .NET 8.X, but most run-time systems initiate garbage collection on a large memory allocation request; that is, a "new" operator request. This is why opening a new strategy window (which allocates new memory) takes extra time.

There might be a .NET 8.X CLR run-time system call to force a garbage collection. Have you ask about that on Stack Overflow?

What you should do is write your code to minimize memory allocation and re-allocations so the need for garbage collection is minimized. For example, try to do more work in stack memory rather than heap memory. Or use StringBuilder to assemble your string variables.

Also, 0th-generation garbage collection does not cause the garbage collector to move data, so try to create temporary local variables that can be garbage collected in this way.

Be sure to buy a processor chip that has as much on-chip L3 cache memory as possible. That will make a big difference. Happy computing to you.

It is possible to force the garbage collector to run:

CODE:
          
    // DEBUG
    DoEvents();
    System.GC.Collect(GC.MaxGeneration, GCCollectionMode.Aggressive, true, true);

It is recommended to do this in a debug session only!

From what I've seen so far there is an "intended" memory leak with BarHistory's cache.
Usually all indicators calculated for a symbol are stored in this cache which speeds things up because the same indicator is never calculated twice.

On the other hand, if an optimization creates many variations of many indicators, these indicators remain in these caches (visible as memory consumption).

If there is a large portfolio, a complex strategy (many indicators) and a long optimization run (many parameter combinations) this may lead to memory problems.

As a last resort I did this (pseudo code):

CODE:
           
           foreach (string symbol in listOfActiveSymbols)
           {
               BarHistory bh = GetBarHistory(symbol);
               bh.ClearCache();
           }

Such a draconian method brings memory footprint of WL down again.

Probably WL should do something similar at the end of each optimization run and evolver run and so forth.

You can use System.GC class Collect methods to force garbage collection: https://learn.microsoft.com/en-us/dotnet/api/system.gc?view=net-8.0

But, you can't just pound away at calling a GC.Collect method as that will most likely kill performance. You have to be judicious in its use, perhaps calling it every x executions at some part of your code. Just don't hammer away at it.

Another thing you can do, if you haven't already done this, is the judicial use of Series to create indicators instead of using new. For example:

CODE:
        // instead of this...
        //_rsi = new RSI(bars.Close, 4);
        // use this...
        _rsi = RSI.Series(bars.Close, 4);

What the Series method does is cache the resulting indicator, according to its parameter values (inputs), in the BarHistory for later re-use. When an indicator has been cached in the BarHistory cache, then a subsequent call will retrieve it from the cache instead of re-creating it.

BarHistory is re-used, alot (per symbol) in single strategy runs and optimization runs from my examination using a test strategy (not published here.) And, if you change a strategy setting, like data range, then the bar history is not reused. Also, changing others would probably have the same effect, but obviously I didn't try them all. But, considering you're doing an optimization run, you're not going to be changing strategy settings (except the parameter values which are done by the optimazation runner, not you.)

You can try using Series to see if that helps, but you're dealing with large datasets and whatever data ranges, and indicators, and parameters and other variables. The complexity is hardly practically predictable, so I would just try it out and see if it helps.

I suggest using Series for indicators that don't use parameter values of your strategy. That keeps the caching somewhat limited, especially during optimization runs, which may be required to avoid filling up memory with too many cached indicators. The benefit is you're not recreating data over and over and then throwing the baby out with the bath water on each strategy run of an optimization for any indicator that uses the same set of values over and over.

And, do yourself a favor, if you do convert some calls to using Series instead of new then comment-out the new call and add another line for the Series call. That way if you try various combinations of the use of new and Series, you can easily flip them back and forth with comments.

And, there is no cache manager for the BarHistory cache, so hence you infer you need to use it in a limited manner.

QUOTE:
I suggest using Series for indicators that don't use [optimizer] parameter values in your strategy.

I was going to suggest the same thing; otherwise, each time an indicator parameter changes, you'll be caching another version of the indicator.

For indicators where the optimizer is changing (optimizing) the indicator parameters, it's probably best to use the "new" operator (no caching) instead of the .Series method (with caching).

Exactly where is the ClearCache() method documented? I can't find it anywhere. Is it a secret feature?

CODE:
BarHistory bh = GetBarHistory(symbol);
bh.ClearCache();

QUOTE:
Is it a secret feature?

Well, its is a public member method of BarHistory, which is shown with Code Editor's (or Visual Studios) code completion feature. (enter a dot behind a BarHistory like bars. )

Let's say it is public but undocumented in WL's Quick Ref system.

I think you should not use this in a strategy. But it may make sense in an extension that calls StrategyRunner.

Cache is just a public member of type ConcurrentDictionary that exists in TimeSeriesBase. (Btw... the documentation is wrong about the type being Dictionary because Intellisense shows its type as ConcurrentDictionary, which is what it should be anyhow for multi-threaded operations.)

So, you can just call bars.Cache.Clear() if you like.