Optimization-memory-leak-with-OneEuroFilter

This is NOT a problem with Finantic's Bayesian Optimizer. It occurs with an Exhaustive Optimization also. Please reflect that in the Title.

After 15 minutes:

Thanks Glitch! As usual, "User error!" 😒

That should have solved the problem, but didn't. 😣

CODE:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Data;
using WealthLab.Indicators;
using WealthLab.TASC;

namespace WealthScript1
{

   public class MyStrategy : UserStrategyBase
   {
      //constructor
      public MyStrategy() : base()
      {
         // Add optimization variables
         AddParameter("period", ParameterType.Int32, 13, 4, 50, 1);
         AddParameter("beta", ParameterType.Double, 0.743298894094, 0.1, 0.9, 0.01);
         AddParameter("rocLength", ParameterType.Int32, 7, 1, 25, 1);
      }


      //create indicators and other objects here, this is executed prior to the main trading loop
      public override void Initialize(BarHistory bars)
      {
         StartIndex = 150;

         int period = 2 * Parameters[0].AsInt;
         double beta = Parameters[1].AsDouble;
         int rocLength = Parameters[2].AsInt;

         smoothedPrices = OneEuroFilter.Series(bars.Close, period, beta);
         PlotTimeSeries(smoothedPrices, "OneEuro", "Price", WLColor.Blue, PlotStyle.Line);

         roc = ROC.Series(smoothedPrices, rocLength);
         PlotTimeSeries(roc, "ROC", "ROC", WLColor.Blue, PlotStyle.Line);
      }

      //execute the strategy rules here, this is executed once for each bar in the backtest history
      public override void Execute(BarHistory bars, int idx)
      {
         if (!HasOpenPosition(bars, PositionType.Long))
         {
            if (smoothedPrices[idx] > smoothedPrices[idx - 1])
            {
               Transaction t = PlaceTrade(bars, TransactionType.Buy, OrderType.Market);
               t.Weight = roc[idx];
            }
         }
         else
         {
            if (smoothedPrices[idx] < smoothedPrices[idx - 1])
               PlaceTrade(bars, TransactionType.Sell, OrderType.Market, 0, "Normal");
         }
      }

      //declare private variables below
      private TimeSeries smoothedPrices;
      private TimeSeries roc;
   }

}

Start of Optimization


After 15 minutes:

I looked at the OneEuroFilter code and it looks squeaky clean. In any case, you are using Series calls for OneEuroFilter and ROC. And, for exhaustive optimization, you've got over 95,000 permutations. The Series calls end up caching the resulting TimeSeries for each permutation. Hence, the memory is just going to keep going up and up and up. So, you might want to change those Series calls to use new (constructor). Of course there are trade-offs. Using new will cause many recalculations, and some more garbage collection, but less overall memory use (I tested it). I don't know what the impact of the overall time is because I didn't want to let it run for 17 hours. :)

So, in Initialize(...) try this...

CODE:
        smoothedPrices = new OneEuroFilter(bars.Close, period, beta);
//...
        roc = new ROC(smoothedPrices, rocLength);

Thanks that's spot on Paul, using new instead of Series should avoid the memory bloat at the expense of much more garbage collection and increased run times.

paul1986,

Thanks for the very helpful tip! I sure do hope that resolves the issue. I will test it now.

That seems to have solved the problem. Thanks paul1986 and Glitch for all the help!

QUOTE:
Using the new [operator] will cause many recalculations, and some more garbage collection, but less overall memory use.... I don't know what the impact of the overall time is ...

That's an interesting question, and the answer will vary with your hardware situation.

What I would do is cache the indicator (with the .Series function) whenever just one of its parameters is being varied (or optimized). But if there are more than one indicator parameters changing, then I wouldn't cache it; use the new operator instead.

The problem is many indicators have an input "source" TimeSeries that may be changing, so if that indicator has one other parameter that changes, then you may not want to cache it.

When I build my strategies, I try limiting the number of parameters between 4 to 6 using not more than 4 or 5 steps. Having many permutations is not helpful.