Optimization-stuck-on-huge-intraday-data

2023-09-19T07:58:38Z - 2023-09-19T07:58:38Z ago

Optimization stuck on huge intraday data

Hello!

I noticed this problem when optimizing the strategy:

If I set the period from 01/01/2005 to 12/31/2019, then my computer starts to slow down very much. I can't switch windows, sometimes I can't even move the cursor, and the optimization progress bar doesn't fill up. In order to return to normal operation, you have to restart the computer; pressing the power button does not always help; sometimes you have to physically disconnect the computer from the power supply.

In several cases, the WL closes spontaneously. If I set the period from 01/01/2005 to 12/31/2005 or from 01/01/2019 to 12/31/2019, then everything works well and the optimization will complete successfully in 2-3 minutes.

I am sending you the strategy and settings code, as well as data:

CODE:

using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Data;
using WealthLab.Indicators;
using System.Collections.Generic;
using WealthLab.MyIndicators;


namespace WealthScript1 
{
   public class MyStrategy : UserStrategyBase
   {
      public MyStrategy() : base()
      {
         AddParameter("Stop", ParameterType.Int32, 1000, 1000, 10000, 500);
         AddParameter("Stop Type", ParameterType.Int32, 0, 0, 2, 1); //0-обычный Stop, 1-Trailing Stop, 2-отключение всех Stop.                  
         AddParameter("LongEnabled", ParameterType.Int32, 1, 0, 1, 1); //0-Long на вход выключен. 1-Long на вход включен.
         AddParameter("ShortEnabled", ParameterType.Int32, 1, 0, 1, 1); //0-Short на вход выключен. 1-Short на вход включен.               
         AddParameter("takeProfit", ParameterType.Int32, 1000, 1000, 10000, 100);
         AddParameter("enableTakeProfit", ParameterType.Int32, 0, 0, 1, 1); //0-выход по takeProfit выключен. 1-выход по takeProfit включен.
         AddParameter("periodEnterPC", ParameterType.Int32, 310, 5, 1000, 10);
         AddParameter("percentWidthChannalPC", ParameterType.Int32, 35, 5, 300, 5);
         AddParameter("индикаторEMA", ParameterType.Int32, 20, 5, 300, 5);
         AddParameter("индикаторEMAHour", ParameterType.Int32, 20, 5, 300, 5);
         AddParameter("индикаторEMADay", ParameterType.Int32, 20, 5, 300, 5);
         AddParameter("индикаторEMAHourВключен", ParameterType.Int32, 1, 0, 1, 1);
         AddParameter("индикаторEMADayВключен", ParameterType.Int32, 1, 0, 1, 1);



         StartIndex = 9;
      }

      public override void Initialize(BarHistory bars)
      {

         indicatorOpenHour = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new TimeSeriesIndicatorWrapper(bars.Open), -1), HistoryScale.Minute60), 0);
         indicatorHighHour = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new TimeSeriesIndicatorWrapper(bars.High), -1), HistoryScale.Minute60), 0);
         indicatorLowHour = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new TimeSeriesIndicatorWrapper(bars.Low), -1), HistoryScale.Minute60), 0);
         indicatorCloseHour = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new TimeSeriesIndicatorWrapper(bars.Close), -1), HistoryScale.Minute60), 0);

         indicatorOpenDaily = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new TimeSeriesIndicatorWrapper(bars.Open), -1), HistoryScale.Daily), 1);
         indicatorHighDaily = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new TimeSeriesIndicatorWrapper(bars.High), -1), HistoryScale.Daily), 1);
         indicatorLowDaily = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new TimeSeriesIndicatorWrapper(bars.Low), -1), HistoryScale.Daily), 1);
         indicatorCloseDaily = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new TimeSeriesIndicatorWrapper(bars.Close), -1), HistoryScale.Daily), 1);

         PlotIndicatorBands(indicatorHighHour, indicatorLowHour, WLColor.LightGray, 0, 20, false, "Price");
         PlotIndicatorCloud(indicatorOpenHour, indicatorCloseHour, WLColor.Brown, WLColor.LightGreen, 2, LineStyle.Solid, 25, false, "Price");

         PlotIndicatorBands(indicatorHighDaily, indicatorLowDaily, WLColor.LightGray, 0, 20, false, "Price");
         PlotIndicatorCloud(indicatorOpenDaily, indicatorCloseDaily, WLColor.Red, WLColor.LightGreen, 2, LineStyle.Solid, 10, false, "Price");


         emaMainSymbolHour = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new EMA(bars.Close, Parameters[9].AsInt), -1), HistoryScale.Minute60), 1);

         PlotIndicator(emaMainSymbolHour, new WLColor(0, 0, 255), PlotStyle.Line, false, "Price");

         
         emaMainSymbolDay = new OffsetInd(bars, new ScaledInd(bars, new OffsetInd(bars, new EMA(bars.Close, Parameters[10].AsInt), -1), HistoryScale.Daily), 1);

         PlotIndicator(emaMainSymbolDay, new WLColor(0, 0, 255), PlotStyle.Line, false, "Price");


         longEnabled = Parameters[2].AsInt == 1;
         shortEnabled = Parameters[3].AsInt == 1;

         takeProfit = Parameters[4].AsInt;

         if (Parameters[5].AsInt == 0)
         {
            enableTakeProfit = false;
         }
         else
         {
            enableTakeProfit = true;
         }


         trailing = false;
         PlotStopsAndLimits(3);


         былПробойНижнегоКаналаPCВходLongВТечениеДня = false;

         былПробойВерхнегоКаналаPCВходShortВТечениеДня = false;

         былПробойВерхнегоКаналаPC = false;

         былПробойНижнегоКаналаPC = false;

         индикаторEMA = new EMA(bars.Close, Parameters[8].AsInt);

         PlotIndicator(индикаторEMA, new WLColor(0, 0, 255), PlotStyle.Line, false, "Price");

         openHourMainSymbolByHourNumber = new List<double>();
         highHourMainSymbolByHourNumber = new List<double>();
         lowHourMainSymbolByHourNumber = new List<double>();
         closeHourMainSymbolByHourNumber = new List<double>();

         EMAHourMainSymbolByHourNumber = new List<double>();

         openDayMainSymbolByDayNumber= new List<double>();
         highDayMainSymbolByDayNumber= new List<double>();
         lowDayMainSymbolByDayNumber= new List<double>();
         closeDayMainSymbolByDayNumber= new List<double>();

         EMADayMainSymbolByDayNumber= new List<double>();
         
         номерЧасаСНачалаТеста = 0;


         if (Parameters[11].AsInt == 0)
         {
            индикаторEMAHourВключен = false;
         }
         else
         {
            индикаторEMAHourВключен = true;
         }

         if (Parameters[12].AsInt == 0)
         {
            индикаторEMADayВключен = false;
         }
         else
         {
            индикаторEMADayВключен = true;
         }

         номерДняСНачалаТеста = 0;

      }

      public override void Execute(BarHistory bars, int idx)
      {
         int index = idx;

         if (idx < Parameters[6].AsInt)
         {
            return;
         }

         if (idx > 0 && idx + 1 < bars.Count)
         {
            if (bars.DateTimes[idx + 1].Hour != bars.DateTimes[idx].Hour)
            {
               openHourMainSymbolByHourNumber.Add(indicatorOpenHour[idx]);
               highHourMainSymbolByHourNumber.Add(indicatorHighHour[idx]);
               lowHourMainSymbolByHourNumber.Add(indicatorLowHour[idx]);
               closeHourMainSymbolByHourNumber.Add(indicatorCloseHour[idx]);

               EMAHourMainSymbolByHourNumber.Add(emaMainSymbolHour[idx]);

               номерЧасаСНачалаТеста++;
            }

            if (bars.DateTimes[idx + 1].Day != bars.DateTimes[idx].Day)
            {
               
               openDayMainSymbolByDayNumber.Add(indicatorOpenDaily[idx]);
               highDayMainSymbolByDayNumber.Add(indicatorHighDaily[idx]);
               lowDayMainSymbolByDayNumber.Add(indicatorLowDaily[idx]);
               closeDayMainSymbolByDayNumber.Add(indicatorCloseDaily[idx]);

               EMADayMainSymbolByDayNumber.Add(emaMainSymbolDay[idx]);

               номерДняСНачалаТеста++;
            }
         }

         if (номерЧасаСНачалаТеста < 2)
         {
            return;
         }

         if (номерДняСНачалаТеста < 2)
         {
            return;
         }


         Position foundPosition0 = FindOpenPosition(0);         

         if (longEnabled == true)
         {
            if (foundPosition0 != null)
            {
               выполнилосьУсловиеИндикатораВLong = false;

               if (Parameters[1].AsInt == 0)
               {
                  Backtester.CancelationCode = 360;
                  ClosePosition(foundPosition0, OrderType.Stop, foundPosition0.EntryPrice - Parameters[0].AsInt, "Выход (Long) stop loss");
               }
               else if (Parameters[1].AsInt == 1)
               {
                  Backtester.CancelationCode = 360;
                  CloseAtTrailingStop(foundPosition0, TrailingStopType.PointHL, Parameters[0].AsInt, "Выход (Long) trailing stop loss");
               }

               if (enableTakeProfit == true)
               {
                  ClosePosition(foundPosition0, OrderType.Limit, foundPosition0.EntryPrice + takeProfit, "Выход по takeProfit");
               }
               
               if((EMAHourMainSymbolByHourNumber[номерЧасаСНачалаТеста - 1] < EMAHourMainSymbolByHourNumber[номерЧасаСНачалаТеста - 2] && индикаторEMAHourВключен || !индикаторEMAHourВключен)
                  && индикаторEMA[idx] < индикаторEMA[idx - 1]
                  && (EMADayMainSymbolByDayNumber[номерДняСНачалаТеста - 1] < EMADayMainSymbolByDayNumber[номерДняСНачалаТеста - 2] && индикаторEMADayВключен || !индикаторEMADayВключен))
               { 
                  ClosePosition(foundPosition0, OrderType.Market, 0, "Выход по индикатору PC для Long");
               }
               
            }
            else
            {
               if ((EMAHourMainSymbolByHourNumber[номерЧасаСНачалаТеста - 1] > EMAHourMainSymbolByHourNumber[номерЧасаСНачалаТеста - 2] && индикаторEMAHourВключен || !индикаторEMAHourВключен)
                  && индикаторEMA[idx] > индикаторEMA[idx - 1]
                  && (EMADayMainSymbolByDayNumber[номерДняСНачалаТеста - 1] > EMADayMainSymbolByDayNumber[номерДняСНачалаТеста - 2] && индикаторEMADayВключен || !индикаторEMADayВключен))
               {
                  _transaction = PlaceTrade(bars, TransactionType.Buy, OrderType.Market, 0, 0, "Buy");
               }
            }
         }

         Position foundPosition1 = FindOpenPosition(1);         

         if (shortEnabled == true)
         {
            if (foundPosition1 != null)
            {
               выполнилосьУсловиеИндикатораВShort = false;

               if (Parameters[1].AsInt == 0)
               {
                  Backtester.CancelationCode = 360;
                  ClosePosition(foundPosition1, OrderType.Stop, foundPosition1.EntryPrice + Parameters[0].AsInt, "Выход (Long) stop loss");
               }
               else if (Parameters[1].AsInt == 1)
               {
                  Backtester.CancelationCode = 360;
                  CloseAtTrailingStop(foundPosition1, TrailingStopType.PointHL, Parameters[0].AsInt, "Выход (Long) trailing stop loss");
               }

               if (enableTakeProfit == true)
               {
                  ClosePosition(foundPosition1, OrderType.Limit, foundPosition1.EntryPrice - takeProfit, "Выход по takeProfit");
               }

               if((EMAHourMainSymbolByHourNumber[номерЧасаСНачалаТеста - 1] > EMAHourMainSymbolByHourNumber[номерЧасаСНачалаТеста - 2] && индикаторEMAHourВключен || !индикаторEMAHourВключен)
                  && индикаторEMA[idx] > индикаторEMA[idx - 1]
                  && (EMADayMainSymbolByDayNumber[номерДняСНачалаТеста - 1] > EMADayMainSymbolByDayNumber[номерДняСНачалаТеста - 2] && индикаторEMADayВключен || !индикаторEMADayВключен))
               {
                  ClosePosition(foundPosition1, OrderType.Market, 0, "Выход по индикатору PC для Short");
               }
            }
            else
            {
               if ((EMAHourMainSymbolByHourNumber[номерЧасаСНачалаТеста - 1] < EMAHourMainSymbolByHourNumber[номерЧасаСНачалаТеста - 2] && индикаторEMAHourВключен || !индикаторEMAHourВключен)
                  && индикаторEMA[idx] < индикаторEMA[idx - 1]
                  && (EMADayMainSymbolByDayNumber[номерДняСНачалаТеста - 1] < EMADayMainSymbolByDayNumber[номерДняСНачалаТеста - 2] && индикаторEMADayВключен || !индикаторEMADayВключен))
               {
                  _transaction = PlaceTrade(bars, TransactionType.Short, OrderType.Market, 0, 1, "Short");
               }

            }
         }
      }

      /*
      public override void NewWFOInterval(BarHistory bars)
      {         
         source2 = bars.Close;
         externalSymbol = GetHistory(bars, "RI.RVI_1MIN_150505");
         indicator = new StochK(externalSymbol, Parameters[3].AsInt);
         indicator2 = new StochD(bars, Parameters[6].AsInt, Parameters[7].AsInt);
         source3 = bars.Low;
         wilderMAMainSymbolDaily = new WilderMA(bars.Close, Parameters[10].AsInt);
         source5 = bars.Low;
         source6 = bars.Close;
         indicator4 = new StochD(bars, Parameters[6].AsInt, Parameters[7].AsInt);
         source7 = bars.High;
         wilderMAMainSymbolDaily = new WilderMA(bars.Close, Parameters[10].AsInt);
         source8 = new WilderMA(externalSymbol.Close, Parameters[10].AsInt);

         if (Parameters[22].AsInt == 1)
         {
            indicatorYurPos = new YurPosDivVolumeCurrentInterest(bars);

            indicatorWilderMAYurPos = new IndOnInd(bars, new WilderMA(bars.Close, Parameters[10].AsInt), indicatorYurPos);
         }

         if (Parameters[23].AsInt == 1)
         {
            indicatorFizPos = new FizPosDivVolumeCurrentInterest(bars);

            indicatorWilderMAFizPos = new IndOnInd(bars, new WilderMA(bars.Close, Parameters[10].AsInt), indicatorFizPos);
         }

         if (Parameters[24].AsInt == 1)
         {
            indicatorDOMQuantity = new DOMQuantityBidDivQuantityAsk(bars);

            indicatorWilderMADOMQuantity = new IndOnInd(bars, new WilderMA(bars.Close, Parameters[10].AsInt), indicatorDOMQuantity);
         }

         if (Parameters[25].AsInt == 1)
         {
            indicatorDOMVolume = new DOMVolumeBidDivVolumeAsk(bars);
         }
      }
      */

      private double pct;
      private TimeSeries multSource;
      private double pct2;
      private double val2;
      private TimeSeries source2;
      private TimeSeries multSource2;
      private IndicatorBase indicator;
      private BarHistory externalSymbol;
      private double value;
      private bool trailing;
      private IndicatorBase indicator2;
      private double pct3;
      private double val3;
      private TimeSeries source3;
      private TimeSeries multSource3;
      private double pct4;
      private double val4;
      private IndicatorBase wilderMAMainSymbolDaily;
      private TimeSeries multSource4;
      private double pct5;
      private double longEnterBelowLevel;
      private double shortEnterAboveLevel;
      private TimeSeries source5;
      private TimeSeries multSource5;
      private double pct6;
      private double val6;
      private TimeSeries source6;
      private TimeSeries multSource6;

      private double value2;
      private bool trailing2;
      private IndicatorBase indicator4;
      private double pct7;
      private double val7;
      private TimeSeries source7;
      private TimeSeries multSource7;

      private IndicatorBase source8;

      private double pct8;
      private double val8;
      private TimeSeries multSource8;

      private Transaction _transaction;

      private IndicatorBase indicator5;

      private IndicatorBase indicatorYurPos;

      private IndicatorBase indicatorWilderMAYurPos;

      private IndicatorBase indicatorFizPos;

      private IndicatorBase indicatorWilderMAFizPos;

      private IndicatorBase indicatorDOMQuantity;

      private IndicatorBase indicatorWilderMADOMQuantity;

      private IndicatorBase indicatorDOMVolume;

      private IndicatorBase indicatorWilderMADOMVolume;

      private IndicatorBase indicatorOpenHour;
      private IndicatorBase indicatorHighHour;
      private IndicatorBase indicatorLowHour;
      private IndicatorBase indicatorCloseHour;

      private IndicatorBase indicatorOpenHourSecondSymbol;
      private IndicatorBase indicatorHighHourSecondSymbol;
      private IndicatorBase indicatorLowHourSecondSymbol;
      private IndicatorBase indicatorCloseHourSecondSymbol;

      private IndicatorBase indicatorOpenDaily;
      private IndicatorBase indicatorHighDaily;
      private IndicatorBase indicatorLowDaily;
      private IndicatorBase indicatorCloseDaily;

      private IndicatorBase indicatorOpenDailySecondSymbol;
      private IndicatorBase indicatorHighDailySecondSymbol;
      private IndicatorBase indicatorLowDailySecondSymbol;
      private IndicatorBase indicatorCloseDailySecondSymbol;

      private double wilderMAMainSymbolDailyYesterdayValue;

      private double highDailyMainSymbolYesterdayValue;
      private double lowDailyMainSymbolYesterdayValue;

      private bool tryToEnterLongAllDay;
      private bool tryToEnterShortAllDay;

      private bool tryToCloseLongAllDay;
      private bool tryToCloseShortAllDay;

      bool longEnabled;
      bool shortEnabled;

      bool enterOnlyCandleStart;
      int candleStartNumber;

      DateTime enterTime;

      bool запретитьТорговлюВнутриДняLong;

      bool запретитьТорговлюВнутриДняShort;

      int режимТорговли;

      int выйтиЧерезКоличествоДней;

      bool выполнилосьУсловиеИндикатораВShort;

      bool выполнилосьУсловиеИндикатораВLong;

      int takeProfit;

      bool enableTakeProfit;

      private IndicatorBase индикаторВерхнийКаналPC;

      private IndicatorBase индикаторВерхнийКаналЛюстры;

      private IndicatorBase индикаторНижнийКаналPC;

      private IndicatorBase индикаторНижнийКаналЛюстры;

      private IndicatorBase индикаторEMA;
      
      private IndicatorBase emaMainSymbolHour;

      private IndicatorBase emaMainSymbolDay;

      bool былПробойНижнегоКаналаPCВходLongВТечениеДня;

      bool былПробойВерхнегоКаналаPCВходShortВТечениеДня;

      bool былПробойВерхнегоКаналаPC;

      bool былПробойНижнегоКаналаPC;

      double верхнийКаналPCНаМоментСделки;

      double нижнийКаналPCНаМоментСделки;

      int номерДняСНачалаТеста;

      int номерЧасаСНачалаТеста;

      List<double> openHourMainSymbolByHourNumber;
      List<double> highHourMainSymbolByHourNumber;
      List<double> lowHourMainSymbolByHourNumber;
      List<double> closeHourMainSymbolByHourNumber;
      List<double> EMAHourMainSymbolByHourNumber;

      List<double> openDayMainSymbolByDayNumber;
      List<double> highDayMainSymbolByDayNumber;
      List<double> lowDayMainSymbolByDayNumber;
      List<double> closeDayMainSymbolByDayNumber;
      
      List<double> EMADayMainSymbolByDayNumber;      

      bool индикаторEMAHourВключен;

      bool индикаторEMADayВключен;
         
      }   
}

https://drive.google.com/file/d/1qei-8Ty4mAeKE7UD-mFgKsdqI56rxMw3/view?usp=sharing

1,023

Solved

15 Replies

Bookmark

Notify

Sort

Cone8

( 4.98% )

2023-09-19T08:13:25Z - 2023-09-19T08:13:25Z ago

In short, you've described that when you try to optimize -running 100 permutations on about about 5 million bars - things start to slow down?

And you didn't mention anything about the machine, its memory, or its memory use?

DenisPolezhaka8

2023-09-19T08:45:44Z - 2023-09-19T08:45:44Z ago

Cone, hi!

Here are my computer details:

Cone8

( 4.98% )

2023-09-19T10:07:13Z - 2023-09-19T10:07:13Z ago

And what was the machine's memory state when you experienced the slowdown?

For 100 permutations, you might need twice that amount of memory.

Glitch8

( 10.62% )

2023-09-19T11:14:38Z - 2023-09-19T11:14:38Z ago

It sounds kind you’ve pushed the computer past its limit, try reducing the amount of data being optimized.

Best Answer

DenisPolezhaka8

2023-09-20T06:26:12Z - 2023-09-20T06:26:12Z ago

QUOTE:

For 100 permutations, you might need twice that amount of memory.

Cone, hi!

I don’t understand what the number of permutations has to do with it. After all, one run works great. What prevents the optimizer from running runs one after another in a row?

I think the problem is load balancing across threads. The optimizer may be trying to make multiple runs in parallel, ignoring the amount of computer resources.

Eugene8

2023-09-20T06:31:15Z - 2023-09-20T06:31:15Z ago

Denis,

In the broad sense, the "computer resources" are managed by the NET framework under the hood.

Have you switched to the non-parallel optimizer and what was the outcome?

DenisPolezhaka8

2023-09-20T06:44:51Z - 2023-09-20T06:44:51Z ago

Eugene, hello!

So, is it really possible to switch to a non-parallel optimizer? How to do it?

I'm not suggesting that we abandon the parallel optimizer. I want the optimizer to correctly decide how many threads it needs and balance the load correctly. If my strategy is heavy or the amount of data is large, then perhaps the optimizer should decide that it cannot perform runs in parallel or perform fewer runs in parallel. I think this should work in reverse as well. If computer resources allow, the optimizer should increase the number of runs performed at a time for faster optimization.

Eugene8

2023-09-20T07:24:24Z - 2023-09-20T07:24:24Z ago

I may be wrong but was just suggesting to switch to the Non-Parallel Exhaustive optimizer.

DenisPolezhaka8

2023-09-20T11:43:33Z - 2023-09-20T11:43:33Z ago

Eugene,

I can't use Exhause (non-Parallel) because I have 200000 permutations and the optimization will take about 80 days. But in this mode it really doesn’t freeze.

Glitch8

( 10.62% )

2023-09-20T11:49:28Z - 2023-09-20T11:49:28Z ago

#10

Are you using Shrinking Window? Sounds like a non-parallel Shrinking Window would help you out here.

DenisPolezhaka8

2023-09-21T06:05:49Z - 2023-09-21T06:05:49Z ago

#11

Glitch, hello!

I don't have such a regime.

Eugene8

2023-09-21T08:14:24Z - 2023-09-21T08:14:24Z ago

#12

Denis,

In English (or Roman languages), modal verbs like Glitch's "would" express likelihood, ability, necessity or intent. Such option does not exist yet as you correctly point out.

Glitch8

( 10.62% )

2023-09-21T12:19:22Z - 2023-09-21T12:19:22Z ago

#13

That's right, I was suggesting we add that as a #FeatureRequest.

Eugene8

2023-09-21T15:40:20Z - 2023-09-21T15:40:20Z ago

#14

Done.

Denis, vote for this #FR: https://www.wealth-lab.com/Discussion/Shrinking-Window-optimization-add-Non-Parallel-version-10180

Eugene8

2023-10-19T18:08:34Z - 2023-10-19T18:08:34Z ago

#15

Hey Denis, meet "Shrinking Window (Non-Parallel)" in B51. https://www.wealth-lab.com/Discussion/Shrinking-Window-optimization-add-Non-Parallel-version-10180

Bookmark

Notify

Sort