How-to-do-calculations-for-a-symbol-list-as-each-symbol-is-processed

SymbolString is a property we use internally and not intended for that purpose. You should be able to use the BacktestData property to get all the BarHistories in the Strategy.

Thanks Superticker and Glitch, very helpful! So the BacktestBegin will only execute once. That will work for me.

I think I am going to build out this code in this thread, it's a strategy that has been published elsewhere and I have some results in python that are encouraging so will share as i make all my mistakes in trying to implement this.

QUOTE:
So the BacktestBegin will only execute once.

That's correct. Now if you want to execute your code for each individual bar, on a bar-by-bar basis, then you should be using PreExecute instead. See the PreExecute example. https://www.wealth-lab.com/Support/ApiReference/UserStrategyBase#PreExecute

Some background: the strategy i am working on is presented in the paper linked below:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4964417

To summarize, it trades 4 commodity etf's and uses a simple momentum strategy. the unique aspect of this strategy is that it uses the correlation between the etfs in a short timeframe and longer timeframe as a filter or switch.

Here are backtest results in python. They are not that great but it's a nice stable equity line and there is much potential here for optimization.

I am currently working on building the correlation calculations. I have the short term correlation stored as UserData but have not yet added the long term correlation which used 252 days. I am somewhat stcuk here, there seems to be only one slot for UserData so how could i store both the long term and short term correlation in UserData? Or should i take an entirely different approach?
Thank you for any advice :)

CODE:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Indicators;
using System.Collections.Generic;

namespace WealthScriptCorrelation
{
   public class MyStrategy : UserStrategyBase
   {
      //the list of symbols that we should buy each bar
      private static List<BarHistory> buys = new List<BarHistory>();

      //create the weight indicator and stash it into the BarHistory object for reference in PreExecute
      public override void Initialize(BarHistory bars)
      {
         // Compute correlation with SPY and store in cache
         corr1 = new Corr(bars, "SPY", PriceComponent.Close, 20);
         bars.Cache["corr1"] = corr1;
      }

      // This is called before Execute, calculating correlation for each symbol in participants
      public override void PreExecute(DateTime dt, List<BarHistory> participants)
      {
         double totalCorrelation = 0.0;
         int count = 0;
         int numSymbols = participants.Count;

         // Iterate through all unique symbol pairs and compute correlation
         for (int i = 0; i < numSymbols - 1; i++)
         {
            for (int j = i + 1; j < numSymbols; j++)
            {
               BarHistory bh1 = participants[i];
               BarHistory bh2 = participants[j];

               int idx = GetCurrentIndex(bh1);
               if (idx < 0) continue;  // Skip if invalid index

               // Compute correlation between bh1 and bh2
               Corr corr = new Corr(bh1, bh2, PriceComponent.Close, 20);
               if (idx >= corr.Count) continue;  // Ensure valid correlation index

               double corrVal = corr[idx];

               // Sum all correlation values and count the number of pairs
               totalCorrelation += corrVal;
               count++;

               
               WriteToDebugLog("Comparing: " + bh1.Symbol + " vs " + bh2.Symbol + " Correlation= " + corrVal);
            }
         }

         // Compute the global average correlation
         double avgCorrelation = (count > 0) ? totalCorrelation / count : 0.0;
         WriteToDebugLog("dt= " + dt + " Global avgCorrelation= " + avgCorrelation);

         // Store the same global average correlation in all symbols' UserData
         foreach (var bh in participants)
         {
            bh.UserData = avgCorrelation;
            WriteToDebugLog("Symbol= " + bh.Symbol + " Stored avgCorrelation= " + avgCorrelation);
         }
      }



      //execute the strategy rules here, this is executed once for each bar in the backtest history
      public override void Execute(BarHistory bars, int idx)
      {
      
      }

      //declare private variables below
         private Corr corr1;
   }
}

QUOTE:
there seems to be only one slot for UserData so how could i store both the long term and short term correlation in UserData?

The UserData "Tag" can also accept a reference (i.e. pointer) to any C# object. So you can declare a "struct" in your MyStrategy class and then create an instance of that struct with the new operator, and attach its reference to the UserData Tag. So first you do the declaration:

CODE:
   public class MyStrategy : UserStrategyBase
   {
      struct Correlations //long and short term stock correlations
      {
         public double shortTerm;
         public double longTerm;
      }

      public MyStrategy()
      {
         . . .
      }

Now that's just a Correlations declaration without storage in the heap memory. You'll need to employ the "new" operator to get storage for your instance.

CODE:
Correlations correl = new Correlations(); //create a storage instance

Now you've created your instance with storage, so you can fill it by value, then assign that instance reference to the UserData Tag (boxing it there).

CODE:
correl.shortTerm = 0.5678;
correl.longTerm = 0.789;
bh.UserData = correl;

Now you're going to ask, "How do I unbox it?" You would do so like any other boxed object. So the compiler has no idea what kind of datatype you referenced to UserData (unless the WL developers created a virtual UserData type [i.e. real-time binding], which they didn't; that's a discussion for Stack Overflow), so you need to cast it back to Correlations to get that datatype back (unboxing it).

CODE:
Correlations correlBack = (Correlations)bh.UserData; //unbox it with a cast to Correlations
WriteToDebugLog(correlBack.shortTerm);
WriteToDebugLog(correlBack.longTerm);

I may have made a few mistakes. Please correct them for me. Check your C# book for an unboxing example.

This topic needs to be split into two with the latter title being "Employing multiple merit variables in PreExecute," which is weird because one typically would fold the multiple merit variables into a single variable. I'm not sure why we aren't doing that.

I think I am doing something horribly wrong with this code. The code runs very slow.. i have a watchlist of four etf's (DBA, DBB, DBE, DBP) and when I run it for 10 years it takes approximately 10 minutes to execute.. Is there something i could do to speed up execution?

CODE:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Data;
using WealthLab.Indicators;
using System.Collections.Generic;

namespace WealthScript1 
{
   public class MyStrategy2 : UserStrategyBase
   {
      public MyStrategy2() : base()
      {
         StartIndex = 252;
      }

      public override void Initialize(BarHistory bars)
      {
         // Initialize and store indicators
         indicator = new ROC(bars.Close, 252);
         monthIndicator = new MnthOfYr(bars.Close);
         
         // set number of longs/shorts
         Threshold = BacktestData.Count / 2;
   
         // Cache indicators for later use
         bars.Cache["indicator"] = indicator;
         bars.Cache["monthIndicator"] = monthIndicator;
      }

      public override void PreExecute(DateTime dt, List<BarHistory> participants)
      {
         double totalCorr20 = 0.0;
         double totalCorr252 = 0.0;
         int count = 0;
         int numSymbols = participants.Count;
      

         // Ensure enough symbols for correlation calculation
         if (numSymbols < 2)
            return;

         // Iterate through all unique symbol pairs and compute both 20-day and 252-day correlations
         for (int i = 0; i < numSymbols - 1; i++)
         {
            for (int j = i + 1; j < numSymbols; j++)
            {
               BarHistory bh1 = participants[i];
               BarHistory bh2 = participants[j];

               int idx = GetCurrentIndex(bh1);
               //if (idx < 0) continue;  // Skip if invalid index

               // Compute 20-day correlation
               Corr corr20 = new Corr(bh1, bh2, PriceComponent.Close, 20);
               double corrVal20 = (idx < corr20.Count) ? corr20[idx] : 0.0;

               // Compute 252-day correlation
               Corr corr252 = new Corr(bh1, bh2, PriceComponent.Close, 252);
               double corrVal252 = (idx < corr252.Count) ? corr252[idx] : 0.0;

               // Sum up correlation values
               totalCorr20 += corrVal20;
               totalCorr252 += corrVal252;
               count++;

               // Debugging logs for each pair comparison
               WriteToDebugLog("Comparing: " + bh1.Symbol + " vs " + bh2.Symbol +
                           " Corr20= " + corrVal20 + " Corr252= " + corrVal252);
            }
         }

         // Compute the global averages
         double avgCorr20 = (count > 0) ? totalCorr20 / count : 0.0;
         double avgCorr252 = (count > 0) ? totalCorr252 / count : 0.0;
         double avgCorrDelta = avgCorr20 - avgCorr252;

         // Store avgCorrDelta in all symbols' UserData
         foreach (var bh in participants)
         {
            // calculate ranking

            int idx = GetCurrentIndex(bh);
            // store userData as adjusted ROC
            bh.UserData =    Math.Sign(avgCorrDelta) * ((IndicatorBase)bh.Cache["indicator"])[idx] ;
            
         }

         // sort participants using the stored UserData field ( ROC adjusted for correlation sign )
         participants.Sort((a, b) => b.UserDataAsDouble.CompareTo(a.UserDataAsDouble));
         // store rank order in cacheKey
         for (int i = 0; i < participants.Count; i++)
         {
            participants[i].Cache["cacheKey"] = i;
         }

      }

   
      public override void Execute(BarHistory bars, int idx)
      {
         idxRank = idx;
         Position openPosition = FindOpenPosition(0);
         Position openPositionLong = FindOpenPosition(PositionType.Long);
         Position openPositionShort = FindOpenPosition(PositionType.Short);
         

         if (bars.Cache.TryGetValue("cacheKey", out object rankObj) && rankObj is int rank)
         {
            
            // if no open position then check viability of long
            if (openPosition == null && CheckTradeCondition(bars, "cacheKey", Threshold, true))
            {
               _transaction = PlaceTrade(bars, TransactionType.Buy, OrderType.Market, 0, 0, "Buy At Market initial");               
            }
            // if no open positoin then check viability of short
            else if (openPosition == null && CheckTradeCondition(bars, "cacheKey", Threshold, false))
            {
               _transaction = PlaceTrade(bars, TransactionType.Short, OrderType.Market, 0, 0, "Short At Market initial");
            }
            // if position is long then check to see if it should be reversed
            else if (openPositionLong != null && CheckTradeCondition(bars, "cacheKey", Threshold, false))
            {               
               ClosePosition(openPosition, OrderType.Market, 0, $"Sell At Market (Rank: {rank})");
               _transaction = PlaceTrade(bars, TransactionType.Short, OrderType.Market, 0, 0,  $"Short At Market (Rank: {rank})");
            }
            // if position is short then check to see if it should be long
            else if (openPositionShort != null && CheckTradeCondition(bars, "cacheKey", Threshold, true))
            {         
               ClosePosition(openPosition, OrderType.Market, 0,  $"Cover At Market (Rank: {rank})");
               _transaction = PlaceTrade(bars, TransactionType.Buy, OrderType.Market, 0, 0,  $"Buy At Market (Rank: {rank})");      
            }
         }
      }
      // longshort: true for long; false for short
      private bool CheckTradeCondition(BarHistory bars, string cacheKey, int threshold, bool longshort )
      {
         if (!bars.Cache.TryGetValue(cacheKey, out object rank) || (int)rank >= threshold && longshort )
         {         
            return false;
         }
         else if (!bars.Cache.TryGetValue(cacheKey, out object rank1) || (int)rank1 < threshold && !longshort)
         {      
            return false;
         }
         bool conditionMet = (idxRank - 1 >= 0) && (monthIndicator[idxRank] != monthIndicator[idxRank - 1]);   
         return conditionMet;
      }

      private IndicatorBase indicator, monthIndicator;
      private int idxRank;
      private Transaction _transaction;
      private List<IndicatorBase> _startIndexList = new List<IndicatorBase>();
      private int Threshold;
      private Corr corr1;
      private double avgCorrDelta;
   }
}

I haven't looked at the code, but uncheck all Event Providers. Often it's the "Loading" process that's taking all the time, not the run time.

Edit - nope. It's the code it seems.

You're recalculating correlations for every permutation of bar histories on every bar... twice.

You need to come up with a scheme to do that once during Initialize using BacktestData and cache the result with a key like $"{bars1.Symbol}-{bars2.Symbol}"

Thanks Cone. I will work on moving the calcs to Initialize. I did not realize that preExecute also runs for every bar just prior to Execute.. It makes sense now.

Could you Corr.Series which employs caching instead of recalculating it over and over again using the “new” operator?

Good point Glitch, I should have thought of that.
Just replace new Corr(.. with Corr.Series(..

I am still struggling to get this script to work correctly. I have take a step back and started with the published "Tactical Asset Allocation v2" strategy and then modified it to approximate what I am trying to do. I do most of the calculations in the Imnitialize phase and then try to do a final sort in the PreExecute phase. I get an error on line 71 when it tries to retrieve the cache value during the preExecute phase. The error is: PreExecute Exception (3/14/2023) Line 71 - The given key 'ROC252' was not present in the dictionary.
I get the same error for all dates in the backtest. What am i doing wrong?

CODE:
using System;
using WealthLab.Core;
using WealthLab.Backtest;
using WealthLab.Indicators;
using System.Drawing;
using System.Collections.Generic;

namespace WealthScript6 
{
   public class TacticalAssetRotation : UserStrategyBase
   {
      //declare private variables below
   
      private string seriesKey = "ROC252";

      //the list of symbols that we should buy each bar
      private static List<BarHistory> buys = new List<BarHistory>();

      //create the weight indicator and stash it into the BarHistory object for reference in PreExecute
      public override void Initialize(BarHistory bars)
      {

         List<BarHistory> participants = BacktestData;
         int count = 0;
         int numSymbols = participants.Count;
      

         // Initialize TimeSeries properly with DateTimes
         TimeSeries sumCorr20 = new TimeSeries(participants[0].DateTimes);
         TimeSeries sumCorr252 = new TimeSeries(participants[0].DateTimes);

         for (int i = 0; i < numSymbols - 1; i++)
         {
            for (int j = i + 1; j < numSymbols; j++)
            {
               BarHistory bh1 = participants[i];
               BarHistory bh2 = participants[j];
               
               // Compute 20-day correlation
               TimeSeries corr20 = new Corr(bh1, bh2, PriceComponent.Close, 20);

               // Compute 252-day correlation
               TimeSeries corr252 = new Corr(bh1, bh2, PriceComponent.Close, 252);

               // Accumulate total correlation values using TimeSeries
               sumCorr20 = sumCorr20 + corr20;
               sumCorr252 = sumCorr252 + corr252;
               count++;

            }
         }

         TimeSeries avgCorrDelta = (sumCorr20 - sumCorr252) / count;
         TimeSeries signedCorrDelta = avgCorrDelta / avgCorrDelta.Abs();

         // Store avgCorrDelta in all symbols' UserData
         for (int i = 0; i < numSymbols - 1; i++)
         {
            TimeSeries avgROC = ROC.Series(participants[i].Close, 200) / 3;
         
            participants[i].Cache[seriesKey] = signedCorrDelta * avgROC;  // Apply sign adjustment   
         }
      }
            
      //this is called prior to the Execute loop, determine which symbols have the lowest average ROC
      public override void PreExecute(DateTime dt, List<BarHistory> participants)
      {
         //store the symbols' AvgROC value in their BarHistory instances
         foreach (BarHistory bh in participants)
         {
            TimeSeries symbolRoc = (TimeSeries)bh.Cache[seriesKey];
            int idx = GetCurrentIndex(bh);  //this returns the index of the BarHistory for the bar currently being processed
            double rocVal = symbolRoc[idx];
            WriteToDebugLog(bh.Cache[seriesKey]);

            //if the indicator isn't valid set a low value to avoid selection in rotation
            if (idx < symbolRoc.FirstValidIndex)
               rocVal = -1.0e6;
            // WriteToDebugLog(rocVal);
            bh.UserData = rocVal; //save the current AvgROC value along with the BarHistory instance
         
         }
         //sort the participants by AvgROC value (highest to lowest), corrected for v2
         participants.Sort((a, b) => b.UserDataAsDouble.CompareTo(a.UserDataAsDouble));
         //keep the top 3 symbols
         buys.Clear();
         for (int n = 0; n < 3; n++)
         {
            if (n >= participants.Count)
               break;
            buys.Add(participants[n]);
         }
      }

      //execute the strategy rules here, this is executed once for each bar in the backtest history
      public override void Execute(BarHistory bars, int idx)
      {
         bool inBuyList = buys.Contains(bars);
         if (!HasOpenPosition(bars, PositionType.Long))
         {
            //buy logic - buy if it's in the buys list
            if (inBuyList)
               PlaceTrade(bars, TransactionType.Buy, OrderType.Market);
         }
         else
         {
            //sell logic, sell if it's not in the buys list
            if (!inBuyList)
               PlaceTrade(bars, TransactionType.Sell, OrderType.Market);
         }
      }
   }
}

You are adding up the TimeSeries in the Initialize method, but it's running into errors because the TimeSeries are different lengths. It's probably better to stick to doing this in the PreExecute method, because you're dealing with only the participants for that bar, and you can work with the individual correlation values instead of the TimeSeries.

Here's a refactor that does all the calculations in PreExecute, and saves both correlations to a new object called CorrHolder. I also show how to get the CorrHolder back in the Execute method.

CODE:

using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Indicators;
using System.Collections.Generic;

namespace WealthScriptCorrelation
{
   public class MyStrategy : UserStrategyBase
   {
      //the list of symbols that we should buy each bar
      private static List<BarHistory> buys = new List<BarHistory>();

      //create the weight indicator and stash it into the BarHistory object for reference in PreExecute
      public override void Initialize(BarHistory bars)
      {
         // Compute correlation with SPY and store in cache
         corr1 = new Corr(bars, "SPY", PriceComponent.Close, 20);
         bars.Cache["corr1"] = corr1;
         StartIndex = 252;
      }

      // This is called before Execute, calculating correlation for each symbol in participants
      public override void PreExecute(DateTime dt, List<BarHistory> participants)
      {
         double totalCorrelation20 = 0.0;
         double totalCorrelation252 = 0.0;
         int count = 0;
         int numSymbols = participants.Count;

         // Iterate through all unique symbol pairs and compute correlation
         for (int i = 0; i < numSymbols - 1; i++)
         {
            for (int j = i + 1; j < numSymbols; j++)
            {
               BarHistory bh1 = participants[i];
               BarHistory bh2 = participants[j];

               int idx = GetCurrentIndex(bh1);
               if (idx < 0) continue;  // Skip if invalid index

               // Compute correlations between bh1 and bh2
               Corr corr20 = Corr.Series(bh1, bh2.Symbol, PriceComponent.Close, 20);
               double corrVal20 = corr20[idx];
               Corr corr252 = Corr.Series(bh1, bh2.Symbol, PriceComponent.Close, 252);
               double corrVal252 = corr252[idx];

               // sum correlations
               totalCorrelation20 += corrVal20;
               totalCorrelation252 = corrVal252;
               count++;


               WriteToDebugLog("Comparing: " + bh1.Symbol + " vs " + bh2.Symbol + " Correlation20= " + corrVal20);
            }
         }

         // Compute the global average correlation
         double avgCorrelation20 = (count > 0) ? totalCorrelation20 / count : 0.0;
         double avgCorrelation252 = (count > 0) ? totalCorrelation252 / count : 0.0;
         WriteToDebugLog("dt= " + dt + " Global avgCorrelation20= " + avgCorrelation20 + " 252=" + avgCorrelation252);

         // Store the same global average correlation in all symbols' UserData
         CorrHolder ch = new CorrHolder();
         ch.CorrAvg20 = avgCorrelation20;
         ch.CorrAvg252 = avgCorrelation252;
         foreach (var bh in participants)
         {
            bh.UserData = ch;
            WriteToDebugLog("Symbol= " + bh.Symbol + " Stored avgCorrelation= " + avgCorrelation20);
         }
      }



      //execute the strategy rules here, this is executed once for each bar in the backtest history
      public override void Execute(BarHistory bars, int idx)
      {
         //get the avg correlations
         CorrHolder ch = bars.UserData as CorrHolder;
         if (ch != null)
            WriteToDebugLog(ch.CorrAvg20 + ", " + ch.CorrAvg252);
      }

      //declare private variables below
      private Corr corr1;
   }
   
   internal class CorrHolder
   {
      internal double CorrAvg20 {  get;  set; }
      internal double CorrAvg252 {  get;  set; }
   }
}

Thanks Glitch. I will try this again and see what I can break.

I was able to get all the sorting and ranking to work but there is something wrong with the data being written to debug log. Here is snip of the code:

CODE:
   public override void Execute(BarHistory bars, int idx)
   {
         Position openPosition = FindOpenPosition(0);
         Position openPositionLong = FindOpenPosition(PositionType.Long);
         Position openPositionShort = FindOpenPosition(PositionType.Short);
         
      //get the avg correlations
      CorrHolder ch = bars.UserData as CorrHolder;
      
      // trade monthly
      //if (monthIndicator[idx] !=monthIndicator[idx - 1])
      {
         WriteToDebugLog(idx  + " execute date =  "+ bars.DateTimes[idx].ToShortDateString() +" "+ bars.Symbol + " corr20 =" + ch.CorrAvg20 + ", " + ch.CorrAvg252 + " ch.CorrROC= " + ch.CorrROC + " ch.CorrRank= " + ch.CorrRank, false);

and here is a pic of the debug log:


The idx value and the date do not align for all symbols which can mess with the ranking. Is there a way to get all of the idx and dates aligned? T
Thanks!

It's perfectly natural for symbols to potentially have different indices on the same dates.