NaN-in-TimeSeries-operations

F#? Ok, but here's my result with the C# code that follows.

CODE:
idx   ts1, ts2   ts1-ts2
0   1, NaN   NaN
1   2, 3   -1

CODE:
         var ts1 = new TimeSeries();
         var ts2 = new TimeSeries();
         
         ts1.Add(1.0, DateTime.Now);
         ts1.Add(2.0, DateTime.Now.AddHours(1));
         ts2.Add(double.NaN, DateTime.Now);
         ts2.Add(3.0, DateTime.Now.AddHours(1));         
         TimeSeries tsa = ts1 - ts2;

         WriteToDebugLog($"idx\tts1, ts2\tts1-ts2");
         for (int n = 0; n < ts1.Count; n++)
         {
            WriteToDebugLog($"{n}\t{ts1[n]}, {ts2[n]}\t{tsa[n]}");
         }

Where is the 1.0 + NaN operation?

Please, dump to debug log `ts1+ts2`.

I have a band indicator with a `window` hyperparameter. It needs `window` amount values to warm up. I wonder then what values should I put to first `window-1` of `TimeSeries` so time series as follow would be equal?

CODE:

price_channel_upper = bars.Open + band_indicator;
price_channel_lower = bars.Open - band_indicator;

I expected to fill them with Double.NaN (as Python numpy, pandas, etc. do), but apparently this doesn’t work.

In Initialize(), where bars is the chart's BarHistory, this statement creates a TimeSeries filled with NaNs that has the same number of elements as the chart's BarHistory.

CODE:
TimeSeries myTS = new TimeSeries(bars.DateTimes);

Sorry, not following how it helps. Having price channel defined as:

CODE:

price_channel_upper = bars.Open + band_indicator;
price_channel_lower = bars.Open - band_indicator;

where `bars.Open` all defined, and `band_indicator` with first `window-1` as `Double.NaN` and rest defined, `price_channel_lower` would have leading `window-1` as `Double.NaN` (which is expected) and `price_channel_upper` would have leading `window-1` equal to corresponding `bars.Open` since `Double.NaN` is interpreted as 0 for `+` operator. And when I compute signal based on comparison with `price_channel_upper`, for leading `window-1` it also won't be `Double.NaN`, but comparison with leading `bars.Open`. Hence, I have to do something extra with leading `price_channel_upper`: fill them as Double.NaN, drop them, etc.

Just assign FirstValidIndex to your indicator's TimeSeries. WealthLab uses this index (automatically in block strategies) to determine when indicators are valid and sets the backtest's StartIndex to the greatest value found for all indicators instantiated in Initialize();

I see that arithmetical operations on TimeSerieses propagate FirstValidIndex, right?

CODE:

band_indicator.FirstValidIndex = 10;
price_channel_upper = bars.Open + band_indicator;
price_channel_lower = bars.Open - band_indicator;
// price_channel_upper.FirstValidIndex is 0
// price_channel_lower.FirstValidIndex is 10

So, this doesn’t work for the same discrepancy.

What's the "discrepancy" exactly? I'm not getting it.

The point is the Strategy doesn't care about anything before the StartIndex. Like what is done for the Block strategies, if you find the greatest FirstValidIndex of all indicators used and assign it to StartIndex, all indicators should be valid at the StartIndex. What's the problem?

Cone,

this means I probably should re-implement `+` myself that properly handles NaNs and FirstValidIndex)

QUOTE:
this means I probably should re-implement `+` myself

I'm not getting it either. You need to set StartIndex large enough (in Initialize) so it avoids all the initial NaNs in all the indicators. (If you're creating an indicator, then you need to set indicator.FirstValidIndex after the last initial NaN for that indicator.)

WL will not simulate any trades until on and after StartIndex, so all the early NaNs will not affect any part of the simulation. This design has been working for all WL users and it should work for you too!

Sorry, not following. Do I understand you correctly, that code should be like:

CODE:

price_channel_upper = bars.Open + band_indicator;
price_channel_upper.FirstValidIndex = max(bars.Open.FirstValidIndex, band_indicator.FirstValidIndex);

Meaning I need to track explicitly `FirstValidIndex` of all indicators used in arithmetical operations? I don't understand why I should give an extra line and a hole for errors here when proper `+` can and should do it inside.

Still, it's a mystery for me on reasons behind Double.NaNs handling in `+` of `TimeSeries`.

I already explained the reasoning, it's because it makes many other indicators and operations work with much less hassle. Now, it's fine if you don't agree with the reasoning but it's not a mystery.

QUOTE:
Do I understand you correctly,..?

CODE:
price_channel_upper.FirstValidIndex = Math.Max(bars.Open.FirstValidIndex, band_indicator.FirstValidIndex);

First, the code above won't compile because bars.Open.FirstValidIndex doesn't exist. bars.Open is not an indicator, so it doesn't have a FirstValidIndex property.

Second, are we talking about how the code looks inside an indicator or inside a strategy? If we are inside a strategy, then you need to set StartIndex so the simulation begins after all the leading NaNs of all the indicators.

CODE:
      public override void Initialize(BarHistory bars)
      {
         sma = SMA.Series(bars.Close,5);
         price_channel_upper = bars.Open + band_indicator;
         StartIndex = Math.Max(band_indicator.FirstValidIndex, sma.FirstValidIndex);
      }

In practice, you probably don't need the Max method because you probably know which indicator in your strategy has the largest FirstValidIndex so you can set StartIndex to that.

First, let me say I "GET" how FirstValidIndex works. What is demonstrated below is a difference in how the TimeSeries operator overlaods of + and - work.

Also, I understand, in reference to the TimeSeries addition overload, as Glitch noted in Post, #3 that it makes things more convenient when combining many indicators.

The following simple test strategy uses the + and - overloads. The output is also shown. Perhaps there should be a difference in the way + and - handle NaN. I don't know. But, the output shows there is a difference. Perhaps - is wrong, perhaps it is right (by design).

CODE:
using System;
using WealthLab.Backtest;
using WealthLab.Core;
using WealthLab.Indicators;

namespace WealthLabStrategies.Test
{
    public class OperatorOverload : UserStrategyBase
    {
        private TimeSeries MinusOverload { get; set; }

        private TimeSeries PlusOverload { get; set; }

        private SMA Sma { get; set; }

        public override void Initialize(BarHistory bars)
        {
            Sma = SMA.Series(bars.Close, 5);
            PlusOverload = bars.Open + Sma; // overload of + used
            MinusOverload = bars.Open - Sma; // overload of - used

            // let's take the make of the two first valid indexes...
            var maxIndex = Math.Max(PlusOverload.FirstValidIndex, MinusOverload.FirstValidIndex);

            // ...and the min of the two first valid indexes
            var minIndex = Math.Min(PlusOverload.FirstValidIndex, MinusOverload.FirstValidIndex);

            WriteToDebugLog($"Sma.FirstValidIndex: {Sma.FirstValidIndex}");
            WriteToDebugLog($"bars.Open.FirstValidIndex: {bars.Open.FirstValidIndex}");

            WriteToDebugLog($"PlusOverload.FirstValidIndex (uses overload of +): {PlusOverload.FirstValidIndex}");
            WriteToDebugLog($"MinusOverload.FirstValidIndex (uses overload of -): {MinusOverload.FirstValidIndex}");
            WriteToDebugLog("");

            // dump the values between the min and the max along with the Sma and the bars.Open
            for (var i = minIndex; i <= maxIndex; i++)
            {
                WriteToDebugLog($"PlusOverload[{i}] = {PlusOverload[i]}, Sma[{i}] = {Sma[i]}, bars.Open[{i}] = {bars.Open[i]}");
            }

            WriteToDebugLog("");

            for (var i = minIndex; i <= maxIndex; i++)
            {
                WriteToDebugLog($"MinusOverload[{i}] = {MinusOverload[i]}, Sma[{i}] = {Sma[i]}, bars.Open[{i}] = {bars.Open[i]}");
            }
        }

        public override void Execute(BarHistory bars, int idx)
        {
        }
    }
}

Here is the output using symbol CLSK...

CODE:
---Symbol by Symbol Debug Logs---
---CLSK---
Sma.FirstValidIndex: 4
bars.Open.FirstValidIndex: 0
PlusOverload.FirstValidIndex (uses overload of +): 0
MinusOverload.FirstValidIndex (uses overload of -): 4

PlusOverload[0] = 18.35, Sma[0] = NaN, bars.Open[0] = 18.35
PlusOverload[1] = 18.4501, Sma[1] = NaN, bars.Open[1] = 18.4501
PlusOverload[2] = 18.44, Sma[2] = NaN, bars.Open[2] = 18.44
PlusOverload[3] = 18.455, Sma[3] = NaN, bars.Open[3] = 18.455
PlusOverload[4] = 36.79082, Sma[4] = 18.43082, bars.Open[4] = 18.36

MinusOverload[0] = NaN, Sma[0] = NaN, bars.Open[0] = 18.35
MinusOverload[1] = NaN, Sma[1] = NaN, bars.Open[1] = 18.4501
MinusOverload[2] = NaN, Sma[2] = NaN, bars.Open[2] = 18.44
MinusOverload[3] = NaN, Sma[3] = NaN, bars.Open[3] = 18.455
MinusOverload[4] = -0.07082000000000122, Sma[4] = 18.43082, bars.Open[4] = 18.36

Just read Post #3 carefully.
Otherwise, we'll just be repeating the same answer.

The point of Post #15 is that the subtraction operator handles NaN differently than the addition operator. I'm simply pointing that out in case there is an issue. I don't know. That's why I stated perhaps the difference between addition and subtraction overloads in handling NaN is by design.

Is the difference between addition and subtraction in the handling of NaN by design?

Yes it’s by design.