Custom-TimeSeries-works-unexpectedly-after-upgrading-from-WL8-Build-18-to-20

https://social.msdn.microsoft.com/Forums/sqlserver/en-US/1444bc9d-3c27-4915-8d41-85580f2a721d/using-systemdata-and-other-using-code-is-grey-out?forum=csharpgeneral

From the link in Reply# 1,

QUOTE:
The code which is greyed out won't change anything when your program runs. Everything that's needed is loaded on demand. ... unless you actually use a type in that namespace / assembly, the assembly that using statement is correlated to won't be loaded.

So the code for List<Generic> is being loaded by default without the usual using statement. Interesting.

So WL8 Build 20 broke my DLL library for some other reason. I wonder why it worked okay with all other builds? Yes, it does call FirstValidIndex, but only to set it.

CODE:
else if (++N == 3) this.FirstValidIndex = beaconBars[incomingIdx];
            if (N > 2) //only drop outliers within mature (large N) moving averages
            {
               double nextMax = movingWindowPeaks.ElementAt(movingWindowPeaks.Count-2); //save second highest maximum
               double outlierTolerance = (movingWindowPeaks.Max - nextMax) / nextMax;
               //exclude highest peak from moving average computation if it exceets the next-highest peak tolerance
               average = outlierTolerance > _outlierToleranceMax ? (sum - movingWindowPeaks.Max) / (N-1) : sum / N; //cal local peak average
            }
            else
               average = sum / N; //cal local peak moving average
            int nextPeakBar = incomingIdxP1 != BeaconBars.Length ? BeaconBars[incomingIdxP1] : ts4Thresholding.Count;
            for (int bar = BeaconBars[incomingIdx]; bar < nextPeakBar; bar++)
               base[bar] = average;

this is the TimeSeries output object of the malfunctioning class. The malfunctioning class returns all NaN's for the output TimeSeries.

I'm using Build 20 B.

I have an external library that uses System.Collections.Generic and System.Linq. In the external library I have a VWAP indicator that generates values for sets of deviations into numerous TimeSeries instances. I'm getting values in all of the TimeSeries instances.

So, I know that may not be much help to you other than to suggest double-checking if you changed some of your code between installations.

I'm trying to roll back to Build 18, but the installer quits saying that a more recent version is already installed. How can I bypass that error message?

The external code I'm having trouble with is doing a SortedSet and a ElementAt() Linq call. But I can't understand how the update to Build 20 would affect that. These are all .NET calls and not WL calls. Everything works fine on WL6, which uses similar code and the WL8 code in question hasn't be changed for 6 months. And I tried some older WL8 strategies that use the same library code, and they don't work either.

I noticed what topic starter has pointed about System.Linq et al. in production B20 when trying to compile an existing code-based strategy (not a 'compiled strategy library') that worked before, making use of Linq, WebClient, XML etc. After ticking all checkboxes generously suggested by the compiler in Tools > Assembly References, it worked.

QUOTE:
about System.Linq et al. in production B20 when trying to compile an existing code-based strategy ... that worked before, making use of Linq, WebClient, XML etc. After ticking all checkboxes generously suggested by the compiler in Tools > Assembly References, it worked.

So you're saying there's a compiler bug that fails to flag missing assembles?

I'm looking at my VS Tools menu (see screenshot), but I don't see any Assembly References listed there. And I can't find any suggestions for adding a reference. Where are you looking?

It all compiles and builds without errors.

The only way we can definitively help is if we had the source code of the library so we could compile it locally and see what is going on. Maybe provide a stripped-down version that is only large enough the illustrate the issue? You can send it to support@wealth-lab.com.

superticker - I noticed you apparently have an invalid WealthLab.ScottPlot reference. Its a long shot, but maybe that is indirectly causing the issue.

The problem is with the PeakTroughCalculator. If you pass it a sparse vector, vossBeacon, with mostly zeros but a few transient spikes; Build 20 finds no peaks but Build 18 and earlier do. The code below for building the sparse vector is incomplete, but you can substitute you own sparse vector creator for that.

BTW, vossPredictor.CrossesOver(vossBandPass,bar) comes from Technical Analysis of Stocks & Commodities, August 2019, by John Ehlers if you're curious. They are part of the Voss Predictor indicators WL has.

CODE:
         TimeSeries dx = DX.Series(bars, paramDxPeriod);
         dxSmoothed = new DSMA(dx,19) / new SMA(dx,50); //normalize baseline with SMA
         dxSmoothed.Description = "Normalized smoothed DX";
         vossBeacon = new TimeSeries(vossDivergence.DateTimes,0.0);
         vossBeacon.Description = "Voss beacon";
         for (int bar = 1; bar < vossBeacon.Count; bar++)
         {
            if (vossPredictor.CrossesOver(vossBandPass,bar))
            {
               SetBackgroundColor(bars,bar,WLColor.Khaki,"predictorPane");
               vossBeacon[bar] = dxSmoothed[bar];
            }
         }
         PeakTroughCalculator beaconPeakTroughs = new PeakTroughCalculator(vossBeacon, 0.2, PeakTroughReversalType.Point);
         WriteToDebugLog(beaconPeakTroughs.PeakTroughs.Count);
         foreach (PeakTrough peakTrough in beaconPeakTroughs.PeakTroughs)
         {
            if (peakTrough.Type == PeakTroughType.Peak) //save only the peaks for rank sorting
            {
               WriteToDebugLog(peakTrough.Value + "=Value" + peakTrough.PeakTroughIndex + "=peak Index");
            }
         }

This code is working good on WL7 and the equivalent on WL6. The modified version of John Ehlers' Voss predictor is one of my best performing strategies.

superticker - I can run a test on my machine. But, what data types are vossDivergence and vossBandPass?

QUOTE:
what data types are vossDivergence VossPredictor and vossBandPass BandPass?

These are WL indicators (VossPredictor, BandPass) located in the TASC indicator library. See http://www2.wealth-lab.com/WL5Wiki/TASCAug2019.ashx for a WL6 strategy. If you want my WL8 conversion, we'll need to start a new topic to post that. :(

But you don't have to use those for testing. Just create any old sparse vector (mostly zeros) and pass it into the PeakTroughCalculator.

I think I found the problem:

vossBeacon = new TimeSeries(vossDivergence.DateTimes, 0.0);

If you change it to:

vossBeacon = new TimeSeries(vossDivergence.DateTimes);

then you get a bunch of values instead of no values. So, apparently, using the fill parameter causes an issue. I have no idea why.

But the vossBeacon is only suppose to hold non-zero values on the crossovers. The Beacon holds two elements of information:

1) Timing. When there would be a Buy opportunity.
2) Magnitude. What's the merit of that Buy opportunity?

So when there isn't a Buy opportunity at a given time, that vossBeacon bar should be zero. A robust statistical analysis follows with all the vossBeacon Buy opportunities (vossBeacon peaks), so it's critical the vossBeacon holds the right timing and magnitude information to statistical compute the best Buy opportunities. That code is not shown.

But I get the gist of your suggestion. One could change the vossBeacon zeros to some minimal number that wouldn't throw the statistical analysis off too much. Not ideal, but doable.

Some beacons can be indicators instead of crossovers. In those cases, there wouldn't be too many zeros in those beacons. In actual practice, the beacon is typically crafted by combining several indicators in a nonlinear way (but that's another topic).

As a workaround keep the change I suggested and change your loop:

CODE:

                if (vossPredictor.CrossesOver(vossBandPass, bar))
                {
                    SetBackgroundColor(bars, bar, WLColor.Khaki, "predictorPane");
                    vossBeacon[bar] = dxSmoothed[bar];
                }
                else
                {
                    vossBeacon[bar] = 0.0;
                }

I don't know what causes the fillValue problem, as I'm not a WL employee. So, I have no insight into the inner workings of the TimeSeries constructor. Maybe Glitch or Cone or Eugene can help.

Well that's the way the code is working now as a sparse vector beacon. It's mostly zeros with transient spikes at the crossovers. I plotted it out to confirm that. But Builds after 18 don't like that.

If you can publish a complete CODE that illustrates the issue I can run it locally and take a look.

Well I'm a little confused. The first code segment based on a random number generator works just fine (to my surprise). The debug statement shows around 70 peaks found.

But the second code segment based of Voss Predictor crossovers fails with zeros peaks found by the PeakTroughCalculator. This second code segment worked for Build 18 but not for Build 20. How is that possible? So why is the second code segment (Voss Predictor) not working, but the first code segment (Random peaks) works fine? I don't see how that's possible.

I was running AAPL when testing this, but I don't think that matters.

CODE:
using WealthLab.Backtest;
using WealthLab.Core;
using WealthLab.Indicators;
using WealthLab.TASC;
using System;

namespace WealthScript2 
{
   public class MyStrategy : UserStrategyBase
   {
      public override void Initialize(BarHistory bars)
      {
         TimeSeries peakBeacons = new TimeSeries(bars.DateTimes,0.0);
         peakBeacons.Description = "Peaks to find";
         Random rand = new Random();
         for (int bar = 0; bar < bars.Count; bar++)
         {
            if (rand.Next(20) > 18)
               peakBeacons[bar] = bars.Close[bar]/100;
         }
         PlotTimeSeries(peakBeacons, peakBeacons.Description, peakBeacons.Description, WLColor.Purple, PlotStyle.Histogram);
         PeakTroughCalculator beaconPeakTroughs0 = new PeakTroughCalculator(peakBeacons, 0.2, PeakTroughReversalType.Point);
         WriteToDebugLog(beaconPeakTroughs0.PeakTroughs.Count + "=#of peaks found");

         

         IndicatorBase vossPredictor = new VossPredictor(bars.Close, 22,5);
         vossPredictor.Description = "Voss predictor";
         IndicatorBase vossBandPass = new BandPass(bars.Close, 22,0.25);
         vossBandPass.Description = "Voss bandpass"; //bandpass filter
         PlotIndicator(vossPredictor, WLColor.BlueViolet, PlotStyle.Line, false, "predictorPane");
         PlotIndicator(vossBandPass, WLColor.IndianRed, PlotStyle.Line, false, "predictorPane");

         TimeSeries dx = DX.Series(bars,13);
         TimeSeries dxSmoothed = new DSMA(dx,19) / new SMA(dx,50); //normalize baseline with SMA
         dxSmoothed.Description = "Normalized smoothed DX";
         TimeSeries vossBeacon = new TimeSeries(bars.DateTimes, 0.0);
         vossBeacon.Description = "Voss beacon";
         for (int bar = 1; bar < vossBeacon.Count; bar++)
         {
            if (vossPredictor.CrossesOver(vossBandPass, bar))
            {
               SetBackgroundColor(bars, bar,WLColor.Khaki, "predictorPane");
               vossBeacon[bar] = dxSmoothed[bar];
            }
         }
         PeakTroughCalculator beaconPeakTroughs = new PeakTroughCalculator(vossBeacon, 0.2, PeakTroughReversalType.Point);
         WriteToDebugLog(beaconPeakTroughs.PeakTroughs.Count + "=beaconPeakTroughs.PeakTroughs.Count");
      }

      public override void Execute(BarHistory bars, int idx) { }
   }
}

This is what I'm seeing, what is missing?

That's what I see too, but look at the debug line output.

CODE:
---AAPL---
66=#of peaks found
0=beaconPeakTroughs.PeakTroughs.Count

The last line is showing 0= which means no peaks were found for the Voss Predictor case, and there are many peaks as you can see from the plot. Something isn't right.

Do you think ...(below)... is confusing the PeakTroughCalculator?

CODE:
vossBeacon.FirstValidIndex

QUOTE:
Do you think ...(below)... is confusing the PeakTroughCalculator?
vossBeacon.FirstValidIndex

That was one of the big breaking changes in B18:
https://www.wealth-lab.com/Software/ChangeLog

* Cleaned up more indicators so they do not plot before their first valid bar containing data.

When I run it, on 10 years of AAPL Daily,
---AAPL---
202=#of peaks found
314=beaconPeakTroughs.PeakTroughs.Count

1 year
---AAPL---
32=#of peaks found
24=beaconPeakTroughs.PeakTroughs.Count

Since the claim about "B20 breaking custom class libraries" is blunt too general I've renamed the topic:

WAS: Upgrading from WL8 Build 18 to 20 breaks custom class libraries
IS: Custom TimeSeries works unexpectedly after upgrading from WL8 Build 18 to 20

Although I don't think we've changed anything that would affect this, I should mention that I'm using the "Near-Build 21" (just in case).

Well, the results on Reply# 21 look good. I only wish I could get those same results on Build 20. So what changed?

What confuses me is why the PeakTroughCalculator works fine for the Random case but not the Voss Predictor case. That's totally weird. See my Reply# 19 output.

I ran the latest code (Post #17) using symbol MSFT, 10 years, daily. I got zero beaconPeakThroughs. So, the discussion about FirstValidIndex gave me a hint. I changed the last for loop "for (int bar = 1; ...)" so that the index start is calculated via the max of the FirstValidIndex of the loop participants plus one...

CODE:
         var startIdx = Math.Max(vossBeacon.FirstValidIndex, Math.Max(dxSmoothed.FirstValidIndex, 
            Math.Max(vossPredictor.FirstValidIndex, vossBandPass.FirstValidIndex))) + 1; // add one for the crossover check
         for (int bar = startIdx; bar < vossBeacon.Count; bar++)

With the above change it found 334 beaconPeakThroughs. So, maybe that info can help determine what has changed since build 18.

I ran the aforementioned code (Post #25) with build 21, using symbol MSFT, 10 years, daily...

for loop starting from 1...
---Symbol by Symbol Debug Logs---
---MSFT---
212=#of peaks found
340=beaconPeakTroughs.PeakTroughs.Count

for loop starting from calculated startIdx (uses max FirstValidIndex of loop participants plus 1)...
---Symbol by Symbol Debug Logs---
---MSFT---
190=#of peaks found
340=beaconPeakTroughs.PeakTroughs.Count

So, for build 21, the case of the loop starting from 1, its finding beaconPeakThroughs. In build 20, for the the case of the loop starting from 1, it found zero beaconPeakThroughs.

So, in any case, it seems you'd want to determine the max FirstValidIndex for all loop participants because you only want to use valid data points.