Norgate-Current-Past-DataSets-Have-Wrong-Bar-Count

By design, since Wealth-Lab 7.

When you load a symbol from Historical DataSets, you get that bars on which the symbol was a "participant".
That's how historical backtests work automatically.

Edit -
176 bars for NVDA caught my attention. That's wrong. NVDA and SHW both entered the Dow30 on 2024-11-08. Open a chart for symbols in the Wealth-Data Dow 30 DataSet and you'll see the correct number - 693 bars.

AAPL and GS come up short too, so there does seem to be a bug with the Norgate integration.

QUOTE:
there does seem to be a bug with the Norgate integration

That is serious...

QUOTE:
Open ... the Wealth-Data Dow 30 DataSet and you'll see the correct number

Ok, I run the above stratgy for the Wealth-Data Dow30 DataSet:

QUOTE:
---Symbol by Symbol Debug Logs---
---MMM---
N=6289
---NVDA---
N=289
---DIS---
N=6289
---GS---
N=3087
---AAPL---
N=2717
---MSFT---
N=6289
---UNH---
N=3338
---CSCO---
N=4170

Now the numbers are somewhat closer to Norgate's Current & Past DataSet but still different by a fair amount.

What is going on here?
How can I ever get a trustworthy backtest?

I already explained it, right?
Wealth-Data numbers are correct for the Historical DataSets.

As for Norgate, we'll have to see what's going on there. You're a programmer and know how bugs work.
On the surface, it simply appears that Norgate's historical dates are wrong for at least NVDA and AAPL for the Dow 30.

Running a historical check, the Norgate results for DJIA Current and Past look correct:

CODE:
4/2/2019   30
   Added: DOW
   Remvd: DD
8/31/2020   30
   Added: AMGN, HON, CRM
   Remvd: RTX, PFE, XOM
2/26/2024   30
   Added: AMZN
   Remvd: WBA-202508
11/8/2024   30
   Added: NVDA, SHW
   Remvd: INTC, DOW
6/29/2026   30
   Added: GOOGL
   Remvd: VZ

I ran DrKoch's code with a modification to show dates, and the following was output for NVDA:

CODE:
---NVDA---
          N=289. Start Date = 11/7/2023 12:00:00 AM, End Date = 12/31/2024 12:00:00 AM

So, NVDA was added to the DOW 30 on 2024-11-08 as indicated by Cone. But, the date above has a start date of 2023-11-07.

I have no idea what is right or wrong here. I'm just providing info.
Here's the modified strategy if you want a quick change...

CODE:
using WealthLab.Backtest;
using WealthLab.Core;

namespace WealthScript3
{
   public class MyStrategy : UserStrategyBase
   {
      public override void Initialize(BarHistory bars)
      {
         WriteToDebugLog($"          N={bars.Count}. Start Date = {bars.DateTimes[0]}, End Date = {bars.DateTimes[bars.Count - 1]}");
      }

      public override void Execute(BarHistory bars, int idx)
      {
      }
   }
}

Here's the answer:

WealthData loads 1 year of "seed" data prior to the date incorporated into the index, whereas Norgate only loads an extra 200 calendar days.

I'm not smart on this historical protocol, but it would seem that Norgate (or maybe WealthLab core) wanted to provide an extra 200 bars, but loaded an extra 200 calendar days. Glitch would know about this.

@paul1986 - use the BarHistory ToString() method ;)

Right, there is some seed data added so indicators can calculate. But the historic dates that symbols were added to an index are honored. So you can count on accurate survivorship bias free backtests.

@Glitch, what's the cause of the difference for the starting seed data for Wealth-Data vs. Norgate? Is that set by the provider?

Yes, it's evidently what Norgate coded in their provider.

Ok, I see: If I access a symbol in the context of a "historical correct" portfolio, I get just the bars for the historical period when the Symbol was part of that portfolio.

If I access the same symbol in any other way, I get the complete available bar history.

Example NVDA:
If I open a Chart Window and enter NVDA in the symbol textbox I get 6693 bars

If I click in the DataSet tree on WealthData->DOW30->NVDA I get 693 bars

If I click in the DaaSet tree on Norgate Data->Dow Jones Industrial Average->NVDA I get 6932 bars

If I click the DataSet tree on Norgate Data - Current & Past-> Dow Jones Industrial Average current & Past->NVDA I get 580 bars

If I click the DataSet tree on Norgate Data - Current & Past-> S&P 500 Current & Past->NVDA I get 6350 bars

If I click the DataSet tree on World Indices -> U.S. - Dow Jones - DJ 30 Industrials->NVDA I get 6932 bars (This is yahoo data)

If I run a portfolio level backtest, the code works on the "short" bar history. If I double click on Backtest Results->Positions->a NVDA position I see a chart with the long bar history.

Probably this is a clever design. But it is very confusing to say the least.

If I want to find out why a certain symbol within a portfolio level backtest yields strange results, it is nearly impossible to run a single symbol backtest on the same (short) bar history, which makes a detailed analysis nearly impossible.

Personally, I never liked it this historical data load logic - especially when you click an active symbol that was removed from an index (e.g., AIG in the Dow 30) and get a chart that ends in the past. ("mind blown" emoji)

Now that I know that it's loading seed data for historical backtests anyway, I don't even understand the point of it now.
The devil must be in the details.

It's a relic from a much earlier time, no doubt it could be done better if we were to re-invent things now. But it does work.

I understand that you are reluctant to touch old, working code. Let me suggest some improvements anyway. Probably you find the time...

If I open a Chart and enter a symbol a dropdown opens and offers me a selection of "matching symbols" that often come form different data providers.

It would help, if this list showed the DataSet also and probably the length of the bar history, its start and end time. As we have seen before these can vary dramatically from DataSet to DataSet.

I'd suggest a similar mechanism, whenever a symbol is loadad as response to a mouse click, for example a click on a Position.

It is not clear and depends on circumstances if I want to see a "long" or a "short" bar history or if a strategy should run on either data.

That part is clear:
the Symbol Selector is not associated with any DataSet, you get the full history the provider offers.

And note:
Wealth-Data is generally cut off at the year 2000. If you need older histories, use another provider like Norgate, Yahoo!, etc.