GetHistory-bars-QQQ-does-not-pull-from-the-right-DataSet

In order to make it pull from the right Dataset I need to close WL8, Start it again, click on the Data Manager, click on the History Providers Tab, checkmark the Dataset that I want to pull the data from, and only then open the Strategy that I am working with, click on Strategy Settings Tab, click on Single Symbol Tab and type the symbol that I need.

Is it the right way to do it?

You don't have to restart. In the Strategy Window, just click "n Symbols Loaded" to reload data for the next Run Backtest (or change a data setting).

Yes, the order of priority to pull historical data are the checked providers top to bottom.

I do not see the "n Symbols Loaded". Where is it?
How can I change the Data Settings of the existing Dataset?

GetHistory will go through the Historical Providers that you have enabled, one by one, in the order that you have them set up in the Data Manager.

To force GetHistory to use a specific DataSet, specify the DataSet name as the optional third parameter.

>>I do not see the "n Symbols Loaded". Where is it?<<

Here:

Clicking on "n Symbols Loaded" does not help. It says Need to Load Data.

Changing the checkmark in the Data Manager > Historical Providers does not switch to another provider for the external Symbol "GetHistory" but leaves it as "NaN" on the chart.
Only restarting the WL8 works to change it.

QUOTE:
Clicking on "n Symbols Loaded" ... It says Need to Load Data.

So now when you run the strategy again, it will now load the data from the checked historical provider that's highest on the Historical Provider list.

After I click on "n Symbols Loaded" and type another Symbol in the Chart window, it does pull the data from the first provider checked on the list but does NOT pull the data for the external Symbol (GetHistory).
May be it is a bug?

QUOTE:
but does NOT pull the data for the external Symbol (GetHistory).

So the targeted provider for the external symbol is broken. Can you tell us the name of that broken historical provider?

Will that broken provider successfully load any symbol or is it broken for just that one symbol. Have you tried a different provider to try to get that problem symbol?

It is ASCII Dataset.
The Symbol itself is not broken. If I type QQQ, which is the "GetHistory" it does show up as the primary.

The code below works for me, but the external symbol is not coming from an ASCII dataset.

CODE:
   public override void Initialize(BarHistory bars)
   {
      BarHistory qqq = GetHistory(bars, "QQQ");
      PlotBarHistory(qqq,"QQQ");
   }

So the problem is one can't use an ASCII dataset for an external symbol. Why didn't you tell us that initially?

I would recommend you subscribe to the Norgate historical data provider so you don't have to rely on ASCII datasets. Also, Norgate includes the GICS industry classification codes (US stocks) and their corresponding industry historical time series data (for each GICS classification).

Why Wealth Lab 8 would not read the "external" from the ASCII DataSet? Is it intentional design?

If I restart Wealth-Lab 8 it starts reading the external symbol "QQQ" from the chosen (check-marked) ASCII Dataset together with the primary symbol.

I have ASCII Datasets for the data from my Wealth Lab 6.

P.S. Would you be able to answer my question in another discussion I just posted "Different Strategy Windows on different DataSets"?

Personally, I wouldn't want the hassle of updating all my ASCII datasets everyday. Subscribing to a historical data provider avoids that daily hassle.

I only suggested Norgate as a historical data provider because you also expressed an interest in the GICS industry classification codes (industry groupings) as well, which Norgate also provides. So you kill two birds with one stone with a Norgate subscription. I think it's well worth it, but you may disagree.

Did you try my suggestion in Post #4 and provide the name of the DataSet in the third parameter of GetHistory?

Forcing the external Symbol to be pulled from a specific DataSet hardcoded in the code will not work for me, since I need it to be pulled from the SAME dataset where the primary symbol is coming from, which might be a different DataSet every time.

Well, if you're running on a DataSet as opposed to a Symbol you can use ExecutionDataSetName to get the name of the DataSet.

I am running the same "Script 1" on individual Symbol from two different ASCII DataSets "A" and "B", which have common Symbols with different end dates.

QUOTE:
from two different ASCII DataSets "A" and "B", which have common Symbols with different end dates.

So simplify the solution by using a single ASCII dataset with one Date Range, then use WL as shown in the screenshot below to set the desired Date Range that you want. Make the solution as simple as possible.

I have been using the simplest solution for almost 25 years now with Wealth-Lab 6 and older versions, where you just choose the DataSet on the left with one click and run the script.

I’m going to work on something for the next build to help with your use case. 👍

Thank you.

I've added a DataSet property to BarHistory. It will contain the DataSet used to load the BarHistory, if available. For example, if you key in a symbol then a DataSet won't be available.

You can use DataSet.Name to get the name of the DataSet in your Strategy for the third parameter of GetHistory.

It could be beneficial to add an option to hardcode the DataSet name directly in the code.

However, I believe it’s even more important to simplify the user interface, allowing users to select a DataSet by clicking on it and then typing the symbol to run the script.

You can hard code the DataSet name. It's the 3rd optional parameter like Glitch said in Post #4 and Post #14.

CODE:
public BarHistory GetHistory(BarHistory synchWith, string symbol, string dataSetName = null)

Hardcoding the DataSet name in the code comes with its own limitations, as I explained in Post #15.

Are you arguing with yourself from Post #23 now?!? ;)

I have no objection to adding extra features, but they shouldn't come at the cost of existing essential ones.

The thing is no one else has mentioned this use case, so while it’s essential for you we can’t possibly account for every “essential” use case every specific user might come up with. Sorry for falling short for you on this one, doing my best to make things work for your use case.

Also appreciate that we had to redesign WL8 completely from scratch without having the WL6 code base to work from. It actually began life as a different product that was intentionally differentiating itself from the Fidelity owned WL6.

This is not just my use case (or maybe it is if I am the only one using the external Symbol feature), it is a software bug.

Here are the detailed steps to replicate the bug:

1. Create an ASCII DataSet named "A" with just two Symbols "QQQ" and "MSFT" with a start date 10/31/2024 till today for both of the symbols.
2. Create an ASCII DataSet named "B" with just two Symbols "QQQ" and "MSFT" with a start date 10/31/2010 till 9/2/2020 for both of the symbols.

Please note that "A" and "B" DataSets have different dates that do not overlap.

Go to DataManager > Historical Providers and check mark only the "ASCII A" DataSet.
Run this script on a Symbol "MSFT":

CODE:
using WealthLab.Backtest;
using WealthLab.Core;
using System.Drawing;

namespace WealthScript6 
{
    public class MyStrategy : UserStrategyBase
    {
        //create indicators and other objects here, this is executed prior to the main trading loop
        public override void Initialize(BarHistory bars)
        {
         //Plot QQQ in its own pane below price
         BarHistory qqq = GetHistory(bars, "QQQ");
         PlotBarHistory(qqq, "QQQ");      
        }

        //execute the strategy rules here, this is executed once for each bar in the backtest history
        public override void Execute(BarHistory bars, int idx)
        {
        }
    }
}

This is what you get:


So far so good.

Go to Data Manager > Historical Providers, uncheck the "A" and chekmark the "B" DataSet.

Come back to the Strategy window, click on Strategy Setting and you should have "MSFT" in the Symbol field already typed there. Click on ">" to run the script. Click on the "Chart" tab.

Bug Number 1: nothing changed even though we already changed the "Historical providers" to "B". WL8 should have pulled it from "B" but it is still pulling from "A". In order to force it to pull it from the "B" we need to type not "MSFT" but "QQQ" and it would pull it from the "B".

Okay, now click on Strategy Setting again and type "QQQ". Click on "Chart" and this is what you see:


Bug Number 2: as you can see in the picture, we are missing bars of the "QQQ" on the "QQQ" pane. Why is that? It is because WL8 is trying to pull the "QQQ" still from the "A" DataBase instead of pulling it from the "B" that primary is pulling right now.
After restarting WL8 this is what you get:

As you can see, WL8 started pulling the external from the correct "B" DataSet where the primary Symbol is coming from.

The Flaw in Design: How am I supposed to work with two Strategy Windows while each is connected to its own A and B DataSets? Is it even possible to achieve this since there is a single global control over the Data Source for the entire application at the time?

>>Bug Number 1: nothing changed even though we already changed the "Historical providers" to "B". <<

Not a bug. It's working exactly as designed. WL8 won't request data again after you run a backtest unless you change the symbol or the DataSet selected. Just re-configuring the Historical Providers isn't enough. If you want to force WL8 to reload the data, click the text "1 Symbol Loaded" and that will clear the cached data and force a reload.

2 is not a bug either. It's just how WL8 is designed. It doesn't matter what DataSet you clicked, that doesn't impact where the secondary symbol is downloaded from.

As I mentioned earlier yes it will be possible to achieve by using the new BarHistory.DataSet property that I added just for you. You can make your call like this after Build 115 is out:

CODE:

        public override void Initialize(BarHistory bars)
        {
         BarHistory external = GetHistory(bars, "ES", bars.DataSet.Name);
         PlotBarHistory(external, "External");
        }

If 2 is not a bug, how can I know where "external" is being loaded from?

It will load based on the order of the Historical Providers, unless you specify the DataSet name as the optional third parameter.

If you're seeing that the symbol doesn't load from the provider you expect after changing the order, then the reason is we also temporarily cache external symbols to avoid delays during backtests. If we keep loading the same symbol again and again on a multi-symbol backtest it would really hurt performance. Your use case is different because you're requesting the same symbol, but you expect it to come from different DataSets. The most fool proof solution for you will be to wait until Build 115 is released (I'll try to get it out tomorrow) and specify the DataSet name using the BarHistory.DataSet property.

But it does not, as I have shown in the example above. Only restarting WL8 helps.

There is no rush on it, as long as we get it right.
I think WL8 should differentiate between Single and Portfolio backtests. Single Backtest should reset the cache on every run.
Also, a cache needs to be reset once before the Portfolio backtest starts.

Does clicking on "x symbols reloaded" also force reload external symbol data?

>>Does clicking on "x symbols reloaded" also force reload external symbol data?<<

No but I think it’s a no brainer to have it do so, I’ll make that change!

@Glitch. Thanks, that's what I noticed with one strategy I have that trades with external symbols. I had to restart WL after new data sync.

Is it a good idea to add a parameter to GetHistory to control whether one wants to force reload data? Then one could programmatically control the behavior.

I think that external should always be loaded from the same DataSet as the primary Symbol by default unless otherwise specified in the code by the user.

My first instinct is leaning toward thinking that’s ok. We’ll give it a try for next build.

It's in place. GetHistory will now first request the data based on the DataSet of the clicked symbol/DataSet. I'll need some time to complete testing so will target a release for tomorrow.

Did you get a chance to see if the changes in Build 115 helped you out with this?