DataSet-Providers-of-Norgate-and-WealthData-with-identical-names-not-visible-in-Strategy-Portfolio-Backtest-dropdown

We've been there before, looks like an intermittent issue. Did you restart WL?

https://www.wealth-lab.com/Discussion/Synonymous-Norgate-DataSets-missing-from-Select-Backtest-DataSet-9601

Cannot duplicate it so far:

WL has been restarted countless times.


The issue is real:


And its only in the Portfolio Backtest dropdown menu of a strategy, not in the dataset tree in side panel nor in the Strategy Monitor. Very strange.
--------------------

Edit:
In the dataset tree in the side panel and in the Str. Monitor all the datasets are listed under their parent folders so there is no overlap with WD datasets. Its only in a strategy's Portfolio Backtest dataset dropdown menu - where *all* datasets are shown in alphanumeric order, irrespective of their parent folders - that this issue is present.

I was wondering... which Assembly Reference(s) I can include that will override WL's default behavior of only displaying one dataset with a unique name and force it to display all?

I don't know what you're talking about with the assembly reference. Where would you put this reference and what would be using it?

It's possible you have a DataSet Provider that is causing a conflict. It could be helpful to know your configuration of Extensions - a screenshot of your Tools > Home Page > Version info ?

By assembly references I meant the ones available under Tools | Assembly References.


Here's the snapshot you requested:

I don't doubt you that the issue is real for you, was just reporting that I haven't been able to duplicate it.

I had been scratching my head why you were not able to duplicate it... the only thing I could come up with was that you, as the chief developer, likely use quite a few assembly references - perhaps even some custom ones - that the average user like me doesn't hence my Q about assembly references.

I don't have any assembly references checked off, and I think it's doubtful that this would play a role. If you hide the WealthData provider then do you see all of the Norgate DataSets, and vice verse?

With Norgate Data (ND) Provider hidden + WL restart, all ND datasets become invisible:



With Wealth Data (WD) Provider hidden + WL restart, WD datasets become invisible and all ND datasets become visible including those previously not visible:


Clearly, on my PC WD is preventing ND datasets with identical names from becoming visible.

In order to pinpoint whether identical-named datasets (DS) are not getting added to DS list or whether its a retrieval/display issue I ran a check:

CODE:
      public override void BacktestBegin()
      {
         //list the DataSets
         foreach (DataSet ds in WLHost.Instance.DataSets)
         {
            WriteToDebugLog(ds.Name + "\t" + "\t" + "\t" + "\t" + ds.DataSetProvider);
         }
      }

Here's a section of the printout:

(compare with screenshots above)

It would appear that identical-named DS don't even get added to the DS list.
Also noteworthy is that while WealthData gets shown as a DS provider (where applicable), the other providers are not listed (except for Yahoo's world indices which are listed as WorldDataSetProvider) and I do have DS from other providers other than Norgate such as MorningStar, Q-Data, etc.

Dear Developers,
Were you able to confirm that not all Dataset (DS) Provider names are being cached? Maybe, just maybe, that's the root cause of what I'm experiencing.
- If you can confirm that then I'll wait for it to be fixed.
- If you're unable to confirm that also then there's something wrong with my WL install and we can talk about reinstalling it and the safest way to do so.

Hi Sammy, I haven’t been able to duplicate your issue on my WL and haven’t yet come up with an idea of what might be causing it. I’m traveling at the moment but will spend some time later and see if I can come up with any ideas.

As a workaround for now you can hide the provider you don’t need to access any DataSets that were rendered inaccessible.

Thanks, I'll wait.
In the meantime I'll use Norgate's ASCII exported datasets in lieu of the missing ones.
-------------------------------------------------

Incidentally, I just discovered that the missing datasets are also missing elsewhere e.g. in Strategy Evolver's dataset selection dropdown menu. There may well be other places/tools where they're missing, haven't checked.

This issue is wide-ranging.

We’ll yeah, it makes sense that they’d be missing wherever they’re being populated.

I wonder if this is effecting any other users.

Good news, I found the cause whilst on a flight and it should be resolved for Build 36!

AWESOME!!
Can't wait for Build 36!

Using B37.
The issue is still present - with all providers activated (none hidden) ND datasets with same name as WD datasets are not visible:


Only when I hide WealthData provider and restart WL do the identical-named ND datasets become visible:



Edited -
Even though B36 was short-lived I remember doing a quick check and do believe I saw *all* the datasets so the issue was fixed... but it has re-appeared in B37 even though it was released as a fix for an unrelated issue.

UPDATE

During the past week (using B37 and now B38) I've periodically checked for the invisible folders (those with duplicate names) and here's what I found:
- The issue is definitely present but is inconsistent.
Most of the time they're not visible (and that's the current state on my PC) but occasionally all folders do become visible. I haven't been able to pinpoint the circumstances that lead to the latter state - but even if I could one can't leave this important issue to the vagaries of chance; rather, it needs to be fixed solid.
----------------------------------------

Apart from explaining what Glitch himself noticed - he didn't notice the issue on one PC but did notice it on another (while traveling) - perhaps the inconsistent nature of the issue itself can provide the developers with a clue as to what may be going on... e.g. is it a caching issue? or perhaps the folder used for caching is an unprotected folder and therefore may be overwritten or emptied out during the day??? etc. (just throwing out a few possibilities).

Hopefully this gets fixed soon.

Just a correction, I have never personally encountered the issue on any PC.

By the way, I checked - there were no folders either. Rebooted the program - appeared. Rebooted again - no more. And the fourth time I rebooted - they did not appear.

@Glitch,
I thought when you wrote in Post #16 that you found the cause on a flight it meant you confirmed the issue on your laptop. My bad.

@fred9999,
Yup, its totally unpredictable.
I have restarted WL, as well as my PC, countless times... there's no knowing when they'll appear... or disappear.

QUOTE:
Yup, its totally unpredictable.

I'm a little confused on why this problem is nondeterministic. Are we saying there's something in the Norgate provider that's not thread safe and needs to be? Is there good reason for the Norgate provider to be using asynchronous threads simply to populate (i.e. display) a directory tree? (I wouldn't think so.)

I wouldn't introduce asynchronous threads unnecessarily. But the problem is WPF runs asynchronously from the WL application by default, which makes me wonder if the WPF implementation has an implementation issue. Try closing and reopening the directory tree while placing the WL application in different states to see if this has an effect. Perhaps there's an "esoteric" WPF dependency that's not covered in a special application situation. (Can the directory tree have more than one display state?)

@superticker may have some hunch here. WL does load the DataSetProviders asynchronously on startup. Maybe it all has something to do with a delay loading Wealth-Data's DataSets (unlike Norgate, this involves network and depends on performance of the server which hosts WD) or something like that. Just a wild guess.

QUOTE:
WL does load the DataSetProviders asynchronously on startup.

And do the built-in Norgate datasets get loaded from a different server than the WealthData datasets? Why are only Norgate datasets affected?

This is all very "speculative" and should be taken with several grains of salt. The WealthData datasets come from a distributed (i.e. cloud-based Azure) database, so they are closest to the WL client and get loaded first. Perhaps the Norgate datasets come from a single-source database far away, so they get loaded later. Moreover, perhaps there's a missing dependency property in the WPF directory-tree code, which would repaint the directory tree if datasets got populated later in WL. This would mean Norgate datasets would be missing in the directory tree--sometimes, but would still be available to the WL application.

And this is just one possibility to explain this nondeterministic behavior. I can think of several others. I think the first thing to do is to determine if this is a WPF problem or a WL application problem. Those having this problem need to provide more clues.

QUOTE:
And do the built-in Norgate datasets get loaded from a different server than the WealthData datasets? Why are only Norgate datasets affected?

They aren't loaded from a server, it's a local DB. Affected they may be due to the identical DataSet names. Just a speculation.

I don't understand why remote WealthData datasets are loaded instead of local ones. After all, we are just adding existing datasets to the dataset directory.
Any update to symbols contained therein or their data, or addition of new datasets, is a separate matter - and we have the WL Downloader for that.

QUOTE:
I don't understand why remote WealthData datasets are loaded instead of local ones.

Indeed, there is a local cache for WD DataSets which is used when you go offline but these DataSets are inherently dynamic due to being surviviorship bias-free so the idea is to download them on startup to catch up the latest composition.

Expanding on Eugene's reply above, here's what I think is happening - and how it may be fixed:

When WL starts, during bootup it creates a directory of the datasets (DS) which is then cached.
During this process of directory creation it calls upon the WealthData (WD) servers to refresh the list of WD datasets/constituents...
- if the response is received in the allotted time all is good and a proper list of datasets is created
- if the response is delayed then the response received from WD servers overrides the prev created list and the identical-named DS from 3rd party providers get overwritten by WD datasets of same name

(A) The problem with updating datasets on WL startup: Here are some of the factors that can affect the time for a response from WD servers (this is not an exhaustive list by any means):
- internet speed, latency, or down
- WD servers load, latency, or down
- superimposed networks (workplace intranets, VPN, and so on)
- use of QoS (Quality of Service, a fancy term for network prioritization/throttling) at the router or app level
- Network card settings
- antivirus/security software settings
- etc.
Their variable impact on the process may explain why sometimes the list is complete and sometimes incomplete.
One could fix this by increasing response time and/or ensuring that a delayed response does not result in overwriting the directory list... but wait, a simpler solution exists, see (B) below.

(B) Solution: Use local datasets: Using local DS is a simple & robust solution that is unaffected by the factors listed above. These are automatically updated whenever one uses the Downloader to download data (which includes datasets and constituents).
=============================

TLDR
It appears during dataset directory creation WL uses remote access to WD servers which is subject to tripping by a whole slew of factors. Besides, the benefit of remote access is unclear as new datasets are created very infrequently/ can be downloaded any time via the Downloader/ and the next time WL starts they will be represented in the local datasets anyway.
Using local datasets, on the other hand, is simple and virtually foolproof.