DataSet-symbols-disappearing-duplicating

Let's wait until you have a step by step procedure to duplicate. Since there is absolutely no detail, evidence or even a screenshot, this might be some user error.

Happy Holidays!

I took screen shots of my symbol set on Dec. 25 and today, Dec 27. Four strategies ran against this data set today, all launched simultaneously by the SM. I had to take 3 shots on each day to get complete symbol set.

On Dec. 25, DataSet was intact. Today I noticed the following changes. I did not manually change the symbol set or open it except for this check.

Symbols duplicated (8)

DKNG
GOOGL
MRK
OXY
PYPL
SDC
SPCE
XLE

Symbols deleted (8)

F
GM
MRNA
MU
NFLX
RBLX
SOFI
SQ

Dec 25




Dec 27

Happy holidays! That's a start, thanks. Now I'd be interested to find out more about the strategies since you're using StrategyRunner and probably other above average features. It's crucial to understand the context and possibly duplicate. What's specific about them? How the SM is set up? Many users run multiple strategies in the SM and you're the first to report so supposedly this may have something to do with your setup.

Looking for potential causes...
Are you using the default Data folder location? (..\AppData\Roaming\WealthLab7\)
Check it in File > Set Data Folder Location

Anything else special that you can think of about your environment? Pen drives? AWS?

QUOTE:
Are you using the default Data folder location? (..\AppData\Roaming\WealthLab7\)

Yes. I see DataSets there. Is that what you are looking for?

Here's what's in the DataSets file for that DataSet. The entry appears uncorrupted and does not have the duplications or deletions I noted above. Could this be a display issue with the DataSets/MyDataSets window? Note that missing symbols are not processed by strategies and duplicated symbols are processed twice, just as displayed in the DataSets/MyDataSets window.

1|419|Most Active (Daily)14|Yahoo! Finance382|AAPL,ABNB,AFRM,AMC,AMD,AMZN,ARKK,BA,BABA,BAC,BB,BBIG,C,CCJ,CCL,CLF,CLOV,COIN,CSCO,DAL,DIA,DIS,DKNG,EEM,EFA,EWZ,F,FB,FCEL,FCX,FUBO,FXI,GDX,GLD,GM,GME,GOOGL,HYG,INTC,IWM,JD,JPM,KWEB,LCID,LVS,MARA,MRK,MRNA,MSFT,MU,NFLX,NIO,NVDA,OXY,PBR,PFE,PINS,PLTR,PLUG,PTON,PYPL,QS,RBLX,RIOT,ROKU,SDC,SLV,SNAP,SOFI,SPCE,SQ,T,TLRY,TLT,TSLA,TSM,TWTR,UAL,UBER,UPST,V,VIAC,WFC,WISH,WMT,X,XLE,XLF,XOM,ZM,382|AAPL,ABNB,AFRM,AMC,AMD,AMZN,ARKK,BA,BABA,BAC,BB,BBIG,C,CCJ,CCL,CLF,CLOV,COIN,CSCO,DAL,DIA,DIS,DKNG,EEM,EFA,EWZ,F,FB,FCEL,FCX,FUBO,FXI,GDX,GLD,GM,GME,GOOGL,HYG,INTC,IWM,JD,JPM,KWEB,LCID,LVS,MARA,MRK,MRNA,MSFT,MU,NFLX,NIO,NVDA,OXY,PBR,PFE,PINS,PLTR,PLUG,PTON,PYPL,QS,RBLX,RIOT,ROKU,SDC,SLV,SNAP,SOFI,SPCE,SQ,T,TLRY,TLT,TSLA,TSM,TWTR,UAL,UBER,UPST,V,VIAC,WFC,WISH,WMT,X,XLE,XLF,XOM,ZM,5|False4|True0|5|False

I notice Dropbox on your screenshot. Is the user data folder shared between applications / devices somehow?

QUOTE:
I notice Dropbox on your screenshot. Is the user data folder shared between applications / devices somehow?

I don't have any WL files or folders in Dropbox and can't find any in the OneDrive backup. However, I am using Windows backup and I remember backing up some WL7 data files. (I am challenged to find my selected backup folders in Windows 11). However, these should be backed up to an external drive and not synced / duplicated.

It looks like an external modification of the file. Do you happen to run multiple instances of WL7?

QUOTE:
It looks like an external modification of the file. Do you happen to run multiple instances of WL7?

Not that I am aware of.

I was not able to reproduce the problem by running strategies, including the one with StrategyRunner. However, I have reproduced the problem twice by shutting down WL7 and restarting, before running any strategies.

Can't tell what happens on shutdown yet but something is out of whack. The DataSets.txt modification date on the screenshot from Post #5 is 12/24 while its composition in Post #2 changed later.

I see that. Since what's in that file does not reflect what I see in V7, is there someplace else V7 can pull the symbols from? Anything else you want me to check locally to see if I can confirm the corruption in a local file?

Does your Strategy code manipulate DataSet objects by any chance?

No, I don't think so. Only connection to Db is through strategy setup UI. Although the application that uses StrategyRunner references the database as follows:

CODE:
         sr.DataSet = DataSetFactory.FindDataSet("Most Active (Daily)");

In case it might be significant, the child strategy also uses this same dataset, which is selected by the strategy set up in the child strategy listed below:

CODE:
         bt = sr.RunBacktest("MD K% for SR")

Re: FindDataSet. Now that's a clue. This is along the lines with what I mentioned in Post #3 re: StrategyRunner.

QUOTE:
This is along the lines with what I mentioned in Post #3 re: StrategyRunner.

Yes. However, with additional testing I now see that every time I restart V7, I get different symbols that become duplicated, and apparently the deleted ones are restored. The symbol count never changes. This occurs without running any strategies between restarts.

eugene can correct me if i’m wrong, but i think those yahoo datasets (most active) are dynamic so i’d expect the symbols to change. but maybe he can investigate the duplicate symbols?

@Glitch. Note that a duplication is always accompanied by a deletion on a 1 to 1 basis.

Dion, although topic starter's DataSet belongs to Yahoo, I don't recognize it as a dynamic one (i.e. part of the "World Indices dynamic DataSetProvider"). It's manually created.

Perhaps a deeply rooted bug inside DataSetFactory or something?

This got weirder.

On the hunch the DataSet file was corrupted, I changed the DS name to [name]-old and created a fresh DS. Without running any strategies, I continued to restart V7 and check both DS's.

The renamed DS is now perfectly fine, and the fresh DS now exhibits the symptom of duplicating random symbols on each restart. It appears that the name is cursed! But I know that there must be a more technical explanation, like the fact that I'm using this DS name in strategies.

The only common thread to duplicating the problem I saw was the restart, so I decided to experiment with the strategies in the Strategy Monitor (SM). I tried dozens of permutations and discovered a repeatable pattern.

First some background. I have 5 strategies activated in the SM. Four of them use the same DataSet (DS) that is being corrupted. Two of those four are rotational strategies.

I started deactivating strategies and saved the workspace as default before each restart. When I deactivated all the strategies, the problem went away. When I activated multiple strategies that use the problem DS, the problem returned. Finally, I was able to repeat the problem with a combination of a rotational strategy and one or more other strategies (that used the same DS) both being activated on startup.

I tried running these strategies individually and at the same time from the SM, but that did not exhibit the problem. It seems to be related to the fact that they are activated. The repeatable pattern I found is that when one of my rotational strategies is activated along with another (rotational or non-rotational strategy), the problem appears. Otherwise, it doesn't.

I hope this makes technical sense and helps to guide what might be going on.

QUOTE:
Here's what's in the DataSets file for that DataSet. The entry appears uncorrupted and does not have the duplications or deletions I noted above.

Wrong. The raw DataSet entry contains two DataSets inside it, it's perfectly corrupt. You should delete it and recreate, making sure it doesn't contain any "service bits":

QUOTE:
1|419|Most Active (Daily)14|Yahoo! Finance382|...ZM,382|AAPL,...,ZM,5|False4|True0|5|False

Ideally you'd rename the WealthLab user data folder and start from scratch, leaving out any user coding (which may look suspicious at the moment) and running only Blocks / Rotation strategies on the recreated DataSet.

I deleted and started to recreate the DataSets. This what I see in the DataSets file after entering the first oneS. Symbols are appearing twice. Is this the same problem you highlighted?

CODE:
1|46|IV ETF14|Yahoo! Finance11|^GSPC,^NDX,11|^GSPC,^NDX,5|False4|True0|5|False

CODE:
1|44|Test14|Yahoo! Finance18|AAPL,AFRM,AMC,AMD,18|AAPL,AFRM,AMC,AMD,5|False4|True0|5|False

I stand corrected. It turns out that the symbols look duplicated in DataSets.txt for some good (technical) reason.
This is intentional and does not indicate corruption. Sorry for the misleading comment.

No worries, Eugene.

What do you mean by "rotation"? The SM does not support rotation strategies. Most likely this may be some bug with your code (that we don't know). Try to reproduce the issue with a sample strategy. That's the shortcut to duplicating and fixing it.

QUOTE:
What do you mean by "rotation"?

I read and commented on a post about rotation strategies not running properly in SM https://www.wealth-lab.com/Discussion/Rotation-strategies-in-Strategy-Monitor-are-not-possible-5790

I wasn't sure if this was the "Blocks" type or C# coded. I am running a couple of C# based rotation strategies, one modeled on an RSI example provided in V7, using daily data. They both run fine in the SM if I run "Run Now" them. I can share one of the programs with you if you like.

QUOTE:
Most likely this may be some bug with your code

I'm not sure what "bug" I'm trying to find, as these run fine for me and yield same results as they did in V6. Maybe you suspect that activating rotation strategies in the SM to causes the disappearing / duplicating symbols issue due to some anomaly in my code? Remember, I don't have to run the code to recreate the problem. But I'm glad to share the code with you if you would like to take a look.

My thinking is that there is a chance of some subtle bug with concurrent access in our code which manifests itself when using the StrategyRunner and DataSetFactory the way you're doing. Remember this:

CODE:
DataSetFactory.FindDataSet("Most Active (Daily)");

QUOTE:
But I'm glad to share the code with you if you would like to take a look.

Thanks but what I'm proposing here is to put aside anything that makes possible to shoot oneself in the foot with (custom code) and try running Blocks strategies on that DataSet to see if the problem can be reproduced. If not, it's the custom code.

Don't mean to be repetitive, but I want to make sure we are on the same page about how to reproduce this problem. You don't have to run any programs (custom or otherwise) to see the problem.

What would help is some clear step by step method to reproduce the issue. I can't reproduce this, and don't know what to do to get it to manifest. If you could boild things down to some specific steps, and provide the Strategy if required, it would be a great help.

QUOTE:
You don't have to run any programs (custom or otherwise) to see the problem.

Since it's a unique report, it's that your code somehow using FindDataSet & StrategyRunner may be the source of the error. And if it's not reproducible with plain vanilla Blocks strategies, you'll know it for sure. That's my logic.

@innertrader while i can't say I've encountered your specific situation I did encounter a similarly perlexing issue with the FindDataSet call in my own code a while back.

The cause ended up being the presence of non-alphanumeric "special" characters in the dataset name. That is ")!%&@#>" type characters. Hyphens (dashes - ) and Spaces seem to be ok. I haven't thoroughly tested all special characters.

Note: The datasets would update without issue and strategies functioned without fault. But when I used the FindDataSet call I encoutered unexpected and seemingly random behaviour.

Once I simplified the dataset names all the seemingly random issues disapeared.

I wonder... If you change your dataset name to something simple and update your code does the issue persist?

@splodge
It's an interesting observation but FindDataSet is among the most frequently called methods under the hood, plus a very simple one to have glaring bugs.

@innertrader
Though you never know so you might want to remove the "( )" from your DataSet's name.

@Eugene you are probably right. The issues I saw were many builds ago now (nearly 20 in fact) so it's possible the cause was corrected in one of the many tweaks that have happened since then. :)

@InnerTrader - other thoughts below: (and the WL Crew need to chime in here and correct any mis understanding I might have here)

You mention using the strategy runner (as a child strategy), in addition to a rotational (or other) strategy activated with the StrategyMonitor.

The SR requires the DataSet to be specified before calling the backtest (Which you show in your code). However, If I am not mistaken, calling a backtest will attempt to update data on a case-by-case basis if not already up-to-date. The fact that it happens on "restart" of WL7 might suggest a dataupdate is firing?? Is it possible this is creating the hidden conflict... one strategy trying to read the data/dataset while another is trying to update it.

If you deactivate the SR strategy does the problem persist? or test deactivating all your strategies and ONLY using Sample Strategies that access the problem dataset (as per Eugene's suggestion above)

QUOTE:
Try to reproduce the issue with a sample strategy. That's the shortcut to duplicating and fixing it.

Alternatively, you could duplicate the dataset (with a different name) and test with the parent strategy pointing to the original and the backtest pointing to the new version... does that eliminate the issue? If so, then you may have found the confilct.

Good Luck. ;)

QUOTE:
What would help is some clear step by step method to reproduce the issue. I can't reproduce this, and don't know what to do to get it to manifest. If you could boild things down to some specific steps, and provide the Strategy if required, it would be a great help.

Please see Post #21

@Eugene @splodge thank you for your suggestions. I will try and report.

Please read Post #21. It doesn't seem like I'm getting through with my findings. I can reproduce the problem without running any program. All that's required is that some combination of my strategies are activated in the Strategy Monitor (SM), and I restart V7. I don't have to run any of them from the SM.

I wonder if the SM does some magic when V7 starts up that may actually conflict with some code in an activated strategy, even if doesn't run it? Or maybe the SM doesn't like StrategyRunner programs? (I'm fishing here.) What makes it difficult to nail down is the problem seems dependent on some combination of different strategies being activated. I've noted my finding in Post #21.

One other clue that might be meaningful is that @Eugene confirmed that the source DataSet file itself is *not* corrupted. It just appears like it is when you open the dataset in the dashboard to see the symbols. However, programs run as if the symbols are corrupted (missing / duplicated), so I'm guessing there's some sort of caching going on when V7 starts up that the SM might be involved in. It looks like there is some symbol cache memory that is getting corrupted.

QUOTE:
Or maybe the SM doesn't like StrategyRunner programs?

What happens if you comment out the SR config/calling code ?

I missed Post 21, I’ll try out the scenario and see if i can duplicate the issue, thanks for providing those details. in the mean time, it would be interesting to see if having each strategy use its own unique dataset results in any issues at your end. the datasets can contain the same symbols.

QUOTE:
All that's required is that some combination of my strategies are activated in the Strategy Monitor (SM), and I restart V7.

Activated means they might run. And if your unknown code-based strategies use the two features named in my Post #31, there may be risk.

QUOTE:
Activated means they might run.

Yes, of course you're correct. But they didn't run as far as I can tell. (Unless SM performs some sort of invisible "test" run on startup.)

The symbol corruption was visible immediately after startup and before any "visible" SM run of these strategies.

Here's another symptom I noticed this morning. XLE looks clean in the expanded symbol list, but it is triplicated in the edit section. Not sure if this might help isolate what may be happening.



Symbol count never changes so two other symbols are now missing.

Restart with no strategies activated in SM cleans it up.

Did we clarify that the problem occurs when Strategies are activated in the SM and WL starts with the SM in a default Workspace?

I've had a chance to test some of the suggestions I received to troubleshoot this problem and here are the results:

1. @splodge @Eugene I removed the special chars from the DataSet name, "(Daily)" to "Daily", but it didn't make any difference.
2. @Eugene I rewrote one of my strategies to eliminate the need for StrategyRunner but it didn't make any difference.

If I activate my 5 strategies (including 2 rotation strategies) in the Strategy Monitor (SM) and restart V7, the problem will appear. If I "Run all Daily+ Now" the problem will also appear.

If I "Run Now" each strategy individually from the SM the problem *does not* appear. It seems to be dependent on some unknown interaction of the 5 strategies in the SM. For now, I will not use the "Run all Daily+ Now" function or Activate these strategies in the SM.

Let’s try to isolate it more. What if you only run the 2 rotation strategies instead of 5 strategies? If we isolate it to these strategies then perhaps you could email them to us and we can try to duplicate the issue?

I first tried activating both my rotation strategies in the Strategy Monitor (SM) and restarting V7. Didn't exhibit the problem. Both strategies ran automatically this morning and results were accurate, no problem exhibited in DataSet.

I will have to try some other combinations of strategies activated in the SM to see which minimal set creates the problem.

Thanks, it will really help if we can nail down the strategy/strategies that are causing this!

@Glitch I have been working to find a combination of strategies that would reliably reproduce the issue. I thought I had it narrowed down to two of them. But after repeated confirmation testing, I find there is an element of randomness to displaying the problem, which is not good news when it comes to troubleshooting.

I cannot get my candidate set of strategies to fail 100% of the time. But I can get these to fail reliably if I test them several times. I can't be sure that these are the *only* two combinations that will show the problem, due to the randomness.

However, I can send these two strategies to you, if you would like. It doesn't take long to run a test. I can send instructions and the DataSet they are using. At a minimum, you can see if you can reproduce the problem on your side. Let me know.

Yes. please send it over!

support@wealth-lab.com ?

yes, thanks

We're looking into what now seems to be a cosmetic issue with the left hand DataSets tree.

1) The DataSet composition in the Data Manager is intact.
2) The "corrupt" DataSet self-restores itself on next startup if you do nothing.
3) According to debugger, your strategies running in the SM operate with the correct (unaltered) copy of the DataSet.

QUOTE:
3) According to debugger, your strategies running in the SM operate with the correct (unaltered) copy of the DataSet.

Interesting. I was getting anomalous trade signals that I traced back to the DataSet corruption, which is how I discovered it. Have you tried running the original version of the strategy (outside the SM) after you see the corruption, to see if this is true for this use case as well? I was seeing differences in signals running strategy in SM or standalone.

QUOTE:
Have you tried running the original version of the strategy (outside the SM) after you see the corruption, to see if this is true for this use case as well?

Yes, I ran both strategies via "Open in strategy window" and got identic Signals.

QUOTE:
Yes, I ran both strategies via "Open in strategy window" and got identic Signals.

Upon startup of WL7 this morning, the DataSet was corrupted so I was able to test the above. My results confirm yours in that the signals are the same whether run in the SM or standalone.

However, I likely misremembered my previous observation.

When I once again restarted WL7 to get a clean DataSet, I re-reran this test. Once again, the run in SM and standalone gave the same signals.

However, these runs did not have the same signals as the runs with the corrupted DS. This is most likely what led me to dig into the cause of the problem initially: getting different results on different runs, particularly after restart.

QUOTE:
We're looking into what now seems to be a cosmetic issue with the left hand DataSets tree.

Based on my tests today, it doesn't appear that this issue is purely cosmetic.

That's great new! Let me know if you want me to test a pre-release.

@Glitch I was perusing this thread and see that you posted this on Jan 10

QUOTE:
Did we clarify that the problem occurs when Strategies are activated in the SM and WL starts with the SM in a default Workspace?

It's probably too late to be relevant and I'm not sure if your question was directed to me, but if it was, I'm sorry I missed it until just now. Yes, that is a concise description of how to duplicate the problem.

QUOTE:
We think we have this finally nailed down, thoroughly testing the fix as I type.

Was a fix for this issue in a recent release?

No, it's coming in Build 50.

Excellent. Happy to beta test if that would be helpful.

We’re going to release it tomorrow, just finished the Build video!

This was fixed in a build update