Please-restore-Data-Tool-in-WL7

The Data Tool's design in WL6 was dictated by our inability to commit to the main code base and Fidelity's absense of interest in development effort. It's not coming back as we know it but we do plan to integrate its features selectively if they make sense in the new paradigm. (By the way, one feature already exists as right click on provider to "Delete local files".)

What features of the Data Tool do you lack the most? And why? We'd like to hear opinions of DT users here.

I'm flummoxed why this tool won't be brought back.

Anyway, to answer your question:
1) Data truncation (incl Remove all data).
2) Remove inactive symbols (incl sub-window: Also remove symbols with last data before {user-defined date}). Currently, I see no way to delete inactive symbols other that to remove than one by one from the dataset, which is very tedious; a right-click option to delete individual symbols would be better than nothing... but the best would be the en block removal via Data Tool.
3) {Wealth-Lab Pro had this feature, but outside of the Data Tool, and applied to Fidelity data only} Change a symbol across all datasets. [This would be a wish-list item.]

No need for alarm. As Eugene said plainly above, we will integrate “Data Tool” selected features into the Data Manager.

2) Removing inactive symbols.
4) Fidelity's feature of renaming symbols that got their name changed.

Making the Data Tool a plug-in, as with WL6, is a nice way a doing it. Especially if you can publish the API so other programmers can also add their own plug-in features for the datasets.

The Data Manager already is plug in enabled. Each Provider can expose a custom UI in the Settings panel. Also I don’t see why we need another dedicated “Data Tool” when the Data Manager already is a Data Tool. Let’s keep things simple and start integrating desired features into a single “Data Tool”.

We're leaning towards smarter ways of accomplishing these tasks. For example, alpha versions of WL7 had a feature (now removed) which reloaded the historical data for a symbol behind the scenes automatically on new split/dividend in fundamental data to ensure there is no gap.

1) Data truncation (incl Remove all data).

"Remove all data" for provider is already there, please see Post #1 for a how-to. Could you clarify why do you think you need to truncate data by DataSet in WL7? To better understand the use case, please provide a specific example (data provider, bar scale, symbol).

2) Remove inactive symbols

OK noted. We'll think about ways how to achieve this - maybe automatically? For example, with a global option "hide inactive symbols from portfolio backtests" - not investigated yet.

4) Fidelity's feature of renaming symbols that got their name changed.

Thanks, "Rename symbol" is on the list of things to do.

3) Change a symbol across all datasets

Thanks for your suggestion. Not yet sure if it's feasible considering the implications but I'll look into it.

Anyone following this thread may find the following links useful:
1) Forum discussion on Data Tool (pre-WL7), goes back years:
https://wl6.wealth-lab.com/Forum/Posts/Data-Tool-utility-for-data-validation-truncation-DataSet-cleanup-31991
2) Data Tool wiki page:
https://www2.wealth-lab.com/WL5WIKI/DataTool.ashx
----------------------------------------------------------

Data truncation
- For a DataSet: (copied from wiki) "Remove all data" is helpful if you want to reload the historical data of an entire DataSet but are reluctant to perform "Reload All History" manually. We know how tedious it could be when many symbols are involved. Click this button to nullify the accumulated data in highlighted DataSet, and then use Data Manager's "Update" button as you would do normally - the entire DataSet will be reloaded from scratch.
- For an individual symbol: For example, if you a recycled symbol (one that was used previously for another security) then either your local data or the provider's data may include data for both securities, including the time gap in-between (which may be years long), so you need to purge old data; Yahoo is a notorious offender. Another situation when you need to clean out old data and load afresh is after a complicated M&A or spinoff when price adjustments need to be made. I'm sure many other situations exist.

Incidentally, in my WL7 build 3, on right-clicking a DataSet I don't see an option to Remove Data, there is one for Delete DataSet which is not quite the same.

We will add "Rename symbol" (on per DataSet basis) in Build 4.

QUOTE:
Incidentally, in my WL7 build 3, on right-clicking a DataSet I don't see an option to Remove Data, there is one for Delete DataSet which is not quite the same.

Like I said, it works if you right click on provider in Data Manager (not a DataSet) and choose "Delete local files".

Re: Data truncation

QUOTE:
- For an individual symbol:

It's possible: right click on a chart, choose "Reload chart data from provider".

QUOTE:
We will add "Rename symbol" (on per DataSet basis) in Build 4.

We will??

QUOTE:
We will add "Rename symbol" (on per DataSet basis) in Build 4.

If you did the rename "by Provider" (like the Fidelity provider) as WLP6 did, that would be best. Just make it work like WLPro did. I used that feature constantly in WLPro.

Data truncation

QUOTE:
It's possible: right click on a chart, choose "Reload chart data from provider".

Good to know. However, there is no knowing when a data provider will cleanup their data, some may never do; in the meantime one's technical analysis is rendered useless while waiting.
--------------------------

Data truncation of an individual ticker BEFORE a date is as valuable as AFTER a date.

You may wish to review the pre-WL7 forum posts on Data Tool... this extremely useful tool was shaped by inputs from countless users over many years.

There are two sides of this coin. Personally, I don’t want data corrected that I’ve already made real world trades on. I don’t want to suddenly have one of my open positions mysteriously disappear and miss an exit signal. But I respect the desire of others that see it differently.

QUOTE:
Data truncation of an individual ticker BEFORE a date is as valuable as AFTER a date.

Porting features like Truncate indiscriminately is something we'd like to avoid. Wealth-Lab 7 has considerably changed in the way it operates with data so we need a more detailed argumentation to consider any additions to WL7. What we're trying to understand here is the real problem a legacy method/tool was solving and possibly act smarter. We could even make improvements at the core level, if required.

For instance, if someone tried to truncate Fidelity's 1-minute data (which always backfilled completely) before a date to speed up his backtest on a slower PC then this may no longer apply since WL7 has "Intraday minute data multiplier". Same for data gaps in Fidelity's data which doesn't exist in WL7 anymore.

Can you show by example a real use case which Truncate could help you with in WL7? Both "before" and "after" a date, with a WL7 data provider and symbol. Thank you.

QUOTE:
... show by example a real use case which Truncate could help you with in WL7? Both "before" and "after" a date, with a WL7 data provider and symbol.

I rarely used truncate, so please don't include this feature on account of me.

That said, when I have used it, it was to recapture stocks prices for the previous day that were filled in by the on-demand provider. If you don't do that with WL6, then that stock will not get correctly updated by the static provider the following morning.

A better solution would be for the static provider to pull corrections in for the last couple of days since the last static update. For example, if the static provider hasn't been run for 3 days, if should request prices for the last 5-6 days to check if there are any corrections even if the on-demand provider has forward filled some of those prices.

QUOTE:
That said, when I have used it, it was to recapture stocks prices for the previous day that were filled in by the on-demand provider. If you don't do that with WL6, then that stock will not get correctly updated by the static provider the following morning.

WL7 does not capture today's partial bar in the static data so this is no longer an issue - nothing to truncate.

QUOTE:
WL7 does not capture today's partial bar in the static data so this is no longer an issue - nothing to truncate.

I wasn't talking about partial bars. I mean if on-demand updates final bar stock price at 10pm (same trading day), then the exchange updates that price at 4am (the following morning). In this case, the static data returns the 10pm price the following trading day rather than the corrected price that occurred overnight.

"Truncate bars" would be an awkward afterthought for that. The correct way is to let the data provider handle it, applying corrections.

>> The correct way is to let the data provider handle it, applying corrections<<

Of course this must be made optional for people like me who do not want these corrections! So hard to please everyone :)

QUOTE:
Of course this must be made optional for people like me who do not want these corrections! So hard to please everyone :)

True. I would imagine this as a checkbox in Preferences > Data/Trading tab, optionally with a numeric up/down for correction depth in bars. Providers could then apply corrections, overwriting the specified numbers of EOD bars. They could opt in/out of the corrections mechanism.

Next on the list of items suggested for inclusion is "Remove inactive symbols". I don't like the notion of removing symbols from a DataSet either, altogether. This way we'd lose the data which may get overwritten by a new company under the same ticker. What if there was a temporary trading halt which caused the exclusion? Point is, we need to design it carefully.

"Remove inactive symbols"
This is meant to remove symbols with no history i.e. 0 bars.

Recycled Symbols:
This is where renaming a symbol would help (e.g. XYZ to XYZ_old, user-defined.)
Also, some providers just start appending new data to old data (ignoring the time gap), atleast initially until they fix it later (or maybe never). This is where Truncate data - both forward & back - would come in useful.

Of course, all these things only apply to compatible providers. Some, like Norgate, do not allow renaming or data editing and also delete old symbols (unless you are on their Platinum plan).

QUOTE:
Also, some providers just start appending new data to old data (ignoring the time gap), atleast initially until they fix it later (or maybe never). This is where Truncate data - both forward & back - would come in useful.

"Truncate data" will not come useful here as another update will append new bars to the end of the file.

QUOTE:
"Remove inactive symbols"
This is meant to remove symbols with no history i.e. 0 bars.

Dead symbol removal looks like a good candidate to work on.

QUOTE:
Dead symbol removal looks like a good candidate to work on.

Will this also work for ASCII files also?

Vince

No. Metastock and ASCII DataSets are not configurable. Also, certain providers are excluded (Finam, MFD, Dukascopy).

Data Tool is shaping up nicely for Build 9. Bracing ourselves for a new round of criticism! ;)

QUOTE:
Bracing ourselves for a new round of criticism! ;)

At least you know people are paying attention! ;)

Nothing escapes the (collective) eagle eye of the users!

Yes you need very thick skin to succeed as a WL admin :D