Default-bar-scales-for-bulk-data-update

Let's not make the already sophisticated data update mechanism even more complex and error prone. Such change would cause an extra preferences for "default bar scales" be exposed to the GUI and can break the bulk update enabled data providers.

Also the suggestion in the linked topic to truncate last few bars of data may help and won't break your bulk update.

This topic keeps coming up. Personally, I like the idea of having default time scales to update.

However, since WL7 will automatically look for the data when you need it, a) you really don't even have to update data, and, b) the issue we always come to is, "why download data that you're not even looking at".

Maybe we could compromise by assigning a default time scale on a Historical Data Provider (HDP) basis? DataSets linked to a HDP would, at a minimum, update the linked DataSets for default scale, if one is assigned.

QUOTE:
However, since WL7 will automatically look for the data when you need it, a) you really don't even have to update data, and, b) the issue we always come to is, "why download data that you're not even looking at".

Good point. There is hardly a need to update IQFeed data in bulk mode at all unless WL7 is used in some scenario that may be employed perhaps by a handful of power users. You can just run your backtests, the rest is managed.

Where the bulk update becomes necessary is the data providers that have special requirements for data update. For example, if a data provider cannot comply with Wealth-Lab always updating DataSet symbols in parallel it therefore has to collect the data scale by scale, symbol by symbol in Bulk Update mode.

IQFeed is a different animal when the Regular Session Only option is employed. We made the RSO option for IQFeed for 2 reasons:

1. Don't request/cache post market data that is ignored,
- and more importantly -
2. correct the intraday opening price to the official Daily bar opening.

Why is the correction necessary? It turns out that the time filter does not filter out pre-market "Form T" trades, some of which occur milliseconds after 09:30:00.

Consequently, IQFeed must update the Daily bars before requesting the intraday bars so that the correction is available. It'll do this automatically for a bulk update, but only if the data for the Daily scale had been requested previously. In other words, if you start fresh by running a backtest on intraday data, the opening corrections will not occur.

QUOTE:
Maybe we could compromise by assigning a default time scale on a Historical Data Provider (HDP) basis? DataSets linked to a HDP would, at a minimum, update the linked DataSets for default scale, if one is assigned.

I love the idea of assigning a default time scale on a HDP! Please make it to be able to assign more than one time scale. I will assign daily scale and 1-minute scale.

QUOTE:
Regular Session Only option

Is that option available on WL7? I cannot find where I can set an option for IQFeed plug-in on WL7.

QUOTE:
Why is the correction necessary? It turns out that the time filter does not filter out pre-market "Form T" trades, some of which occur milliseconds after 09:30:00.

Exactly. You have updated WL6 IQFeed extension for that purpose. Does WL7 plug-in support the correction?

QUOTE:
Consequently, IQFeed must update the Daily bars before requesting the intraday bars so that the correction is available. It'll do this automatically for a bulk update, but only if the data for the Daily scale had been requested previously. In other words, if you start fresh by running a backtest on intraday data, the opening corrections will not occur.

That means the bulk update (if it supports the default time scale) has to update the Daily scale first prior updating any other intra-day time scale.

QUOTE:
Is that option available on WL7?

Right here -


And, right. When you add new symbols or refresh IQFeed, the trick is to make sure you update the Daily data first. Thereafter, you can do bulk updates, which will always request the Daily bars first before intraday.

QUOTE:
Right here -

Awesome! Somehow I missed it.

QUOTE:
Thereafter, you can do bulk updates, which will always request the Daily bars first before intraday.

Perfect.

Now I really hope if the default time scale is considered for the feature request.

Problem is, Wealth-Lab 7 DataSets are scale agnostic. Suppose you have two DataSets with two different symbols in each. Introducing a default bar scale into the historical data provider would just make it backfill the data for all these four symbols in both scales.

To achieve the goal of updating all symbols we do not need this request. It's already possible to do so on a per provider basis. Introducing default bar scales would not help update two symbols in 1-minute scale and the other two in Daily.

So I do not quite understand this request.

The point is to provide a default scale that is always updated, whether or not you've already requested data for that symbol/scale.

I'll add that this would apply for linked providers only. If the linked provider doesn't support the scale, then it's a no-op. Clearly, if you created a DataSet for a Provider, there must be at least 1 scale you want to collect data for.

Just add a scale dropdown control to the Data Preferences. Make sure this scale is always updated for linked providers.

The biggest push back that we've had for this idea involve the Y! World Indices. Probably no one would want to update all of those international indices, so these would be an exception - or it could be optional.

In my opinion, exposing a control for default bar scales like what @kazuna expects would have a negative effect on the Data Manager's user experience, introducing clutter from adding unnecessary option. That would not make the product really better.

Forcing default scales without exposing them may be as arguable as a scale dropdown in the Data Preferences because of World Indices (yes, your concerns sound very valid) and maybe other use cases that we don't know yet.

And my point is that we can already implement (optionally) such bulk update logic in a data provider (that doesn't already have it) and update data in a default scale.

QUOTE:
Just add a scale dropdown control to the Data Preferences. Make sure this scale is always updated for linked providers.

It is going to be more than one scale selectable (e.g. 1-minute and daily), isn't it?

We haven't decided on whether it's really beneficiary for the product to have this option or it would just add unnecessary feature creep.

QUOTE:
I love the idea of assigning a default time scale on a HDP! Please make it to be able to assign more than one time scale. I will assign daily scale and 1-minute scale.

The way I understand Cone's idea, it may be having a default bar scale hardcoded on a per provider basis that is not exposed in the GUI as an arcane / single-user option and thus is not subject to be configured. For me it would make sense then. Like if Morningstar only supports Daily data there's zero point to look for other scales.

This is already in the provider base class, via the SupportsScale method.

Hmm wouldn't SupportsScale just list all supported scales? In this context we're rather thinking about a new property for default scale to drive provider updates in bulk update mode when none of its DataSets have been updated yet.

Rather than add yet another property wouldn’t it make sense to just use daily if available?

I like the idea.