How-to-reduce-Loading-Data-time

Hi,

Since it's your custom data provider it's generally up to you to improve its performance. Once Wealth-Lab has loaded data in this strategy backtest window, there is no need to reload it again until there's a new bar in the data.

1. Create a DataSet linked to your provider.
As long as the data are up-to-date, Wealth-Lab won't have to request the data from other [checked] Historical Data providers.

2. This one's important:
Turn off (uncheck) Event Providers if you don't need their data.

If I do just both of those. A 10-yr backtest using the S&P 500 loads all the data in under 2 seconds.

@Eugene @Cone

Thanks for the solutions you provided.
The current situation is that I build a extension to conect sql database with WL7. Everytime runnung a strategy on sql database, it seems loading all the symbols and culumns.
The database contains 4000+symbols and 40+ columns with 20yealrs, it is too inefficiency to load all columns in for one backtest.
It seems the reasonalable way to use data in sql is to load only the specific columns I mentioned in strategy. 
Is there any other solution (ex.function in strategy or fixing extension) that can help to load only the specific columns of data in one time backtest?  

Wealth-Lab is only loading whatever your data provider is programmed to do. For example, when it should request only the last data for a daily update as per guidelines, the historical provider should comply rather than return all available data.

Another bottleneck may be an incorrect or absent indexing of the database table.

@Eugene @cone
Please give me more hint. I don't know how to fix this problem.

Cone and yours truly have given you hints but haven't received any feedback on how you used this information in your troubleshooting. What about Events? Did we miss the database schema and indexing? Your data provider code? There is positively nothing to work with. 🤷

It's not easy to understand how to help becasue this is your own custom component here. The interface itself is very basic, as you saw. You're provided the symbol, date range and/or max number of bars you should return. It's just your job to get the appropriate data from your DB in the most efficient manner as you can.

Now, we have a Feature Request in queue to develop a SQL Provider, and it's nearing the top, so I imagine we'll start working on it soon.

@Glitch

I send you a compressed package of Extensions API file and SQL file ( Data ) .

Will u check the problem, please ?

How is the Event Providers tab set up in the Data Manager?

@Eugene

QUOTE:

How is the Event Providers tab set up in the Data Manager?

No, problem is like the question title.

where the most likely problem occurs?

Event providers (especially those like Chart Patterns) can and will considerably slow down the process of data loading.

@Eugene

QUOTE:

Event providers (especially those like Chart Patterns) can and will considerably slow down the process of data loading.

No, isn't Event providers

I using the Extension API (DataSetProvider) let WL7 connect MySQL to run backtest

problem is run backtest loading data takes a lot of time.

Look for the bottleneck. Put some stop watches on the methods. Maybe you're doing something like reconnecting to the database for every symbol?

We can't tell and I don't know what else we can suggest. We use the same API for about 2 dozen of our own providers. The API isn't the problem.

QUOTE:
I using the Extension API (DataSetProvider) let WL7 connect MySQL to run backtest

I knew this from the start. 🤦‍♂️

I'm asking to show us how your Event Providers tab is configured. Can you show a screenshot??

@Cone, @Eugene

Ok, Ok maybe i misunderstand something ..

QUOTE:

Event Providers tab is configured. Can you show a screenshot??

Event Providers tab screenshot :



Thx.

Please uncheck WealthData provider on this tab to prevent further data requests for event items which take time but cannot be satisfied due to inapplicable symbols.

@Eugene

Hi,

QUOTE:

Please uncheck WealthData provider on this tab to prevent further data requests for event items which take time but cannot be satisfied due to inapplicable symbols.

uncheck all has been tried.

result is same.

Before anything else:

In a recent build we made many Event providers keep from making data requests for incompatible symbols. I checked and you're running outdated builds 8 (!!!) and 21. Because they are not supported please upgrade both computers to the latest version.