**I wish to thank the developers for adding Data Tool functionality to the Data Manager(DM) in Build 9, it was missed.** :)
--------------------------------------
[Just to be clear the issue of slownes is about loading symbols for display in the DM, NOT about downloading data.]
After upgrading to B9 I noticed that that was a marked delay in loading symbols in the DM, especially for larger datasets. {Whether this was not present before the Data Tool was added, or was there but less marked, I don't know as I never did this 'speed test' before B9.} I decided to test this objectively and the results are tabulated below (YMMV):
Issues:
1a) When it comes to loading larger datasets, the DM is painfully slow. This makes it virtually impractical to add/remove symbols, make a quick symbol edit and so forth to large data sets. Please speed this up.
1b) There is actually no need for the DM to display full details about the symbols every time a dataset is clicked; having a button such as Symbol Details in WL6 is an easy way to switch from displaying symbol names only (the default option, but allows editing of symbols) to full Symbol Details (required for Data Tool functions).
2) In addition to, and regardless of, point #1 above please add a right-click menu way to add symbols to a dataset, bypassing display entirely; this is a super-fast way to add either individual or a block of symbols.
3) Clicking Bars column doesn't sort, unlike other columns.
4) There should be a way to see the # of symbols in a dataset w/o having to load all the symbols.
5) When switching datasets, the previous dataset's info & symbols stay on-screen; ideally, the slate should be wiped clean.
Hope the developers can fix these issues soon.
--------------------------------------
[Just to be clear the issue of slownes is about loading symbols for display in the DM, NOT about downloading data.]
After upgrading to B9 I noticed that that was a marked delay in loading symbols in the DM, especially for larger datasets. {Whether this was not present before the Data Tool was added, or was there but less marked, I don't know as I never did this 'speed test' before B9.} I decided to test this objectively and the results are tabulated below (YMMV):
Issues:
1a) When it comes to loading larger datasets, the DM is painfully slow. This makes it virtually impractical to add/remove symbols, make a quick symbol edit and so forth to large data sets. Please speed this up.
1b) There is actually no need for the DM to display full details about the symbols every time a dataset is clicked; having a button such as Symbol Details in WL6 is an easy way to switch from displaying symbol names only (the default option, but allows editing of symbols) to full Symbol Details (required for Data Tool functions).
2) In addition to, and regardless of, point #1 above please add a right-click menu way to add symbols to a dataset, bypassing display entirely; this is a super-fast way to add either individual or a block of symbols.
3) Clicking Bars column doesn't sort, unlike other columns.
4) There should be a way to see the # of symbols in a dataset w/o having to load all the symbols.
5) When switching datasets, the previous dataset's info & symbols stay on-screen; ideally, the slate should be wiped clean.
Hope the developers can fix these issues soon.
Rename
We’ll do our best, thanks for the detailed reports.
QUOTE:
1b) There is actually no need for the DM to display full details about the symbols every time a dataset is clicked;
I like the way it works now and I feel the extra button in WL6 is suboptimal.
QUOTE:
2) In addition to, and regardless of, point #1 above please add a right-click menu way to add symbols to a dataset, bypassing display entirely; this is a super-fast way to add either individual or a block of symbols.
The "Configure" {DataSet} button is the way to add a block of symbols via right-click menu.
Since there was no data load in WL6 w/o Symbol Details or Data Tool tab, "Instantaneous" actually means N/A = Not Applicable. Clicking a DataSet in the left side tree in WL7 (e.g. for the purpose of adding symbols via Configure) is also "Instantaneous".
I believe that the head to head comparison of the Data Tool between WL6 and WL7 that you made is incorrect. Considering that the WL7 data mechanism works differently, the Data Tool performs more work internally. Here I mean the Source column which displays the actual data provider. And this useful feature does incur extra CPU cycles on WL7. Without this you might not be aware of cases when say Yahoo data is N/A and the symbol in your DataSet is actually sourced from Wealth-Data.
How often does a typical user load DataSets of 4,500 symbols into the DT? Personally I'm not convinced that a slight delay is a serious issue here.
I believe that the head to head comparison of the Data Tool between WL6 and WL7 that you made is incorrect. Considering that the WL7 data mechanism works differently, the Data Tool performs more work internally. Here I mean the Source column which displays the actual data provider. And this useful feature does incur extra CPU cycles on WL7. Without this you might not be aware of cases when say Yahoo data is N/A and the symbol in your DataSet is actually sourced from Wealth-Data.
QUOTE:
1a) When it comes to loading larger datasets, the DM is painfully slow.
How often does a typical user load DataSets of 4,500 symbols into the DT? Personally I'm not convinced that a slight delay is a serious issue here.
I have the same issue here, loading 4k dataset, the program just not responsing when trying to click the dataset icon in DM window.
@johnliang
What data provider does it belong to?
What data providers are checked in Data Manager > Historical Providers?
What data provider does it belong to?
What data providers are checked in Data Manager > Historical Providers?
@Eugene,
A lot to digest in what you wrote so let me try to tackle points one by one:
Thanks for clarifying that, I thought Configure meant setting up the API key, etc. Perhaps the name can be changed to Add Symbols...? Also, note that if you right-click on a dataset inside the Data Manager you still have to wait for all the symbols to load - which can be agonizing for large datasets.
Whether a user checks/cleans the data monthly or weekly doesn't matter, one expects the loading to be quick. And the delay is very significant, not "slight".
WL6.9 Data Tool also has a 'Remove inactive' option, and is so much faster.
------------------------------
Don't get me wrong, I'm glad the Data Tool is here but perhaps the time-consuming subsections of the Tool that are bottlenecking the loading can be made optional and maybe marked with an asterisk? And when the user wants to use these a popup message can inform this will take (significant) time?? I'm sure the developers can figure out a smart way.
As things stand now, the loading of large datasets is just too slow.
A lot to digest in what you wrote so let me try to tackle points one by one:
QUOTE:
The "Configure" {DataSet} button is the way to add a block of symbols via right-click menu.
Thanks for clarifying that, I thought Configure meant setting up the API key, etc. Perhaps the name can be changed to Add Symbols...? Also, note that if you right-click on a dataset inside the Data Manager you still have to wait for all the symbols to load - which can be agonizing for large datasets.
QUOTE:
How often does a typical user load DataSets of 4,500 symbols into the DT? Personally I'm not convinced that a slight delay is a serious issue here.
Whether a user checks/cleans the data monthly or weekly doesn't matter, one expects the loading to be quick. And the delay is very significant, not "slight".
QUOTE:
Without this you might not be aware of cases when say Yahoo data is N/A ....
WL6.9 Data Tool also has a 'Remove inactive' option, and is so much faster.
------------------------------
Don't get me wrong, I'm glad the Data Tool is here but perhaps the time-consuming subsections of the Tool that are bottlenecking the loading can be made optional and maybe marked with an asterisk? And when the user wants to use these a popup message can inform this will take (significant) time?? I'm sure the developers can figure out a smart way.
As things stand now, the loading of large datasets is just too slow.
Personally I'm not experiencing the slow load. I even updated the Nasdaq Composite which has almost 4,000 symbols and it loads into the DM in about 4 seconds for me. That's why we are asking for more details.
I updated Russell 1000 and then clicked on Nasdaq Composite which took 5 seconds to load (for the junction of 335 symbols with data out of 3726). Only had Yahoo historical provider activated at the moment of test.
Hello Eugene,
I have created an ASCII dataset loaded from my local csv data, whenever I click on this dataset in DataSets subtab it stop responsing for more than a minute.
Thanks
I have created an ASCII dataset loaded from my local csv data, whenever I click on this dataset in DataSets subtab it stop responsing for more than a minute.
Thanks
John, it's because the ASCII provider has to perform a lot of data parsing to display the table.
Thanks for your explaination Eugene
just wondering why only ASCII dataset has to perform parsing data each time while other dataset, like yahoo, won't having this slow display issue?
just wondering why only ASCII dataset has to perform parsing data each time while other dataset, like yahoo, won't having this slow display issue?
Because WL7 uses its built in persistent storage mechanism to save the data downloaded from Yahoo, and many other such providers, into an optimized binary file format.
@johnliang
A binary caching for ASCII files a la WL6 might help.
In WL6, caching ASCII data speeds it up transparently to the user. On first access a shadow copy of the data is created in a binary file which gets recreated on change to the source ASCII. Such file caching frees WL up from all the parsing and number conversion and does not incur a RAM penalty.
You might want to create a new topic with this feature request and click #FeatureRequest for it to be added to the wishlist - where you could vote it up!
https://www.wealth-lab.com/wishlist
A binary caching for ASCII files a la WL6 might help.
In WL6, caching ASCII data speeds it up transparently to the user. On first access a shadow copy of the data is created in a binary file which gets recreated on change to the source ASCII. Such file caching frees WL up from all the parsing and number conversion and does not incur a RAM penalty.
You might want to create a new topic with this feature request and click #FeatureRequest for it to be added to the wishlist - where you could vote it up!
https://www.wealth-lab.com/wishlist
This issues I pointed out are real.
Of course, everybody's experience will differ as there are many variables involved - not just # of symbols but also how much data (bars) they have, concurrent processes running on the PC, and so forth.
I did my job by bringing them to the developers' attention.
Of course, everybody's experience will differ as there are many variables involved - not just # of symbols but also how much data (bars) they have, concurrent processes running on the PC, and so forth.
I did my job by bringing them to the developers' attention.
QUOTE:
I did my job by bringing them to the developers' attention.
Three of us tried to reproduce the issue but couldn't. Like you said, there may be some other variable involved - like the order of activated data providers etc.
I am loading a Nasdaq composite DataSet using Ameritrade and it took almost two hours to load this to backtest it on a strategy. What am I doing wrong to take this long to load my data set for full data? Is 15 minute and 30 minute and daily times that much of a difference for reloading the same data set and timeframe over and over again? If I change the backtesting scale it results in another 2 hour load time for the data. Alpaca I was experiencing the same data set load delay for any scale other than daily.
Thanks!
Thanks!
Ameritrade has data loading limitations, so the provider has built in throttling to cope with these limitations. Same for Alpaca. If you are trying to load intraday Nasdaq composite these providers are not a good fit. The only really good fit would be IQFeed.
Oh you're right. I just tried it with Yahoo and it loaded in a minute, not using Ameritrade for backtesting again, thank you!
Your Response
Post
Edit Post
Login is required