Is it possible to somehow enable multithreading during portfolio backtesting? I am testing a portfolio of about 300,000,000 bars and at the same time the processor is loaded at two percent, because of this this process takes about 5 hours just for backtesting
Rename
WL8 is multithreaded.
Each symbol of a portfolio backtest "runs" in its own thread.
So we need more details about your case:
* how many symbols
* what data provider
* is your strategy data driven or computation driven? I.e. how complex are calculations at each bar?
Each symbol of a portfolio backtest "runs" in its own thread.
So we need more details about your case:
* how many symbols
* what data provider
* is your strategy data driven or computation driven? I.e. how complex are calculations at each bar?
391 symbols
Provider Binance
1 m timeframe
Provider Binance
1 m timeframe
Here some ideas for "hard core debugging":
* Use a (very) small data range (1-2 days). Run the strategy twice. (first is with data loading from binance, second run is without data loading from binance)
* Show the portfolio stats in DataManager. How long does it take for DataManager to show all symbol stats? (This is data access only)
* run an empty strategy with otherwise the same settings. How long would this take? (this is data access + WL-interface code)
* run a single symbol, how long does this take? (This is your code, single threaded)
* Use a (very) small data range (1-2 days). Run the strategy twice. (first is with data loading from binance, second run is without data loading from binance)
* Show the portfolio stats in DataManager. How long does it take for DataManager to show all symbol stats? (This is data access only)
* run an empty strategy with otherwise the same settings. How long would this take? (this is data access + WL-interface code)
* run a single symbol, how long does this take? (This is your code, single threaded)
No, it's not possible. The backtests are deterministic. Each bar of data processed requires the previous bar of data to have been completed. So, theoretically we could run each symbol in its own thread, but we'd have to somehow synchronize all the threads or create new threads each bar of the backtest, which would defeat any gain from the parallel processing.
In short it's not gonna happen.
Optimizations however are run in parallel.
In short it's not gonna happen.
Optimizations however are run in parallel.
On the one symbol thats take 45 seconds
And not when i test 1 symbol or all symbols processor using only 2 %
And not when i test 1 symbol or all symbols processor using only 2 %
Then I would like to ask about the RAM, is it cleared during portfolio testing or is all test data stored in it until the end of the test?
@AICMF - if you're using any data events (I'm talking about events for symbols like NextEarnings, not C# events), then go to the Data Manager and see if a bunch of event providers are checked. If so, try unchecking those that you don't need. Perhaps querying of one or more providers is causing pacing with those event provider(s).
I'm trying to figure out how much there should be in the machine for such testing.
Al the data is loaded up front and needs to remain in memory for the complete backtest.
If you share a stripped down strategy that’s exhibiting this slow run time we might be able to troubleshoot.
If you share a stripped down strategy that’s exhibiting this slow run time we might be able to troubleshoot.
Your Response
Post
Edit Post
Login is required