by Glitch:
Will let you compare the results of two (maybe more?) backtests, and support its own extendable visualizers.
by Eugene:
Perhaps in addition it'd make sense to store the strategy code for making code comparisons a la Reports-Lab or Snapshots for WL6
http://www2.wealth-lab.com/WL5WIKI/pvSnapshots.ashx
Will let you compare the results of two (maybe more?) backtests, and support its own extendable visualizers.
by Eugene:
Perhaps in addition it'd make sense to store the strategy code for making code comparisons a la Reports-Lab or Snapshots for WL6
http://www2.wealth-lab.com/WL5WIKI/pvSnapshots.ashx
Rename
Looks like this will be the next major feature we develop, let's brainstorm any ideas for how you'd like this new feature to work.
One feature that I would like to see included is a Source Code comparison tool now that the strategies are no longer in XML files.
Vince
Vince
Some ideas/features (in a brainstorm manner):
- I would like to be able to restore strategy from the “table row”. So if talking about a table as a performance comparison tool, I would like to select the row and to get the strategy by this. Something like: double click and I see the strategy code, I can set strategy params, I can backtest with the same params or different params and settings. If it can not be implemented – the good alternative would be data that let you do the same – strategy name, param names and values, settings that were used while backtesting and so on, so you can do it “manually”.
- So, again, the “table” can contain this peases of information: strategy name or code, params names and values, backtesting setting that were applied, result metrics.
- I don’t know the core idea of this tool, but maybe something like this. Before or just after the backtest you can check the checkbox “add to comparison”. And this one will be added to the “comparison table”. Or maybe every one goes there and then you use filters or Delete button or something.
- You have this nice Metrics report for backtesting and I think every Metrics of it can be added to the “table”, but I would like to have something like a tool for check/uncheck exact metrics (or other columns) to leave the table adaptable and user friendly. The same for settings maybe – so if you are a kind of backtesting geek you can have your 100 columns table, but if you are not – you can make it 10-20 columns or something.
- About filtering. Some field would be used more often – I think ticker/dataset – is one of this. So if you use the concept I described earlier – it would be easy to filter by the dataset, if you don’t -there must be something to let you filter by this and other fields of that kind. So the idea is to let user get subsets for the most common usecases – by dataset, by strategy by “long only/short only” etc.
Maybe you plan something very different from what I wrote, so I’ll stop brainstorming for a while, to get the main vector from you).
- I would like to be able to restore strategy from the “table row”. So if talking about a table as a performance comparison tool, I would like to select the row and to get the strategy by this. Something like: double click and I see the strategy code, I can set strategy params, I can backtest with the same params or different params and settings. If it can not be implemented – the good alternative would be data that let you do the same – strategy name, param names and values, settings that were used while backtesting and so on, so you can do it “manually”.
- So, again, the “table” can contain this peases of information: strategy name or code, params names and values, backtesting setting that were applied, result metrics.
- I don’t know the core idea of this tool, but maybe something like this. Before or just after the backtest you can check the checkbox “add to comparison”. And this one will be added to the “comparison table”. Or maybe every one goes there and then you use filters or Delete button or something.
- You have this nice Metrics report for backtesting and I think every Metrics of it can be added to the “table”, but I would like to have something like a tool for check/uncheck exact metrics (or other columns) to leave the table adaptable and user friendly. The same for settings maybe – so if you are a kind of backtesting geek you can have your 100 columns table, but if you are not – you can make it 10-20 columns or something.
- About filtering. Some field would be used more often – I think ticker/dataset – is one of this. So if you use the concept I described earlier – it would be easy to filter by the dataset, if you don’t -there must be something to let you filter by this and other fields of that kind. So the idea is to let user get subsets for the most common usecases – by dataset, by strategy by “long only/short only” etc.
Maybe you plan something very different from what I wrote, so I’ll stop brainstorming for a while, to get the main vector from you).
I think it would be helpful to explain the original idea of this tool (it was inspired from a request by mjj3).
The new tool will allow you to compare two or more already executed strategies. I imagine the strategy window will have some new mechanism to push its results into some kind of holding area, and the held results from multiple strategies could then be added to the new comparison tool.
The new tool could have custom visualizers to report on multiple strategy sets, maybe showing the equity curves, and correlations.
The new tool will allow you to compare two or more already executed strategies. I imagine the strategy window will have some new mechanism to push its results into some kind of holding area, and the held results from multiple strategies could then be added to the new comparison tool.
The new tool could have custom visualizers to report on multiple strategy sets, maybe showing the equity curves, and correlations.
I definitely want something like this for working on a single strategy.
While working with the strategy, your code evolves. Params values change. I would like to store all the results because you change something, backtest something, then you got a great Idea to improve your strategy, sometimes these branches lead to nothing. And this process is spontaneous sometimes, you may miss a really good subidea. But if you had an instrument that stores strategy code, params values, backtesting results, you could look though these results of a single evolving strategy and find those braches, that were not paid enough attention.
The only problem I can see is dll strategies – to just store the copy of dll file is not enough as you can not see your code and cannot continue development from this point.
While working with the strategy, your code evolves. Params values change. I would like to store all the results because you change something, backtest something, then you got a great Idea to improve your strategy, sometimes these branches lead to nothing. And this process is spontaneous sometimes, you may miss a really good subidea. But if you had an instrument that stores strategy code, params values, backtesting results, you could look though these results of a single evolving strategy and find those braches, that were not paid enough attention.
The only problem I can see is dll strategies – to just store the copy of dll file is not enough as you can not see your code and cannot continue development from this point.
Or maybe someone can share one’s experience so I could avoid this problem by better organizing the process using existing tools)
Back in the day there used to be a separate app called Reports Lab that stored backtest results as well as the corresponding strategy code, if my memory serves me right. It was very useful.
When that was no longer available I started storing backtest results summary, any strategy changes & my logic behind them the Strategy Summary; it was a separate tab in prev versions of WL and one could view/scroll large segments at a time almost like a page. I do it to this day.
Now in WL7 one just has a mini window in the Strategy Settings tab to view your stored notes, if any.
When that was no longer available I started storing backtest results summary, any strategy changes & my logic behind them the Strategy Summary; it was a separate tab in prev versions of WL and one could view/scroll large segments at a time almost like a page. I do it to this day.
Now in WL7 one just has a mini window in the Strategy Settings tab to view your stored notes, if any.
For me it is more important to have a text field for comments what I changed from run to run.
And a selectable set of metrics from various scorecards to monitor progrss from run to run.
These two features would support my explorations quiet well.
And a selectable set of metrics from various scorecards to monitor progrss from run to run.
These two features would support my explorations quiet well.
QUOTE:
text field
+
We already have a feature request for selectable scorecard metrics. And, having a more robust note taking capability isn't exactly what this feature is about, but perhaps it can be rolled into it. Hmmm.
Seems like we are mixing a few different use cases.
a) Backtest Result Comparisons (simply the difference between two different backtests ) - Could be used to compare 2 different strategies, 2 different versions of the same strategy (no need to access trade details)
b) In-sample period vs. Out-of-sample period - this is different than a) as you might want to be able to access both results along with the details of each backtest (trades, etc) to allow for further analysis.
c) Multi-Strategy Portfolio Analysis - (I see this as a different tool) Imagine you have 20-50 viable strategies each with their own underlying datasets/money mangement. You want to answer what is the best combination of strategies and their weights? The basic approach to this is using a Minimum Variance Testing approach that optimizes your favorite fitness function combination (i.e Sharpe Ratio, etc). Research in this space I believe would be hugely valuable and doesn't exist too many places. It's also the closest thing I've come to finding the holy grail.
Trying to answer:
- What combination of fitness functions are the best predictors of future performance?
- What is the best timing and approach to rebalancing your strategy portfolio(i.e. when to change up your portfolio of strategies).
- Using Ensemble Strategies - https://en.wikipedia.org/wiki/Ensemble_learning
- Solving for strategies that may not be profitable but will significantly reduce your risk allowing you to optionally leverage up your portfolio more.
d) Strategy Version Management - It would be handy to be able to compare the results of different versions of the same strategy and be able to rollback to a prior version.
a) Backtest Result Comparisons (simply the difference between two different backtests ) - Could be used to compare 2 different strategies, 2 different versions of the same strategy (no need to access trade details)
b) In-sample period vs. Out-of-sample period - this is different than a) as you might want to be able to access both results along with the details of each backtest (trades, etc) to allow for further analysis.
c) Multi-Strategy Portfolio Analysis - (I see this as a different tool) Imagine you have 20-50 viable strategies each with their own underlying datasets/money mangement. You want to answer what is the best combination of strategies and their weights? The basic approach to this is using a Minimum Variance Testing approach that optimizes your favorite fitness function combination (i.e Sharpe Ratio, etc). Research in this space I believe would be hugely valuable and doesn't exist too many places. It's also the closest thing I've come to finding the holy grail.
Trying to answer:
- What combination of fitness functions are the best predictors of future performance?
- What is the best timing and approach to rebalancing your strategy portfolio(i.e. when to change up your portfolio of strategies).
- Using Ensemble Strategies - https://en.wikipedia.org/wiki/Ensemble_learning
- Solving for strategies that may not be profitable but will significantly reduce your risk allowing you to optionally leverage up your portfolio more.
d) Strategy Version Management - It would be handy to be able to compare the results of different versions of the same strategy and be able to rollback to a prior version.
Yes, A and B could be Visualizers for this tool we’re working on now.
C could be a super advanced feature of the MetaStrategy or maybe like you suggest a completely different tool.
C could be a super advanced feature of the MetaStrategy or maybe like you suggest a completely different tool.
Regarding D) Version Management
I remember a friend of yours recommended using Visual Studio Code as the development environment instead of (or in conjunction with) the code control being used currently.
That would be an amazing feature down the road. Already integrated with Git, etc.
I find it difficult working on two different computers without using compiled strategies to keep things in sync.
Another cool feature would be having the definition of the datasets be server based to allow for synching across computers, sharing between users and when we can publicly share strategies, share the dataset with it so results should tie out perfectly between users.
I remember a friend of yours recommended using Visual Studio Code as the development environment instead of (or in conjunction with) the code control being used currently.
That would be an amazing feature down the road. Already integrated with Git, etc.
I find it difficult working on two different computers without using compiled strategies to keep things in sync.
Another cool feature would be having the definition of the datasets be server based to allow for synching across computers, sharing between users and when we can publicly share strategies, share the dataset with it so results should tie out perfectly between users.
Regarding b)
My main thoughts on this is to allow for in/out of sample backtests along with various tests/visualizers to avoid curve fitting.
Perhaps you could add a separate input on the Backtest Data control (or add another tab for in/out of sample testing) that allows you to specify in/out of sample % and a dropdown with various ways to define the out of sample data (i.e. put the oos in the front/middle/back) along with various measures/visualizers to compare the two results.
You could then also add various tests when comparing to random data, noisy data, etc
My main thoughts on this is to allow for in/out of sample backtests along with various tests/visualizers to avoid curve fitting.
Perhaps you could add a separate input on the Backtest Data control (or add another tab for in/out of sample testing) that allows you to specify in/out of sample % and a dropdown with various ways to define the out of sample data (i.e. put the oos in the front/middle/back) along with various measures/visualizers to compare the two results.
You could then also add various tests when comparing to random data, noisy data, etc
I still have not crystalized this tool in my mind completely. But I've had some ideas to throw out there.
Perhaps this is a new and improved Strategy Ranking, where you can either add existing backtests from a strategy window, or drag and drop strategies in. The results are presented in tabular format with possible other visualizers.
The tool could also offer a Clone option, which would open a new instance with the same strategies and settings. You could then change the date range for example, run them all again, and quickly compare "in-sample" and "out-of-sample" runs.
I believe the other ideas we covered, advanced note taking, strategy versioning, and built-in "out-of-sample" comparisons, would deserve their own feature requests.
Perhaps this is a new and improved Strategy Ranking, where you can either add existing backtests from a strategy window, or drag and drop strategies in. The results are presented in tabular format with possible other visualizers.
The tool could also offer a Clone option, which would open a new instance with the same strategies and settings. You could then change the date range for example, run them all again, and quickly compare "in-sample" and "out-of-sample" runs.
I believe the other ideas we covered, advanced note taking, strategy versioning, and built-in "out-of-sample" comparisons, would deserve their own feature requests.
Finally started working on the development for this - it's taking shape as the new incarnation of the Strategy Ranking tool, but will have some handy features for comparing the same Strategies run under different date ranges or other settings!
Your Response
Post
Edit Post
Login is required