Recommended-Backtesting-Period

The first thing to decide is, what is your trading style. Is it buy and hold, swing trading, or intraday. If it is buy and hold I suggest at least 5 years. I'm a swing trader and so I go back 6 months. But initially I would include the pandemic period just to see how it would be handled.

QUOTE:
So, the danger when using a 20 year period is to optimize mainly on stock market algorithms, that don't exist anymore nowadays.

This really depends upon the Market inefficiency that you are attempting to capture. Some of them persist, some do not. You will find out quickly. I suggest that you don't assume, but rather test.

QUOTE:
The huge downturn and the following huge upturn could influence strategies

This period should be in your "Out of Sample". If your strategy cannot navigate this period "unseen", I suggest that you need to work on the strategy.

Vince

We (here in Russia) describe this (if I try to translate) as: it has never happened and then it happened again. I mean – there is nothing incredibly special with this covid market impact.

It depends on what are you doing and how are you doing this.

I prefer long-living ideas – like trends), which are as old as the market itself). At this point the more data I have the better. Of course, the strategy result will evolve and you’ll see it while analyzing backtesting results, like equity curve or other results.

In common, longer backtesting period will highly likely prevent you from curve-fitting.

I day trade 5-minute volatility (VIX products) using trend following and VIX Term Structure. Therefore, I include 2008 until current date to capture extreme volatility events, such as 2008, Bexit, XIV meltdown, and Covid 19. I train the strategy on 2017-current and then I examine it against 2008 - 2116.
I agree with Replikant_M. Each day comes with different set of circumstances, so there is no day that looks like the other and 'There will be blood/surprise' you haven't seen before.

QUOTE:
... it seems that around 8 years ago, there was a change at stock exchange markets.

You are correct. In 2015, Janet Yellen announced that the FOMC (i.e. "The Fed") would begin to raising interest rates, which started later that year. This created a big turn in the market.

And I agree with you. I wouldn't simulate way back into the past because the market climate will be very different. But there's also a statistical risk of over fitting. If the WL optimizer is fitting model parameters ("preferred values"=PVs) per stock, then you need enough trades per stock to properly fit all those model parameters. You require at least 5 degrees of freedom for random error (DF>=5 for error); otherwise, your model is over fitted. Check your statistics textbook! So if you have 6 PVs for the optimizer to fit plus 5 DF for error, you now need at least 11 trades per stock during your simulation period. If you don't have that for a given stock, then the fit is a crap shoot, and you shouldn't be trading that stock.

So you need to set your simulation range "just" large enough so you get enough trades per stock so your aren't over fitted. And that probably means some stocks may not be tradable because they haven't traded enough to solve for all their PV parameters reliably. So just delete those stocks (that have less than 5 DF for error) from you datasets. Bottom line, trust what your statistics professor taught you about model fitting; he is right!

OK, thank you all for your feedback!

I am buying at market open. Then holding the positions between 1 and 3 days and then selling again at market open. I work on strategies for an optimal entry, but the exit happens in a fixed amount of days (I just backtest, how many days are optimal).

It is very interesting what strategies you use. And of coure, it might indeed be a good idea to have a closer look at statistics.

I created a strategy on the S&P500 index. It consists of 1 long and 2 short sub strategies. I backtested and optimized it heavily and in the backtesting I could achieve a profit of more than 60% per year for the last 20 years.

I am trading it since the beginning of the year, but so far I acquired a loss with it. So, now I am wondering if this is just a random drawdown or maybe I made some serious mistakes when developing it.

60% per year looks like a mistake).

You can merge two equities: in sample and out of sample (traded one). I think you can see if out of sample one looks natural for the equity or not.

60% APR for daily scale stock trading is slightly on the high end, but it’s not impossible. The best systems on WealthSignals.com approach this with years of out of sample performance tracking.

QUOTE:
60% per year looks like a mistake).

Agreed. I've never gotten above low 50's on a 10-year backtest with 225 symbols.

QUOTE:
60% per year for the last 20 years

Is this with Margin? If so, I might think that the Risk profile might be a bit rough.

BTW, what amount of slippage are you using in your backtest? Also, what type of Orders are you using?

Vince

Margin Factor is 1.10. Position sizing is 33.33%. So the 10% margin is just for securing that the strategy can trade, if a share got too expensive. Still it is very risky. It had around +60.50% per year in the years 2001-2020 (in sample) with a maximum drawdown of 30%. Unfortunately in 2021 (out of sample) it has so far -13.37%. So maybe it is over optimized.

It works on the S&P500. I wonder if it is a good idea to use this dataset. Maybe it might be wiser to use the S&P100 as you don't have smaller companies in that dataset. But it was much easier for me to acquire a good profit for me on the S&P500 - especially with short strategies.

Since I have a minus in 2021 either it is very bad luck, or I did major mistakes when developing this strategy.

What amount of slippage are you using in your backtest?

What type of Orders are you using?

How many trades in your Training set?

What is your average profit %?

QUOTE:
I created a strategy on the S&P500 index.... I backtested and optimized it heavily ... and ... achieve a profit of more than 60% per year for the last 20 years.

I am trading it since the beginning of the year, but ... I acquired a loss with it. So,... is this just a random drawdown or maybe I made some serious mistakes when developing it.

This is an interesting question. In the short term, anything is possible. But ...

... what about the long term? Perhaps the APR isn't the best metric to be optimizing against. I look at the Net Profit and Win Rate. And I have a Win Rate of 70% for in-sample simulations. But this does not guarantee my trading will always be profitable. However,...

... I have not wrapped Volume into my analysis yet. And I have not taken sentiment into account either. And both of these are important factors. So, bottom line, a 70% Win Rate is not that revealing.

The question remains, what metric should be used to judge the success of a strategy? I think the answer lies with analyzing the Equity Curve somehow.

@ Carova:

I only use "Buy at Market Open" and "Sell at Market Open" orders. So there should be no slippage. Over the 20 year period there are 4.173 trades (2.580 long and 1.593 short).

With the long strategy I always sell, when the position is older than 3 bars.
With one short strategy I always sell, when the position is older than 2 bars, with the other when older than one bar.

Average Profit (All Trades) = 0.71%
Average Profit (Long Trades) = 0.83%
Average Profit (Short Trades) = 0,52%

@ superticker:

Somehow, the annualized gain is what you want to be high. But of course, I see your point. Let's compare two scenarios:

First scenario / average annualized gain = 50%
Starting capital = 100
First year profit = 25 -> Capital = 125
Second year profit = 75 -> Capital = 218.75

Second scenario / average annualized gain = 49%
Starting capital = 100
First year profit = 49 -> Capital = 149
Second year profit = 49 -> Capital = 222.01

So, in scenario 1, the annualized gain is higher, but in scenario 2 the profit is higher, because of the fact that the annualized profit in the second scenario is more stable.

So, I think you are right and it should be optimized towards overall profit!

QUOTE:
I think you are right and it should be optimized towards overall profit!

My reaction is inappropriate for this current topic, so lets switch it into this alternative topic (below). And you can vote up the feature request for this alternative topic.
https://www.wealth-lab.com/Discussion/Meta-Scorecards-and-for-Multi-Objective-Optimization-5947

wernerhh,

Is the DD depth and duration similar to the data from your backtest?

Vince

@ Carova:

The Drawdown is not as huge and not as long as in my backtesting. So, I should go on with the strategy, I guess.

However, I did some optimization work, which might be not useful for "out of sample" data, so I am a bit concerned. I guess, I will reduce the capital invested.

@ superticker:

I made a mistake: The APR is exactly calculated from the profit (it is not an average of the individual annualized gains). So, it makes no difference if the strategy gets optimized towards the APR or the overall profit.

QUOTE:
The Drawdown is not as huge and not as long as in my backtesting. So, I should go on with the strategy, I guess.

When the realtime results are within the scope of backtesting results it is difficult to assess whether or not a system is still working, unfortunately.

Vince

Vince: Yes, but I guess if you play around too long, if you try to change each and every parameter to make a strategy more profitable, it is very likely that it was overoptimized. I also did not do an out of sample test.

Werner

QUOTE:
I also did not do an out of sample test.

I would strongly suggest that an OoS test be a part of your system design process. It has saved me a tremendous amount of trouble in my work.

Vince

Vince: Thank you!

Can you suggest a period for the OoS test?

At the moment I am going for nine years (2011-2020) to optimize, and then use one year (2020-2021) as OoS. But is this one year enough? Another problem is, that the last year might not have been a typical year on the stock exchange.

Werner

Werner,

I suggest that you consider the normal procedures that are followed in Machine Learning circles. The OoS is generally 20%-50%. I routinely use ~30-35%.

The key is that the OoS period performance should be "essentially identical" to the Training period performance. Of course, "essentially identical" is in the eye of the beholder! ;)

Vince

Thank you, Vince. I guess I go for the last 10 years. 7 of it will be the "in sample" and the last 3 the "out of sample". Of course the lockdown crisis will then fall into the "oos" period, but maybe I can compare which strategy performs the most stable in this test.

Anyway, for me finding an optimal strategy with Wealth Lab is like the saying "Even a blind hen sometimes finds a grain of corn." It is a lot of randomly trying out things. I wonder if it would be better to have a software, which can automatically create a strategy randomly - e.g. by taking some parameters of a list of hundred. And then do the backtesting automatically. So, you have this software work for some weeks, and then the software presents the optimal strategy to you, and you don't have to get into the "blind hen" mode.

Best regards
Werner

QUOTE:
I wonder if it would be better to have a software, which can automatically create a strategy randomly - e.g. by taking some parameters of a list of hundred. And then do the backtesting automatically.

There are some neural network (NN) packages designed for trading that do exactly that. You need to tell them which indicators to use as inputs, however. And you may be able to do some of that with the NN extension of Wealth-Lab.

I've never used NN for stock trading. But theoretically, NN generally performs better than linear systems when it comes to nonlinear event data (as we see with trading simulations).

We can't talk about NN details here. Start a new topic instead.

QUOTE:
Of course the lockdown crisis will then fall into the "oos" period

Werner,

Actually this is a good thing. Any robust strategy needs to be able to handle these types of occasions that will invariably happen. When you find something that works well OoS you will know that you have a better chance at future success.

Vince

QUOTE:
7 of it will be the "in sample" and the last 3 the "out of sample"

I would make the 1st 3 the OOS. Optimize against the most recent data, then verify against the older data. Of course, if your strategy is truly robust, it doesn't matter.

OK, thanks for your feedback!

Vince: Would that be possible practically? You optimize a strategy for - let's say - 2011 to 2018 with market conditions without huge crises. And then it performs well for 2018-2021 - including the huge downturn during the lockdown period and then the upswing? I guess a strategy which can do this must be either very diversificated or very basic and very short-term.

Werner

Werner,

I believe that my worst performance is coming in the future. The OoS test is my "first peek" into the future.

The 2018-2021 period gives you a sense of whether or not you could have "survived/continued to trade" afterwards. A good strategy needs to acknowledge a variety of potential outcomes and address how to handle them effectively. The initial design is crucial. It needs to include all of the factors required before you even begin testing.

Vince

Hi Vince,

I think you are very gifted and talented regarding the creation of trading strategies.

I have to say, that I can spend hours, days and weeks focusing on increasing the return of a strategy. However, I don't have good approaches to minimize the maximum drawdown or to achieve high returns OOS.

But I gonna stick to the approach of my mentor: He says, that if you have several different strategies with different approaches, the maximum drawdown will get lower.

Wealth Lab does not supoort a function, where you can split your capital and invest certain shares according to certain strategies. But I want to create a program which calculates the maximum combined drawdown from backtest results of different strategies.

I think I gonna follow the saying "Let's hang on to what we got".

Werner

Werner,

I have been doing mathematical modeling for over 50 years and developing trading strategies for 40 years. During that time I have made more "mistakes" than I could possibly count. My current approach is an attempt to avoid as many of my past mistakes as possible. I don't always succeed. ;)

Vince

QUOTE:
Wealth Lab does not supoort a function, where you can split your capital and invest certain shares according to certain strategies.

Have you explored MetaStrategy?

Vince: That sounds like a good practical approach. It must have been interesting to work in these past decades and their different conditions.

Eugene: Thanks! I found Meta Strategy and checked it out.