Last-bar-missing-in-ASCII-data-backtest-not-charts

I've noticed the same, while doing some tests last weekend. Even spent a lot of time trying to figure out if I did something wrong, but don't think it was the case.

Eventually, I recreated the historical databases; and, while doing so, I've noticed that since a few weeks, my provider (NorgateData) is now allowing to export the data to Metastock format instead. I've done that and stopped using the ASCII format - and indeed the problem is also not present for me with Metastock data.

I suspect it's not the case, but is it possible that you're adding the bar to the ASCII file after you've run the strategy? In this case, to get the update, you would need to click on the "1 Symbol(s) Loaded" message in the status bar in order to reload the data.

What I actually suspect is that something is tricking the logic that turns ASCII files into binary files. The ASCII Provider does this in order to reload the data more quickly, but it should read and convert the new text data file when its modified date is after the binary file date.

Are you updating the ASCII file from the same machine or copying from a different place that may be in a different time zone?

Yes, the text file is located on the same computer as WL8 (WL7).
I already mentioned that in WL7 everything is OK. So the text file is fine. And on WL8 there is no last bar. (The last two pictures in the post)

We will get it fixed for Build 7, Eugene is looking at this right now.

So far I haven't been able to duplicate it, I created an ASCII DataSet with data up to yesterday (5/19) and it's bringing that bar of data in to the backtest.

I asked Eugene to take a closer look at his end.

Neither I could duplicate it. I even assigned the NYMEX market to some symbols of my test ASCII DataSet to make sure our settings match. @fred9999 could you re-run it? Maybe it's the time of day that makes difference.

Just converted Nasdaq 100 index from Norgate to text format. The same - there is no yesterday's bar.

This is strange: I also use NorgateData; and, as I wrote above, I was experiencing the same as fred9999 (in the meantime, I switched to Metastock format); but now just did a test, from scratch, exporting a new ASCII watchlist from NorgateData and importing it as a ASCII dataset in Wealth-Lab B6 and I also cannot reproduce it anymore.

@fred9999, could it be that the backtesting is pulling the data from another, higher-priority historical provider (such as Wealth-Data), which in turn was not yet updated?!
Nevermind: I just realised that you are using indeed the ASCII DataSource (which then cannot be Wealth-Data; nor any default provider)

Can we see your Strategy Settings?

The setting is standard, by default.

Yesterday, after creating the DataSet from scratch, I couldn't reproduce this issue.

In the meantime, the DataSet (EOD) got updated with yesterday's bar; and now, I am experiencing it again. It's exactly as described by fred9999.

I think that the following procedure will help reproducing this:
- Create a ASCII dataset; even if with just one file (e.g. AAPL.csv), with the EOD quotes (Date, Open, High, Low, Close, Volume) up to a certain date (e.g. 20220520).
- Open a Chart for that symbol: the last date is probably correct (20220520)
- Run a backtest (with any strategy) for that same symbol: the last date is also probably correct (20220520).
…
- Now simulate the update of the dataset, by simply opening the AAPL.csv file in a text editor and adding an entry for the next day (e.g. 20220521).
- Open a Chart for that symbol: the last date is probably correct (20220521).
- Run a backtest (with any strategy) for that same symbol: the last date is probably “stuck” at the previous bar (20220520).

Are you opening a new Strategy window after the update, or trying to run the same Strategy again by just clicking Backtest? WL8 will cache the loaded data for efficiency's sake, so if you want to see any updates, open a new window, or click the label at the bottom that says "N Symbols Loaded" which will also force a new data load.

I’ve tried to open different strategies, even re-starting Wealth-Lab; but the problem persists.

QUOTE:
- Now simulate the update of the dataset, by simply opening the AAPL.csv file in a text editor and adding an entry for the next day (e.g. 20220521).

@alkimist: but AAPL cannot normally have a 20220521 bar because it's Saturday, the backtester's job is to filter it out with regard to the U.S. market.

I understand - maybe I used an unfortunate example, because of the weekend. Still, it was not clear from my description above, but even the bar of 20220520 (last Friday) is not being picked up by the backtest.

Eugene can you double check the ascii dataset binary cache? is it recreated if the ascii data that the cache is based on is updated?

The cache is intended to speed up big intraday files and applies only if two conditions are met: 1) ASCII file contains intraday data and 2) is larger than 20K lines. All topic starter's files are EOD and have 10K lines at most so cache is not at play.

OK but still, if the underlying ASCII data is updated, does the cache know this and regenerate?

I've been updating my ASCII data daily, and my backtests on the ASCII DataSet is still showing the correct last bar. alkimist, the next time you see the issue could you email us some of the ASCII files that are causing the issue, I can recreate the DataSet here locally and see if I can duplicate it with your data.

QUOTE:
OK but still, if the underlying ASCII data is updated, does the cache know this and regenerate?

Yes, if the ASCII file last write timestamp is newer than the cache file's.

I don't even need to reproduce anything. I take any text data and the last bar is missing during testing. For example, I found old Forex data that I updated many years ago. And even here there is no last bar)

OK let's start trying to identify possible differences. Maybe it's a time zone thing? What time zone is your computer in, I'll set mine to the same and see if I can reproduce this.

Also, make sure you didn't define a Footer line in the ASCII definition.

Also, it would be interesting to see what happens if you manually duplicate the last line in the file, save the file, the restart WL and run it again. Is it just not reading the last line, or is it something else?

Finally, email us a copy of the file so we can try it locally.

Time Zone: UTC+03:00 Moscow





File USDJPY.csv sent to support@w******.com

Thanks, got it. I cannot reproduce it while having configured my PC for this timezone. Have you made any adjustments to "Markets & Symbols"? If so can you email us your Markets.txt and Symbols.txt from the WL8 user data folder?

Files Markets.txt and Symbols.txt sent

Not finding yet how the Symbols.txt would have affected USDJPY. :/

USDJPY did not enter the file Symbols.txt. Futures only.

Right, I meant to tell that Markets & Symbols supposedly don't play a role.

I spent a couple more hours around this: unfortunately, I don’t have much to add that can help reproduce this.

I’ve tried again with a new ASCII dataset, created from scratch; I even renamed the symbol to a weird string to make sure it is unique among all the symbols in the various datasets.
I’ve tried to play with the timezone of my PC; opening multiple strategies; even restarting Wealth-Lab between operations.

Still, I arrive at the same: the last bar shows in the Chart, but it’s ignored in the Backtest. I think I will park this until the next build, maybe a new installation somehow fixes this behaviour. In the meantime, since my data provider now allows export to Metastock, this is actually not a big problem for me anymore.

I'm sorry none of us were able to reproduce this, still thinking about it though and hopefully something will come to me. Maybe we can schedule a Zoom call to see firsthand, we might be missing something that we could see on the call. If you're interested email us at support@wealth-lab.com.

We've got a handle on it now. I'm pretty confident it will be fixed for Build 7.

That's great. Thank you!

Just a quick update, after installing B7: as far as I can see, the problem has indeed been solved. I tried multiple tests and could not reproduce the problem anymore.

Thank you very much for the fix!