DataSetProvider-API-Symbols-Loaded-in-backtest-are-less-than-the-symbol-count

2021-08-17T06:32:00Z - 2021-08-17T06:32:00Z ago

DataSetProvider API: Symbols Loaded in backtest are less than the symbol count

I using Extension API of DataSetProvider let WL7 to read MySQL

open WL7 to display the dataset, and the symbol(s) is also fully displayed

then run backtest with this dataset on WL7,

the problem that occurs is that after the run backtest, Symbol(s) Loaded is not complete symbol data. The complete data has 626 Symbols, and the Symbol(s) Loaded result is 600 or 57x...618

and I clicked on the Symbol(s) Loaded to re-execute the run backtest, and the loaded data is different from the last loaded data. The missing data is not fixed, it's random.

What is the logic behind ?

First Run Backtest result：

Clicked Symbol(s) Loaded Second Run Backtest result：

Run Backtest code：

CODE:

using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Indicators;
using System.Drawing;
using System.Collections.Generic;

namespace WealthScript2 
{
    public class MyStrategy : UserStrategyBase
    {
        public override void Initialize(BarHistory bars)
        {
        }

        public override void Execute(BarHistory bars, int idx)
        {
        }
    }
}

Thx.

#WealthLab7

1,546

9 Replies

Bookmark

Notify

Sort

Eugene8

2021-08-17T07:02:29Z - 2021-08-17T07:02:29Z ago

"N symbols loaded" does not necessarily indicate an issue. Have you verified that all your symbols have data in daily scale? You can check it in the Data Manager.

P.S. I wonder whether your DataSet Provider API choice is on purpose because usually we build static providers against this API:
https://www.wealth-lab.com/Support/ExtensionApi/HistoricalDataProvider

Probably because the symbol list is fixed with what's included in your MySQL table and it makes little sense for on demand symbol entry capability such as in New DataSet Wizard (although these too can be limited like Metastock or ASCII), right?

changhengyiu8

2021-08-17T08:57:07Z - 2021-08-17T08:57:07Z ago

QUOTE:

"N symbols loaded" does not necessarily indicate an issue. Have you verified that all your symbols have data in daily scale? You can check it in the Data Manager.

The dataset in data manager information.

The data has been sorted by bar count ascending.

QUOTE:

P.S. I wonder whether your DataSet Provider API choice is on purpose because usually we build static providers against this API:
https://www.wealth-lab.com/Support/ExtensionApi/HistoricalDataProvider

Is using GetHistoryInternal of HistorialDataProvider ?

In the GetHistory of DataSetProvider explain : WL7 calls this method to request historical data from your DataSet

Isn't use GetHistory ?

Thx

Eugene8

2021-08-17T17:39:23Z - 2021-08-17T17:39:23Z ago

QUOTE:
Is using GetHistoryInternal of HistorialDataProvider ?

Not quite sure what you mean. Your class that inherits from DataSet that overrides GetHistory and returns data.

Eugene8

2021-08-17T17:41:11Z - 2021-08-17T17:41:11Z ago

@Glitch

Dion, what else do you think the difference in symbol count vs. symbols loaded could be attributed to?

Glitch8

( 12.53% )

2021-08-17T17:52:56Z - 2021-08-17T17:52:56Z ago

The only explanation is that some of the symbols returned null or zero bars. Perhaps the HDP is coded in such a way that the parallel processing of WL is causing sporadic symbols to error out and not load.

changhengyiu8

2021-08-18T01:28:17Z - 2021-08-18T01:28:17Z ago

QUOTE:

Perhaps the HDP is coded in such a way that the parallel processing of WL is causing sporadic symbols to error out and not load.

I thought so at the beginning.

I repeated the code several times and used log4net to print the log, but couldn’t find the problem. no returned null and zero bars.

GetHistory is written like this：

CODE:

        public override BarHistory GetHistory(string symbol, HistoryScale scale, DateTime startDate, DateTime endDate, int maxBars, GetHistoryControlBlock cb) {


            if (!this.Symbols.Contains(symbol)) return null;
            if (scale == null) return null;

            BarHistory barHistory = new BarHistory(symbol, scale)
            {
                SecurityName = symbol
            };
            using MySqlConnection connection = DbUtils.GetConnection();

            string querySymbol = String.Format(@"SELECT symbol_code,
                                                        symbol_name,
                                                        stock_code,
                                                        date,
                                                       open,
                                                        high,
                                                        low,
                                                       close,
                                                        volume,
                                                        amount,
                                                        pbvalue
                                                 FROM `re_nct`.`all_data` 
                                                WHERE symbol_code = '{0}'", symbol);
            MySqlCommand commandSymbol = new MySqlCommand(querySymbol, connection);
            MySqlDataReader dataReaderSymbol = commandSymbol.ExecuteReader();

            TimeSeries tsAmount = new TimeSeries();
            TimeSeries tsPbvalue = new TimeSeries();

            DateTime dateTime;

            while (dataReaderSymbol.Read()) {
                dateTime = DateTime.Parse(dataReaderSymbol["date"].ToString());
                double.TryParse(dataReaderSymbol["open"].ToString(), out double open);
                double.TryParse(dataReaderSymbol["high"].ToString(), out double high);
                double.TryParse(dataReaderSymbol["low"].ToString(), out double low);
                double.TryParse(dataReaderSymbol["close"].ToString(), out double close);
                double.TryParse(dataReaderSymbol["volume"].ToString(), out double volume);

                barHistory.Add(dateTime, open, high, low, close, volume);

                double.TryParse(dataReaderSymbol["amount"].ToString(), out double amount);
                tsAmount.Values.Add(amount);
                tsAmount.DateTimes.Add(dateTime);

                double.TryParse(dataReaderSymbol["pbvalue"].ToString(), out double pbvalue);
                tsPbvalue.Values.Add(pbvalue);
                tsPbvalue.DateTimes.Add(dateTime);

            }
            dataReaderSymbol.Close();

            barHistory.NamedSeries.Add("amount", tsAmount);
            barHistory.NamedSeries.Add("pbvalue", tsPbvalue);

            DbUtils.CloseConnection(connection);
            return barHistory;
        }

Thx.

Eugene8

2021-08-18T05:14:04Z - 2021-08-18T05:14:04Z ago

I'm by no means a MySql expert. Effectively, what you're doing isn't much different from the way the WL6 database provider requests data from a database. However, due to improvements in WL7 you have to pay attention to thread safety.

1. Is DbUtils.GetConnection is a singleton?

2. Have you tried to reuse the connection vs. DbUtils.CloseConnection?

3. Are MySqlConnection and ExecuteReader thread safe?

https://bugs.mysql.com/bug.php?id=59887
https://bugs.mysql.com/bug.php?id=59944

Glitch8

( 12.53% )

2021-08-18T11:12:53Z - 2021-08-18T11:12:53Z ago

You can easily see if parallel processing concerns are behind any anomalies by using this pattern to force the HDP into non-parallel mode:

CODE:

public override BarHistory GetHistory(string symbol, HistoryScale scale, DateTime startDate, DateTime endDate, int maxBars, GetHistoryControlBlock cb) 
{
   lock(_lock)
   {
      //your HDP logic
   }
}

private static object _lock = new object();

changhengyiu8

2021-08-19T02:16:33Z - 2021-08-19T02:16:33Z ago

@Eugene @Glitch

I will try again.

Thx

Bookmark

Notify

Sort