Strategy-Monitor-with-my-custom-Alor-SDP-does-not-trigger-execution-on-UpdateHeartbeat

What is your computer time zone set at? The Moscow Market is defined for UTC+3:00.

It looks like maybe the Alor Streaming Provider is not sending quotes in the correct (Moscow) time zone?

Thanks.
My PC is UTC+3.00 (Moscow).

If I'm right, Alor sends as UTC + 0.00.

changing

CODE:
public void UHB()
        {
            UpdateHeartbeat(DateTime.Now);
        }

to

CODE:
public void UHB()
        {
            UpdateHeartbeat(DateTime.UtcNow);
        }

didn't help, but maybe there something left in this direction, I'll keep on experimenting.

The heartbeat should remain you're PC's local time. You can change that back. The reason is that heartbeat is independent of market, and a SDP could conceivably support even multiple markets.

It's the Alor streaming provider that needs to send the quotes in the proper time zone for the market of the symbol (Moscow).

Oh, I see, thanks for your help!

Heartbeat should be in computer local time ... UpdateHeartbeat(DateTime.Now)

Sorry, deleted the post above.

I use made changes in this lines:

CODE:
UpdateTick(s.Symbol, DateTime.Now.ToLocalTime(), data.GetProperty("last_price").GetDouble(), data.GetProperty("volume").GetDouble() - s.LastVolume, data.GetProperty("prev_close_price").GetDouble());

CODE:
                        BarData bars = new BarData(UnixTimeStampToDateTime(data.GetProperty("time").GetInt64()).ToLocalTime(), data.GetProperty("open").GetDouble(), data.GetProperty("high").GetDouble(), data.GetProperty("low").GetDouble(), data.GetProperty("close").GetDouble(), data.GetProperty("volume").GetDouble());
                        UpdateStreamingBar(s.Symbol, scale, bars);

Added: ToLocalTime().

I also replaced it with Now instead of UtcNow:

CODE:
public void UHB()
        {
            UpdateHeartbeat(DateTime.Now);
        }

Maybe there are some other places to change the time zone in SDP class - I didn't find any more. I mean it didn't help).

it looks like your streaming provider is updating quotes in local time. it should not use local time, it should explicitly use the market time zone. you can use MartketDetails ConvertLocalTimeToMarketTime as a shortcut.

Since your computer is in Moscow time the code should be ok as far as i can see. but maybe not portable to other users. when you run quotes now what kind of time stamps are you seeing?

QUOTE:
Since your computer is in Moscow time the code should be ok as far as i can see. but maybe not portable to other users. when you run quotes now what kind of time stamps are you seeing?

The same, It would be correct if you add P.M.).

You can send your provider dll to us via email and we could help troubleshoot?

I can even send the whole SDP project)), I just need to get token for a sandbox first.

QUOTE:
I can even send the whole SDP project)), I just need to get token for a sandbox first.

Done).

Hey Replikant,

I have the SDP code and I see that it's just using DateTime.Now to determine the tick timestamp. When I stream this in the testbed it's indeed giving me quotes in my local computer time zone. I found the same thing in the Quotes tool. Can you try the Streaming Testbed and check the timestamp?

btw - it's using .NET ToShortTimeString to format the time. This uses the format specified in the current CultureInfo DateTimeFormatInfo.ShortTimePattern property. I wonder if this property simply is incorrectly not displaying an AM/PM on your computer?

https://docs.microsoft.com/en-us/dotnet/api/system.datetime.toshorttimestring?view=net-5.0

OK one idea I had = is the Alor Historical Provider assigning a value to the Market property of the BarHistory?

It should assign MarketManager.FindMarket("Moscow Stock Exchange")

If the historical data has a different Market assigned (like the default USAStocks) then I could see why the heartbeat update would fail.

QUOTE:
I have the SDP code and I see that it's just using DateTime.Now to determine the tick timestamp. When I stream this in the testbed it's indeed giving me quotes in my local computer time zone. I found the same thing in the Quotes tool. Can you try the Streaming Testbed and check the timestamp?

QUOTE:
It should assign MarketManager.FindMarket("Moscow Stock Exchange")

If the historical data has a different Market assigned (like the default USAStocks) then I could see why the heartbeat update would fail.

I'll check that. Will this only effect if I use Alor HDP as linked source for the dataset selected for the strategy at Strategy Monitor? Does it mean it should work fine with other HDPs (withing this guess).

Dion, that timestamp in Quotes (from Post #20) looks like a smoking gun?

QUOTE:
It should assign MarketManager.FindMarket("Moscow Stock Exchange")

I found this:

CODE:
MarketDetails md;
            if (!SymbolMarket.TryGetValue(symbol, out md))
            {
                string exchange = "MOEX";
                s = GetJsonRoot(WebApiURL + $"/md/v2/securities?query={symbol}&exchange={exchange}&format=Simple")[0];
                string primary_board = s.GetProperty("primary_board").GetString();
                string market = $"{exchange}_{primary_board}";

                md = MarketManager.FindMarket(market);
                if (md == null)
                {
                    md = new MarketDetails();
                    md.Name = market;
                    md.BaseTimeZone = "Russian Standard Time";
                    md.BenchmarkSymbol = "SBER";
                    md.DisplayDecimals = 2;
                    md.QuantityDecimals = 0;
                    md.Source = Name;
                    for (int dow = 1; dow < 6; dow++)
                        md.SpecifyMarketHours(dow, TimeSpan.FromHours(7), TimeSpan.FromMinutes(23 * 60 + 49));

                    //md.SpecifyMarketHours(DayOfWeek.Monday, TimeSpan.)

                    List<DayOfWeek> tdow = new List<DayOfWeek>();
                    tdow.Add(DayOfWeek.Monday);
                    tdow.Add(DayOfWeek.Saturday);
                    tdow.Add(DayOfWeek.Wednesday);
                    tdow.Add(DayOfWeek.Thursday);
                    tdow.Add(DayOfWeek.Friday);

                    md.TradingDaysOfWeek = tdow;

                    MarketManager.Markets.Add(md);
                    md = MarketManager.FindMarket(market);
                    SymbolMarket[symbol] = md;
                }
            }

It looks like we create our own MarketDetails, we needed to separate derivatives and stocks as I remember - I don't remember why) - something with lots maybe - stocks deal with lots, futures - with single contracts.

As long as you assign that to the BarHistory Market property in the HDP, and the MarketDetails has the correct time zone, it should be OK.

CODE:
tdow.Add(DayOfWeek.Monday);
tdow.Add(DayOfWeek.Saturday);
...

So on Tuesdays the market is not trading? (Of course we know not it's not true). Today is Tuesday.

QUOTE:
So on Tuesdays the market is not trading? (Of course we know not it's not true). Today is Tuesday.

Thanks for bug report, Eugene, fixed for the next build)).

QUOTE:
As long as you assign that to the BarHistory Market property in the HDP, and the MarketDetails has the correct time zone, it should be OK.

What does this mean? Does this mean that according to the screen from comment #23 it can't be the reason and I need to search somewhere else?

And as debugging with strategy monitor is quite costly can I use Quotes for this? like the goal is to see the same datetime in Quotes as in testbed?

OK, we discovered why the Quotes are appearing 12 hours delayed, it's a formatting issue (display only) that's fixed for Build 16.

As for your code above, I'm sure sure where that's from? The point is, your HDP must assign the correct MarketDetails instance to the BarHistory that it returns in GetHistoryInternal (its Market property).

QUOTE:
OK, we discovered why the Quotes are appearing 12 hours delayed, it's a formatting issue (display only) that's fixed for Build 16.

And it's not the reason for my core issues? Just for quotes? - As heartbeat effects Strategy Monitor.

QUOTE:
As for your code above, I'm sure sure where that's from? The point is, your HDP must assign the correct MarketDetails instance to the BarHistory that it returns in GetHistoryInternal (its Market property).

Ok, I'll work with that. The screen was from Alor HDP.

No it wasn't the reason, it was only displaying incorrectly.

I hope you'll keep on helping with my struggling)).

I added this to Execute():

CODE:

public override void Execute(BarHistory bars, int idx)
        {
         File.AppendAllText("C:\\Users\\Mihail\\Desktop\\data_.txt", bars.DateTimes[idx].ToString() + "\n");
...
}

This is the result prints, can you see this gap?

17.08.2021 15:58:00
17.08.2021 15:58:00
17.08.2021 15:59:00
17.08.2021 15:59:00
17.08.2021 16:00:00
17.08.2021 16:00:00
17.08.2021 21:03:00
17.08.2021 21:03:00

Current local time: 21:xx:xx

I tried Alor HDP and Finam HDP. I used _MOEX_ market in both cases, result is the same (I mean this gap).

Here is market settings:



Here is strategy settings:

This doesn’t give me a clear picture. You added some code that appends lines to an existing file. The gap could simply mean you ran a strategy, waited a while, and then ran it again.

We already can see when the strategy executed in the SM log pane, so i don’t think that appending to this log file is providing anything more useful.

Can you print the value of the BarHistory Market property to double check that?

The file was empty befor I ran the strategy in Strategy Monitor - so these data are from HDP and SDP, like HDP data.. gap... SDP data.

My guess.

then it looks like there might be an issue with the HDP population the intraday bars, incorrect time zone or time?

your appending a line to the file for each Execute call. are you running more than one symbol at a time? this is a lot of file access which might be problematic, but maybe pretend the symbol too so you can see which symbol each line represents?

Yes, there were 2 or 3 symols in the dataset:

Here is the result of printing bars.Market.Name from Execute()...

US Stocks17.08.2021 15:52:00
US Stocks17.08.2021 15:53:00
US Stocks17.08.2021 15:54:00
US Stocks17.08.2021 15:55:00
US Stocks17.08.2021 15:56:00
US Stocks17.08.2021 15:57:00
US Stocks17.08.2021 15:58:00
US Stocks17.08.2021 15:59:00
US Stocks17.08.2021 16:00:00
US Stocks17.08.2021 22:15:00

I had the same "heartbeat does not effect" problem for all combinations of Russian HDP, SDP, including Eugene's extension. So maybe it's something on your side.

It's hard to catch such bugs on your side as you are concentrated on US markets. If something is hardcoded it will work fine for US).

The problem is as I suspected, the BarHistories are assigned the USAStocks market, not the Russian market. This would need to be changed in your Historical Data Provider.

Another thought is, we could assign the property to the BarHistories within the StrategyMonitor itself, I don't think this would have any side effects but we'd have to test it.

But, really your HDP should assign the correct Market to the BarHistories it returns. I think that if you just override GetSymbolMarket in the HDP, then WL7 will use that value and assign the correct market.

Thanks, Glitch!

Although it looks like a whole lot of work for me)).

Alor HDP gets Symbols data from broker on init to fill Tick Size (at least) as we need this to be not rejected by broker when sending orders. But as I can see in Markets and Symbols table - the Market column is blank, so this code can't get the Market, so I guess it sets default one.

CODE:

_history = new BarHistory(symbol, scale);
_history.SecurityName = GetSecurityName(symbol);
_history.Market = GetSymbolMarket(symbol);

You needn't call GetSymbolMarket.

You need to IMPLEMENT GetSymbolMarket, like this maybe ...

CODE:

public override MarketDetails GetSymbolMarket(string symbol)
{
   return MarketManager.FindMarket("_MOEX_");
}

Or maybe your HDP supports multiple markets? In which case just add whichever logic makes sense based on the symbol.

QUOTE:
Although it looks like a whole lot of work for me)).

No pain no gain 😀

Now I know where to go, and what to do). Thanks guys!

QUOTE:
No pain no gain 😀

Now I know you'll get those account positions working in QUIK!

QUOTE:
Alor HDP gets Symbols data from broker on init to fill Tick Size (at least) as we need this to be not rejected by broker when sending orders. But as I can see in Markets and Symbols table - the Market column is blank, so this code can't get the Market, so I guess it sets default one.

I added _MOEX_ market to Symbols file, so now HDP gets market from Symbol.

I ran the strategy using Strategy Monitor and the first candle close run works fine - on a heartbeat, but on every next candle heartbeat doesn't work, so I guess it's not only HDP, but SDP I must look at - it looks like I must set the market somewhere in SDP as well.

Yes, you should implement GetSymbolMarket in the SDP too. Follow the same logic as you did for the HDP.

QUOTE:
Yes, you should implement GetSymbolMarket in the SDP too. Follow the same logic as you did for the HDP.

So, I need help, again).

This is my code from HDP:

CODE:
private BarHistory _history;
...
_history = new BarHistory(symbol, scale);
_history.SecurityName = GetSecurityName(symbol);
_history.Market = GetSymbolMarket(symbol);
_history.SymbolInfo = GetSymbolInfo(symbol);

But I didn't find any BarHistory object in HDP, what object should I set Market to?

Hey Replikant, Eugene must have had a misfire, but StreamingDataProvider does not have the method GetSymbolMarket. If you overloaded that method in your HistoricalDataProvider, and just ensure that the StreamingDataProvider uses the correct DateTimes for the Market's time zone it should be enough.

The streaming processing will attempt to convert the heartbeat DateTime (which is local time) to the time zone associated with the HistoricalDataProvider Market.

Do you have code like this in the HDP?

CODE:

public override MarketDetails GetSymbolMarket(string symbol)
{
   return MarketManager.FindMarket("_MOEX_");
}

My bad, that's wrong advice. You should rather call the .ConvertLocalTimeToMarketTime(dateTime) method of your MarketDetails object in your SDP.

QUOTE:
Hey Replikant, Eugene must have had a misfire, but StreamingDataProvider does not have the method GetSymbolMarket. If you overloaded that method in your HistoricalDataProvider, and just ensure that the StreamingDataProvider uses the correct DateTimes for the Market's time zone it should be enough.

Hey, Glitch, looks like it almost work now), but first a couple of questions/comments:

- Can it be that symbols are processed in two groups one after another if heartbeat works? I try to understand whether it works or not - sometimes symbols (dataset of 5 symbols) are processed in two passes, although it's expected to be one as heartbeat moment is the same for all tickers.

- There are also tickers in this dataset that do not have postmarket - so no new ticks now (when I do my testing), but heartbeat should effect them too - so there is some logic about having candles or something, but what about symbols with low liquidity with rare trades and sometimes missing candles - it will not work or what is the logic for that.

There is a chance that heartbeat always worked for me but I couldn't see it because of these two factors))).

Yes, the SM can execute in multiple passes. It all depends on how timely the data updates occur.

If there is no data at all within a candle, even the heartbeat will not cause it to execute. A strategy can only execute if there is data in the bar being processed.

QUOTE:
If there is no data at all within a candle, even the heartbeat will not cause it to execute. A strategy can only execute if there is data in the bar being processed.

Oh, and this will not be a problem as I can see now as the strategy will be triggered after the last candle before the gap and after the first after the gap and no need to do something within the gap.

QUOTE:
Yes, the SM can execute in multiple passes. It all depends on how timely the data updates occur.

So maybe the reason is that some symbols has ticks even before the heartbeat (which is at the end of the first second of a new candle in my case).

Could you please look at these logs:

18.08.2021 23:23:00: Status = Processing
18.08.2021 23:23:19: Ran Strategy on AFLT,SBER: 2 Signals, Run Time=18,55sec
18.08.2021 23:23:41: Ran Strategy on MAGN,OZON,MGNT: 3 Signals, Run Time=20,88sec
18.08.2021 23:23:50: NextRun set to 18.08.2021 23:24
18.08.2021 23:23:50: Status = Incomplete
18.08.2021 23:23:50: 4 Symbols not Processed: MAGE,SAGO,GRNT,MTLR
18.08.2021 23:24:00: Status = Processing
18.08.2021 23:24:09: Ran Strategy on AFLT: 1 Signals, Run Time=9,25sec
18.08.2021 23:24:33: Ran Strategy on SBER,MAGN,OZON,MGNT: 4 Signals, Run Time=22,63sec
18.08.2021 23:24:50: NextRun set to 18.08.2021 23:25
18.08.2021 23:24:50: Status = Incomplete
18.08.2021 23:24:50: 4 Symbols not Processed: MAGE,SAGO,GRNT,MTLR
18.08.2021 23:25:00: Status = Processing
18.08.2021 23:25:11: Ran Strategy on MGNT,AFLT: 2 Signals, Run Time=10,56sec
18.08.2021 23:25:29: Ran Strategy on MAGN,OZON,SBER: 3 Signals, Run Time=16,65sec
18.08.2021 23:25:50: NextRun set to 18.08.2021 23:26
18.08.2021 23:25:50: Status = Incomplete
18.08.2021 23:25:50: 4 Symbols not Processed: MAGE,SAGO,GRNT,MTLR
18.08.2021 23:26:00: Status = Processing
18.08.2021 23:26:12: Ran Strategy on SBER: 1 Signals, Run Time=11,26sec
18.08.2021 23:26:26: Ran Strategy on MAGN,OZON,MGNT,AFLT: 4 Signals, Run Time=13,36sec
18.08.2021 23:26:50: NextRun set to 18.08.2021 23:27
18.08.2021 23:26:50: Status = Incomplete
18.08.2021 23:26:50: 4 Symbols not Processed: MAGE,SAGO,GRNT,MTLR
18.08.2021 23:27:00: Status = Processing
18.08.2021 23:27:06: Ran Strategy on SBER: 1 Signals, Run Time=5,42sec
18.08.2021 23:27:15: Ran Strategy on MAGN,OZON,MGNT,AFLT: 4 Signals, Run Time=8,41sec
18.08.2021 23:27:50: NextRun set to 18.08.2021 23:28
18.08.2021 23:27:50: Status = Incomplete
18.08.2021 23:27:50: 4 Symbols not Processed: MAGE,SAGO,GRNT,MTLR
18.08.2021 23:28:00: Status = Processing
18.08.2021 23:28:06: Ran Strategy on SBER: 1 Signals, Run Time=5,75sec
18.08.2021 23:28:14: Ran Strategy on MAGN,OZON,MGNT,AFLT: 4 Signals, Run Time=7,14sec

Comments:

- "4 Symbols not Processed: MAGE,SAGO,GRNT,MTLR" - these are symbols with no candles (do not have postmarket) - it's OK.
- 20 seconds per pass - is when my slow laptop was out of power and slowed down even more).

- As I can see passes are run one by one with no parallelism, is that true?. Looks like a place for emprovement as symbols that have data have to wait vetween heartbeat moment and the next pass.

Just my guesses and some thoughts, nothing more). I'll also try on my powerful PC to check the perfomance.

that’s not true. multiple passes only occur when there is some delay in the updates. if all of the symbols were updated at once then there would only be one pass (i think one of Cones videos on the software page shows this with the S&P 100).