Yahoo-Dividends-for-mutual-funds-doesn-t-include-capital-gains

Another thing I noticed is that I seem to be missing the latest dividend. For example, if you get the dividends for FCNTL on Yahoo Finance, the last dividend was on Dec 9, 2022, but the last dividend returned from WL 8 is Feb 4, 2022. Providing code and results in debug log...

CODE:
using WealthLab.Backtest;
using System;
using System.Text;
using WealthLab.Core;
using WealthLab.Indicators;
using System.Drawing;
using System.Collections.Generic;

namespace WealthScript1 
{
    public class MyStrategy : UserStrategyBase
    {
        //create indicators and other objects here, this is executed prior to the main trading loop
        public override void Initialize(BarHistory bars)
        {
         List<EventDataPoint> DividendList = bars.GetEventDataPoints("Dividend");

         foreach (EventDataPoint item in DividendList)
         {
            //try
            {
               StringBuilder sb = new StringBuilder(1000);
               sb.Append(item.Date.ToShortDateString()).Append('\t');
               sb.Append(item.Value.ToString()).Append('\t');

               if ( item.Details.Count == 0)
               {
                  sb.Append("<< no details >>");
               }
               WriteToDebugLog(sb.ToString());
            }
            //catch (Exception)
            {
            }
         }
      }

        //execute the strategy rules here, this is executed once for each bar in the backtest history
        public override void Execute(BarHistory bars, int idx)
        {
            if (!HasOpenPosition(bars, PositionType.Long))
            {
                //code your buy conditions here
            }
            else
            {
                //code your sell conditions here
            }
        }

      //declare private variables below

   }
}

QUOTE:
---Symbol by Symbol Debug Logs---
---FCNTX---
<< rows from 12/15/1980 through 2018 deleted >>
2/8/2019 0 << no details >>
12/13/2019 0 << no details >>
2/7/2020 0 << no details >>
12/11/2020 0 << no details >>
2/12/2021 0 << no details >>
12/10/2021 0 << no details >>
2/4/2022 0 << no details >>

What I see on Yahoo Finance:

QUOTE:
Another thing I noticed is that I seem to be missing the latest dividend.

Have you tried out different event providers? There's no shortage of them in Extensions: Alphavantage, FinancialModelingPrep, Morningstar, Nasdaq, Quandl, Tiingo, WealthData and YCharts. Maybe some of them even include capital gains?

I had not tried any of the extensions as I have the basic subscription. That would cost me extra. Currently I only have Wealth-Data and Yahoo Finance for data providers that provide events. Wealth-Data returns null for mutual fund symbol FCNTX.

I added a key for AlphaVantage and enabled it. It does seem to find FCNTX and my dividend test strategy is getting dividend and capital gains in the dividend events including those on 12/9/2022 even though I see "Data Source: Yahoo! Finance" on the chart status. Due to the order of my event providers, it appears to be using the first event provider with the symbol - I'm assuming AlphaVantage.

I unchecked AlphaVantage as an event provider (in Data Manager). When I close and reopen my dividend test strategy then Run Backtest, I only get dividend in the dividend events (assume using "Yahoo Finance" events), but it now includes those on 12/9/2022. I used the 'Historical Data Loading Testbed' feature to test which providers had symbol 'FCNTX' since my last post. Is it possible this refreshed event data for the provider when it didn't when I ran the strategy?? I'm a little confused why 12/9/2022 data is present with only the Yahoo Finance provider active for events when I couldn't get it earlier. I'm pretty sure I tried pop up menu on the chart "Reload Chart Data from Provider". Does this just mean price data only? If so, how can I force event data to be reloaded. It could get confusing if my price data is updated, but my events data is old.

It looks like setting up AlphaVantage to get dividend events will fix my issue so I can now see capital gains with mutual funds. Thanks.

Last, it looks like I can reorder Historical Providers in Data Manager with drag and drop. It doesn't look like I can reorder Event Providers. Is that correct?

Event providers aren't associated with other providers. If you check Event provider(s), all will be queried for every symbol you request - the order is immaterial. Note that using multiple Events can result in significant delays when requesting data. (Read: check only the ones you really need.)

How do I know which event provider is providing dividends in my case? I believe AlphaVantage is providing the correct data whereas Yahoo Finance is incomplete (due to the lack of capital gains data) for mutual funds. I'm comparing funds in the strategy, so a mix of event providers could give me inconsistent results.

See the EventDataPoint class. ProviderName is the source.

If I link a dataset to a provider, will it only use that provider for prices and events?

I answered my own question by running it against "Fidelity Select Sector funds" dataset that is linked to "Yahoo Finance" (standard dataset with WL). The answer is that it will still use multiple event providers similar to the example below.

When I added the provider name to the dividend items for FCNTX, I got a result I wasn't expecting:

QUOTE:
---Symbol by Symbol Debug Logs---
---FCNTX---
<< rows from 12/15/1980 through 2018 deleted >>
Yahoo! Finance 2/8/2019 0 << no details >>
AlphaVantage 2/8/2019 0.106 << no details >>
Yahoo! Finance 12/13/2019 0 << no details >>
AlphaVantage 12/13/2019 0.464 << no details >>
AlphaVantage 2/7/2020 0.097 << no details >>
Yahoo! Finance 2/7/2020 0 << no details >>
Yahoo! Finance 12/11/2020 0 << no details >>
AlphaVantage 12/11/2020 1.246 << no details >>
AlphaVantage 2/12/2021 0.4 << no details >>
Yahoo! Finance 2/12/2021 0 << no details >>
AlphaVantage 12/10/2021 1.627 << no details >>
Yahoo! Finance 12/10/2021 0 << no details >>
AlphaVantage 1/4/2022 0.215 << no details >>
Yahoo! Finance 2/4/2022 0 << no details >>
AlphaVantage 2/4/2022 0.215 << no details >>
Yahoo! Finance 12/9/2022 0.077 << no details >>
AlphaVantage 12/9/2022 1.22 << no details >>

Right away I see issues:
* Fidelity has a capital gains distribution of 1.143 on 12/9/2022. That matches what Yahoo Finance has on their site (but missing in the provider data) but is different than 1.22 that AlphaVantage is providing. I suspect AlphaVantage provider added the dividend and capital gains values to get to the 1.22 (1.143 + 0.077). If Yahoo is fixed to return a separate capital gains value of 1.143 both Alpha and Yahoo capital gains values will be provided due to the discrepancy (see below).
* Item 1/4/2022 from AlphaVantage isn't listed at Fidelity.

Mixing data like this can make it hard to find data errors.

This data range looks suspicious for FCNTX too:

QUOTE:
Yahoo! Finance 12/28/2001 0.022 << no details >>
AlphaVantage 12/28/2001 0.22 << no details >>
Yahoo! Finance 6/7/2002 0.23 << no details >>
AlphaVantage 6/7/2002 2.3 << no details >>
Yahoo! Finance 12/27/2002 0.005 << no details >>
AlphaVantage 12/27/2002 0.05 << no details >>
Yahoo! Finance 12/26/2003 0.004 << no details >>
AlphaVantage 12/26/2003 0.04 << no details >>
AlphaVantage 2/6/2004 0.01 << no details >>
Yahoo! Finance 2/6/2004 0.001 << no details >>
Yahoo! Finance 12/23/2004 0.0035 << no details >>
AlphaVantage 12/23/2004 0.035 << no details >>

Notice there's a 10x difference between Yahoo and Alpha data on the same date causing both to be provided. I'm not sure why. According to Yahoo web site, only a dividend was issued, but because these two values were different both were provided on the same day. It's a data issue but made worse by merging different values. Discrepancies seem to cause both values to be returned. In my case I would be adding them together for that date assuming they are dividend and capital gains.

I just wanted to share my results as it wasn't what I was expecting. Hopefully it'll help others. I'm not sure how I want to handle this in my strategy.

First, let me say that Y! is a free and unreliable data provider.

In the last year, they starting limiting requests from ips, so if you want to update more than a few dozen symbols, just don't count on it. You'll be blocked.

For me, I entered and updated FCNTX this morning. The 12/9/2022 0.077 dividend is there. I don't use or select Yahoo! data normally, so when I do it for things like this, I get the data because I won't be blocked.

Finally, the Yahoo! historic dividends are correct. They're adjusted for the 10:1 split registered in August 2018. Apparently, AV doesn't perform that adjustment.

Not many providers seem to carry dividends for FCNTX. The YCharts dividends for FCNTX don't seem to be adjusted either.

P.S. But taking SPY for example, the last few dividends by Yahoo, YCharts and Morningstar which I sampled all do match.

QUOTE:
Cone: First, let me say that Y! is a free and unreliable data provider.

In the last year, they starting limiting requests from ips, so if you want to update more than a few dozen symbols, just don't count on it. You'll be blocked.

Good to know. I'll try to limit using it. I try to use Wealth-Data as a quote source as it's the most reliable among the free sources to use with WL. Wealth-Data lacks mutual funds so I need to find another reliable source that also includes dividends and capital gains distributions in events. Even paid sources seem to lack mutual funds (although I'm not considering them at the moment).

One trick I used in the past was to download all data at the beginning of the day and then go offline to run my strategies. That meant I'd hit the providers only once during the day and hopefully get all the data I needed before I got blocked. It was faster to run strategies multiple times (no hangs loading data) and the data was consistent while developing strategies. I believe WL has some of that functionality too.

QUOTE:
Cone: the Yahoo! historic dividends are correct.

Yes, dividends are correct, but I need capital gains distributions as well as dividends with mutual funds. Capital gains are available on the Y! web site but missing in the Yahoo! event provider. It appears AlphaVantage includes capital gains. The event providers need to be consistent.

Thanks for finding the reason for the discrepancy in the data between 2001 and 2004. That was the only window where I spotted a discrepancy so I didn't look as far out as 2018. I looked for an event like a split prior to 2010. AlphaVantage must adjust for the split otherwise I would have seen more discrepancies prior to 2018. I only noticed issues 2001 - 2004.

QUOTE:
Eugene: Not many providers seem to carry dividends for FCNTX.

That's my problem using mutual funds in my strategies. My MF rotation strategies are less reliable with WL8 than the Fidelity version because I have a hard time finding data providers that have all the MF data I need - prices and distributions (div+cap gains).

* * *

Today I'm having trouble reproducing my test samples in previous posts, so I'll try to comment by memory. (It could be Yahoo! has blocked me from FCNTX requests.)

The issue I have is it appears the events from multiple providers are merged together when requesting dividends. When there are no discrepancies between the data providers, the result is good; however, discrepancies in the data causes an interleaving of data that is a mess that is difficult to work with. I would prefer if the result from one provider was returned rather than interleaving data like I saw in my previous post.

A nice to have would be an audit showing there is discrepancies between providers. I would expect this in an error log. However, if only one event provider is needed, perhaps we wouldn't have to send a request to multiple reducing the risk of being blocked.

QUOTE:
I believe WL has some of that functionality too.

Right, in WL we have 1) offline mode and 2) avoid making repeated queries and otherwise try to be smart w.r.t. downloading event data under the hood.

QUOTE:
Capital gains are available on the Y! web site but missing in the Yahoo! event provider

Whatever is not returned by our provider is missing on "the Y! web site" by which I mean the API endpoint the provider queries. Anything else like some web pages at Y! Finance are irrelevant.

QUOTE:
The issue I have is it appears the events from multiple providers are merged together when requesting dividends.

No. Each event provider, if enabled by its checkbox, captures its dividend item data and caches it in its own folder. They do not crosstalk.

QUOTE:
Whatever is not returned by our provider is missing on "the Y! web site"

Is it possible Y! has separate apis for "dividends" and "capital gains"? Yahoo separates the two on their web site, so they might separate it in the api as well. WL (and Fidelity WLP) providers have typically combined the two as dividends in the past.

QUOTE:
No. Each event provider, if enabled by its checkbox, captures its dividend item data and caches it in its own folder. They do not crosstalk.

Then why am I seeing the interleaving results in Post #8. That result is hard to work with. It seems like something is combining results.

I'll admit I'm running it today and I don't see the same result as Post #8. If I uncheck AlphaVantage as an event provider and check Yahoo, I get no dividend results, so I'm assuming Yahoo has stopped returning event data for FCNTX. Now my result is only AlphaVantage dividend data if I check both event providers in Data Manager. That's what I want to see but Post #8 is what I will probably see if Yahoo returns results in the future.

Only checking AlphaVantage events provider might be my temporary solution to run my mutual fund strategies. Too bad as Yahoo data seems to be more accurate than AlphaVantage, but Yahoo lacks the capital gains I need. I think I've had issues with both Yahoo and AlphaVantage blocking me in the past after too many requests.

QUOTE:
Is it possible Y! has separate apis for "dividends" and "capital gains"?

You're right, Y! separates capital gains from dividends. We could introduce a new event item in WL8 for Capital Gain for on demand requests. They would show up on charts as a separate item along with Dividend and Split. What do you think?

I like it 🤷🏼‍♂️

QUOTE:
Y! separates capital gains from dividends. We could introduce a new event item in WL8 for Capital Gain for on demand requests.

I think you need to be consistent between event providers with the results. Since AlphaVantage provider for mutual fund dividends appears to include capital gains and dividends (as EventDataPoint.Name = "Dividend"), I think Yahoo event provider should too. That would be consistent with Fidelity's WLP as well for legacy strategies migrated (with Fidelity i remember getting two results for the same day, dividend and capital gains, which I had to sum together in my strategy). If you're concerned about breaking existing code, you could put a check box on the Yahoo event provider "include capital gains with dividends" in Data Manager. I think this only affects strategies that use mutual fund securities so I assume there are fewer strategies that would be affected.

I like the idea of dividends and capital gains showing up separately on the charts, but as event data to strategies it would be nice if they were combined.

Take AlphaVantage event provider. It appears to sum capital gains and dividends for a date to show one result. Class EventDataPoint has a Details collection which could be used to show the dividend and capital gains breakdown which you could use to show in the chart (popup text on the dividend icon on the chart to see breakdown). Currently the Details collection doesn't appear to be populated so you are less likely to break existing code by populating it with this breakdown.

Those are my thoughts.

We should be careful not to add extra check boxes that would distract or confuse the majority of users.

Combining the dividend and capital gain events could introduce complexity, making the code error-prone in the times of Yahoo being less reliable.

Alphavantage's API doesn't seem to return capital gain.

Here's what's coming up in B28:

QUOTE:
Alphavantage's API doesn't seem to return capital gain.

AlphaVantage values returned from GetEventDataPoints("Dividend") appear to sum dividend and capital gains as Dividends. For example, look at my Post #8. AlphaVantage for 12/9/2022 shows 1.22 as result. Fidelity web site shows dividend = 0.077 and capital gains = 1.143 on that date -- sum is 1.22 (see https://fundresearch.fidelity.com/mutual-funds/fees-and-prices/316071109?type=o-NavBar). That's why I say AlphaVantage includes capital gains.

You may not have an api from AlphaVantage that allows you to see the breakdown, but the value reflects both dividends and capital gains.

I like your chart addition for Yahoo!, but it doesn't help my strategy unless I can see dividends and captial gains with my call to GetEventDataPoints("Dividend"). In the past, I found providers to return both dividends and capital gains as dividends for mutual funds.

Yahoo event provider for dividends is unusable for me with mutual funds in it's current state since it lacks capital gains.

We will not combine capital gain with dividend in Yahoo provider because it can really cause more trouble but you might consider adjusting your strategy to sum up dividends with the upcoming event item.

I'll look at it when the release comes out, but providers should be consistent whenever possible.

For those like me who don't trade mutual funds, you might not know the difference between dividends and capital gain distributions...

https://finance.yahoo.com/news/understanding-differences-between-dividends-distributions-154220285.html

WealthLab doesn't deal with tax consequences, so I don't see a reason to make it a different item... but you might be able to do something with it in your script.

OK I changed my mind. Let's avoid creating the new event and instead embed capital gain into dividend for mutual fund symbols. If the two events happen on one bar then their values get summed up. When there's a capital gain w/o a dividend on the same bar it will simply be inserted as a new Dividend. Below is an example for FCNTX. Hope this helps.

Thanks Eugene. That appears to be what I was looking for in the Yahoo event provider.

Cone: You're right the dividend and capital gains have different tax consequences but we don't need to be concerned in WL. (I actually use them in a non-taxable account to avoid any tax issues.) Most mutual fund investors don't see much of a change in value in their holdings as the dividend and capital gains distributions are reinvested automatically. However, the charts show a big drop because the price is adjusted for the distributions. I need to add the distributions to the price to prevent the sudden drop from signaling a sell. In FCNTX, it was a 9% drop overnight due to the distributions most of that being due to capital gains. There were plenty of other mutual funds that had similar behavior, but I've been using this one as an example.

I'm still concerned about the interleaving of AlphaVantage and Yahoo entries shown in Post #8. It seems to happen when there's a discrepancy between event providers values. That could cause an over adjustment due to distributions.

Eugene, your chart looks different than mine in Post #22. Do you have settings that cause the price to be adjusted for dividends automatically?

I recently installed B30 and the code in Post #1 will not compile as 'EventDataPoint' could not be found. Am I missing something??

@thodder - try updating your CandleSticks extension, at least, if not all extensions.

@thodder
There is a breaking change, code in Post #1 is missing a new reference to

CODE:
using WealthLab.Data;

Eugene:

Thanks that fixed it for me.

Fyi, the code sample in help for GetEventDataPoints method needs this fix as well. That sample wouldn't compile too.

QUOTE:
Fyi, the code sample in help for GetEventDataPoints method needs this fix as well. That sample wouldn't compile too.

Fixed, thanks.