Where-can-I-download-years-of-1-minute-bars-historic-data-for-SPY

Hello.

If you are looking for free data, you could look at the sources listed here. (just for starters). If it is really only about the SPY and a limited time frame of 1 or 2 years, then the freely available sources could already help. If it is a matter of indices and intraday data in general, one should rather deal with a provider. There are certainly newer sites, but google will certainly help.

https://www.quantshare.com/sa-636-6-new-ways-to-download-free-intraday-data-for-the-us-stock-market

I use the following data providers and have no complaints so far. The price-performance ratio is ok. These sources also offer indices in larger packages or as separate/specialised packages.

1. https://eodhistoricaldata.com/
2.https://firstratedata.com/bundle/all

Especially eodhistorical data is interesting because it will be integrated into WL here. I think the focus was initially on EOD data, but they now also offer historical intraday data and many other functions provided via tools and API. I think it has a good reputation and good quality data in general.

FirstRate Data is focused on intraday data, but now also offers eod data with a small API to retrieve the data. The data can also be easily downloaded manually.

A new provider extension for EODHistoricalData is coming up. It will include historical, event and streaming providers. Of course, 1-minute data is covered but it's updated by EODHD after the market close.

Besides the natural choice of 1-min data (your broker's data e.g. IBKR), Alphavantage provider comes bundled and DataExtensions contains connectors to some paid services like FinancialModelingPrep, Tiingo and TwelveData: https://www.wealth-lab.com/extension/detail/DataExtensions

Mybe someone can share a SPY intra-day database using Wetransfer.com ?

I am still not able to get a free ASCII file of intra-day SPY in order to use for strategy backtesting and optimization, even after I spent the entire weekend trying to find it out.

The good thing is that I am making my first steps into programming. Here is code to download data from TWS, it is not useful to me right now I don't know if because I don't have specific "Market Data permission for API" or because the code is simply useless. Maybe it's useful for you. Let me know. It uses Iron Pithon and most run on a console (I used Spyder).

CODE:
# -*- coding: utf-8 -*-
"""
Created on Sun Mar 12 20:47:48 2023

@author: diego
"""

from ibapi.contract import Contract
from ibapi.client import EClient
from ibapi.wrapper import EWrapper
from ibapi.common import *
import pandas as pd
import queue
import time

class IBapi(EWrapper, EClient):
    def __init__(self):
        EClient.__init__(self, self)
        self.data = []  # Initialize variable to store candle
        self.queue = queue.Queue() # Initialize queue to store historical data

    def historicalData(self, reqId:int, bar):
        self.queue.put([bar.date, bar.time, bar.open, bar.high, bar.low, bar.close, bar.volume])

    def historicalDataEnd(self, reqId:int, start:str, end:str):
        self.queue.put(None) # Add None to queue to signal that historical data is complete

def run_loop():
    app.run()

app = IBapi()
app.connect("127.0.0.1", 7496, clientId=123)

contract = Contract()
contract.symbol = "SPY"
contract.secType = "STK"
contract.exchange = "SMART"
contract.currency = "USD"
contract.timeZone = "US/Eastern"
contract.primaryExchange = "NYSE"

# Define time range for historical data
end_datetime = "2022-03-10 23:59:59"
duration = "10 D"
barsize = "1 min"

# Request historical data
app.reqHistoricalData(1, contract, end_datetime, duration, barsize, "TRADES", 1, 1, False, [])

# Wait for historical data to be received
start_time = time.time()
while True:
    try:
        data = app.queue.get(timeout=1) # Try to get data from queue every 1 second
        if data is None:
            break # Exit loop if None is received from queue (signaling end of historical data)
        app.data.append(data)
    except queue.Empty:
        if time.time() - start_time > 600:
            print("Timeout: historical data not received after 600 seconds")
            break

# Convert historical data to dataframe
df = pd.DataFrame(app.data, columns=["date", "time", "open", "high", "low", "close", "volume"])

# Combine date and time columns and convert to datetime format
df["datetime"] = pd.to_datetime(df["date"] + " " + df["time"], format="%Y%m%d %H:%M:%S")

# Drop original date and time columns
df = df.drop(columns=["date", "time"])

# Rearrange columns
df = df[["datetime", "low", "high", "open", "close", "volume"]]

# Save to CSV
df.to_csv("SPY_1min_10D.csv", index=False)

app.disconnect()

If you trade enough the data at IB is free, and even if you don't it's only like $10/month. So get the subscription, use WealthLab's IB Provider to download the data, and then you can do something better with your time than recreating an IB data downloader.

QUOTE:
Mybe someone can share a SPY intra-day database using Wetransfer.com ?

Give us some credit, we've invested enough time into the data provider development to do such odd things 😂

fwiw, I'm sure it hasn't always been the case, but at least at the moment IB isn't to be returning highly granular data prior to about August 2021.

I give you lot of credit, boys. You're the dream team!! :D

The thing is actually: in my case, it´s taking more time to download the SPY 1 minute bar data for 4 years than the remaining time for designing the strategies, and I don't know why. I have the subscriptions to Market Data in my accounts, I stream 1 minutes bars and I trade them, but I don't know why loading historical data in this resolution is taking so long. Right now it's been more than two hours and it's not finishing downloading 4 years of 1 minute bar history for SPY. It's not my computer which very above minimum requirements and also it's not my Internet speed.

It's just IB, it takes a very long time to download the data.

The best solution would be IQFeed, here I downloaded 4 years of SPY 1 minute data in about 5 seconds.

Here is the screen-capture of a strategy and the log. I just called to IB support and my my Live Data Market subscriptions are working.

IQFeed is too expansive for what I need.

Still trying to load the data from somewhere. It's taking hours for these 4 years of 1 minute bars of SPY. I checked with IB support and my subscription to Live Market Data is ok. Is IB so bad as a provider of historical data?

Maybe there is a problem with my settings. Do I have too many providers?

Am I the only one who have problems with this?

I created a Tiingo-based dataset with just SPY. Trying to update from the Data Manager and / or the strategy. The same problem: it takes too long.
Alpaca: same same.
I didn't try with QData but I am sure it's going to have the same problem.

There most be something wrong with my settings and or system but after 3 days of being stuck, messing around, I have not a clue

Yes, backfill of 1-min data does take time because providers need to request a lot of history in chunks and face a rate limiting (throttling) policy.

My test with another provider failed too...

It looks like the only reliable way of working with intraday data without paying expensive monthly fees is importing an ASCII file and limiting the backtests and optimization to the data feed. If I understood help files, after importing an ASCII for the first time the data is stored in the cache as a binary file, thus if I limit the backtests to its corresponding dates and if "Obten data fom slected DataSet only" is checked, then loading intra-day data will be faster. Is this correct?

https://firstratedata.com/i/index/SPX
The sample file for 1-min data goes back to April 2022. That's a start and took one second to download.

Apart from the throttling for IB, my experience is that downloading data outside market hours works better. But within market hours, I would try to download data from my paper account. Imho, general data traffic also has an impact.
https://interactivebrokers.github.io/tws-api/historical_limitations.html

You can start the download before taking a long break, like sleeping. It will be done when you arise... and you'll never have to request that data again, updates will be quick.

After 4 days of trying all sort of things to load 1 minute bar data in a reasonable time and without the need of having my computer "on" overnight, I sadly failed, then I decided to buy a data file to load and use as a backtest offline database.

Now I am thinking I failed again and wish I would rather invited my wife to dinner. When I load a data file containing just a couple of days 1 minute bar information, it performs well. Now I am trying to load the entire dataset (SPY, 1 minute bar from 2005 to 2023) and the process bar on the bottom of the screen seems frozen. How long may it take to process this file assuming my hardware complies with the minimum requirements and better?

Please show what screen you are talking of. (Data Manager / Strategy Monitor)
Loading ascii 1-minute data for many hundred symbols and 15 years can take a while until everything is cached.

But just one symbol (SPY) should not take much time independent on the range of data.
What provider did you choose and how does the ascii data looks like?

Edit: load the data in offline mode if possible (ascii data). Before repeating the process close and restart WL.

Just another additional note. Move your Historical Provider to the top of the list.
If you use a linked dataset that is not a problem at all. But using unlinked datasets it will try to get the data in the order the providers are listed.

One hour passed and the process almost did not moved, and I am trying to load only a couple of months!


I canceled the process one more time...

Here is how the file looks like:


Everything seems fine but it doesn't load.

I mean: it loads into the dataset but it keeps loading forever when I want to perform a backtest

Now I am backtesting 3 days, it works but it is taking as much time as when I download online historical data.

Another note. If you have still problems loading the 1-min data you should check your task manager for memory usage while the strategy load the data. If your machine begins to swap memory because your RAM is to small, it will take forever. WL is not designed to handle data which does not fit to memory.

The solution would be to "cut" your file before the caching process manually. E.g. use 10 years or 8, maybe five years of the data.

Thank you very much MIH!

I followed this procedure and went "OFFLINE MODE" and I changed the range to "ALL DATA" and everything loaded in a few seconds. Then I turned WL back online and it looks like it is working fine, also I cloned the strategy and it manages the file very well.

Thank you very much again!!
Ah: the file is from https://firstratedata.com/
I will invite my wife to dinner anyways :D :D :D




SPY behaviour in next image I think is because I have limmited my settings in preferences to 1000 bars because it looks like the strategy was processed (not sure, it's a theory).

Congratulations, enjoy! :-)