How Arctic Shift Rebuilt Public Access to Reddit Data

What is Arctic Shift?

The Reddit data project Arctic Shift publishes free monthly dumps of Reddit posts and comments, a public API over the same records and a browser based search tool, all maintained by GitHub user ArthurHeitmann. The project README states its purpose as making Reddit data accessible to researchers, moderators and everyone else. Pick the access mode by volume: the web tool for one thread, the API for a few thousand records, the dumps for the whole corpus.

The three modes read the same underlying data, so a query you prototype in the web tool answers the same way through the API.

What data does Arctic Shift cover?

Arctic Shift covers Reddit posts and comments collected through the official Reddit API, with private and quarantined subreddits excluded. The dump index runs from June 2005, near the start of Reddit itself, through releases published in the months just past, so the archive is historical and ongoing at the same time. Check the repo's download links file before planning a study, because that file is the authoritative statement of which months exist.

The file format is one JSON object per post or comment, which makes a dump a newline delimited stream you can process without loading it whole. Since November 2023 the project retrieves each item twice, roughly 36 hours apart, so a record carries the edits, deletions and score changes that landed in that window. Since April 2024 the project distributes files exclusively as .zst, so you need Python 3.10 or newer and the zstandard library to read them.

How does Arctic Shift relate to Pushshift?

Arctic Shift's dumps continue the series that Pushshift published, which ends at March 2023. The project's own file documentation draws the line explicitly: its collection changes apply to later dumps and do not apply to the earlier Pushshift dumps of 2023-03 and before. Researchers who built pipelines on Pushshift file paths therefore point the same pipeline at Arctic Shift for anything after that date.

The two archives sit alongside each other rather than one replacing the other in place. Arctic Shift's README lists Pushshift and PullPush next to itself, each with its own removal request route, which is the practical map of where Reddit archive data lives today. For the platform side of that story, Reddit API pricing covers what changed for anyone reading Reddit at volume, and how RedReplier compares with Pushshift covers the monitoring side.

What are the Arctic Shift API rate limits?

Arctic Shift publishes no numeric rate limit, and its API documentation instead states that a normal user making a couple of requests per second has nothing to worry about. Limiting is dynamic and tied to server load, so the ceiling moves with what everyone else is asking for at that moment. Write your client to back off rather than to a fixed budget.

When you do hit the limit the API answers 429 and returns X-RateLimit-Reset and X-RateLimit-Reset-At headers telling you when the limit clears. Read those two headers and sleep until the reset instead of retrying blind. Fixed quotas work differently on the official platform, and Reddit API rate limits walks through those numbers.

What do people use Arctic Shift for?

Three groups use Arctic Shift: academics building datasets, moderators reconstructing what happened in their own communities, and anyone recovering a post that has since been deleted. The archive answers questions the live Reddit API cannot, because the live API returns what exists now and the archive returns what existed then. That gap is the whole reason the project exists.

Dataset work runs against the dumps, since a study that needs a year of one subreddit is a file operation rather than thousands of requests. Moderation work runs against the API and the web tool, where you pull a user's comment history or a removed thread's tree. Both sit inside the wider practice covered in social media data collection, and the legal frame for reusing that data is the subject of is web scraping legal.

When is Arctic Shift the wrong tool?

Arctic Shift is the wrong tool when you need to know about a conversation while it is still open. The archive is retrospective by design, published on a monthly cadence, with no alerting and no uptime commitment on the hosted API. A dataset released next month cannot tell you that somebody asked for a product like yours this morning.

That job belongs to a monitoring product. RedReplier watches Reddit, X, Bluesky, Facebook and Hacker News together, ranks mentions by buying intent and explains with AI why each one was flagged, then emails you on your plan's interval. See Reddit keyword monitoring for that setup, and how Reddit search works for what the site's own search returns before you reach for an archive at all.

Frequently Asked Questions

Is Arctic Shift free to use?

Yes. The dumps, the API and the web search tool are all free, and the API needs no key. The costs that remain are yours: disk space for the dumps and the compute to decompress and process them.

Does Arctic Shift include deleted posts and comments?

Records are captured twice, roughly 36 hours apart, so a post deleted after that second pass stays in the archive as it was captured. A post deleted within minutes of being made may never have been collected. The project also runs a removal request process for people who want their own content taken out.

What file format do the Arctic Shift dumps use?

Files ship as .zst compressed streams containing one JSON object per post or comment. Reading them takes Python 3.10 or newer with the zstandard library installed. The repo includes scripts for iterating a dump without decompressing the whole file to disk.

How far back does Arctic Shift data go?

The dump index starts in June 2005 and runs through the most recent monthly release. The earlier part of that range comes from the Pushshift dumps of March 2023 and before, which Arctic Shift republishes unmodified.

Does Arctic Shift cover private subreddits?

No. Collection runs through the official Reddit API, which excludes private and quarantined subreddits, so neither appears in the dumps. Anything you find in the archive was public on Reddit at the time it was captured.

Can Arctic Shift replace a live Reddit monitoring tool?

No. Monthly dumps and an archive API answer questions about the past, while monitoring answers questions about the last hour. Use Arctic Shift for research and a monitoring tool for anything you intend to reply to.

Who maintains Arctic Shift?

GitHub user ArthurHeitmann maintains Arctic Shift, publishing the dumps, the API and the web search tool from the same repository. The project README states its purpose as making Reddit data accessible to researchers, moderators and everyone else. One maintainer runs all three access modes on the same underlying data.

Where do I download the Arctic Shift dumps?

The monthly dumps are download links published directly in the GitHub repo, separate from the API and the web tool. Check the repo's download links file before starting a study, since that file is the authoritative record of which months exist. Each linked file arrives as a .zst stream, ready for the scripts the repo includes for iterating it without decompressing the whole thing to disk.

What should my code do when Arctic Shift returns a 429?

A 429 response comes back with X-RateLimit-Reset and X-RateLimit-Reset-At headers stating exactly when the limit clears. Read those headers and sleep until the reset, rather than retrying on a blind schedule or a fixed request budget. Arctic Shift publishes no numeric rate limit in the first place, since the ceiling moves with whatever load the server is under.

Can I use Arctic Shift and Pushshift data in the same pipeline?

The two archives connect directly, since Arctic Shift's dumps continue right where Pushshift's series ends at March 2023. Arctic Shift's own file documentation notes that its collection changes, like capturing each item twice, apply only to dumps after that date and not to the earlier Pushshift files. A pipeline built on Pushshift file paths can point at Arctic Shift for anything from after March 2023 onward.