When is web scraping legal? Four questions, four separate answers

Is web scraping legal?

No single statute answers "is web scraping legal", so the question splits into four that separate bodies of law answer independently: computer access law, contract law, copyright and database rights, and data protection law where the records describe a person. Each has its own trigger and its own remedy, so one scrape survives the first question and loses the second. Answer all four for your target before you write the client.

This page describes decisions by named courts and regulators, with dates. It is not legal advice.

Last verified: 8 September 2026

What did hiQ Labs v. LinkedIn decide about the CFAA?

The Ninth Circuit affirmed a preliminary injunction barring LinkedIn from blocking hiQ's scrapers, because hiQ had raised serious questions about whether LinkedIn could invoke the CFAA at all. The opinion is hiQ Labs, Inc. v. LinkedIn Corp., No. 17-16783, by Judge Berzon, filed 18 April 2022 on remand from the Supreme Court. The panel reasoned that the CFAA's "without authorization" clause may not reach a site that demands no authorization from visitors and then refuses one bot by letter.

Check the posture before citing it. A preliminary injunction affirmed on a sliding scale asks only whether the plaintiff raised serious questions while the balance of hardships tipped sharply its way, and decides nothing on the merits.

What did Van Buren change about "exceeds authorized access"?

The Supreme Court held that a person "exceeds authorized access" under the CFAA only by taking information from areas of a computer off limits to him, not information he could reach for a forbidden purpose. Van Buren v. United States, No. 19-783 was decided 3 June 2021, six to three, Justice Barrett writing. The Court framed both CFAA clauses as a "gates-up-or-down inquiry" and rejected the government's reading because it would attach criminal penalties to "a breathtaking amount of commonplace computer activity."

One distinction caps how far Van Buren helps a scraper: it construed "exceeds authorized access" in 18 U.S.C. 1030(e)(6), which covers a user already inside with credentials. Scraping a logged-out page raises the other clause, "without authorization," which the hiQ panel said Van Buren did not address directly.

Can a scrape be legal under the CFAA and still breach terms of service?

Yes, and hiQ proves it: the plaintiff that won the CFAA point lost the contract point in the same case. Judge Edward M. Chen of the Northern District of California ruled on summary judgment in case 17-cv-03301-EMC on 27 October 2022, published 4 November 2022. He found that "the relevant language of the User Agreement unambiguously prohibits hiQ's scraping and unauthorized use of the scraped data," and that hiQ breached it by scraping and through contractors who created false identities. He granted LinkedIn judgment on the false identities and held the scraping breach for trial because hiQ's waiver and estoppel defenses raised disputed facts. A consent judgment filed on 8 December 2022 entered $500,000 against hiQ plus a permanent injunction against automated collection without LinkedIn's written permission.

The two claims cost different amounts to bring: a civil CFAA claim under 18 U.S.C. 1030(g) needs at least $5,000 of loss in a one-year period under (c)(4)(A)(i)(I), while breach of contract carries no threshold. EU law separates them too. In Ryanair Ltd v. PR Aviation BV, Case C-30/14, decided 15 January 2015, the Court of Justice held that Directive 96/9/EC does "not preclude the author of such a database from laying down contractual limitations on its use by third parties" where the database attracts neither copyright nor the sui generis right.

Does copyright protect the data you scrape?

Copyright protects original expression and never the underlying facts, so a price, a timestamp or a vote count belongs to nobody. Feist Publications, Inc. v. Rural Telephone Service Co., 499 U.S. 340, decided 27 March 1991, Justice O'Connor writing, rejected the "sweat of the brow" doctrine and confined copyright in a compilation to its original selection, coordination or arrangement. The wording of a post is a separate asset from the fact that it exists, owned by whoever wrote it.

Training a model on scraped material does not convert the copying into fair use. In Thomson Reuters v. Ross Intelligence, No. 1:20-cv-613-SB, Circuit Judge Bibas, sitting by designation in Delaware, revised his earlier ruling on 11 February 2025 and granted Thomson Reuters summary judgment on fair use over Westlaw headnotes copied to build a competing research tool, finding the use not transformative. The EU adds a layer the United States lacks: the sui generis database right protects substantial investment in a database's contents.

What changes when the records describe a person?

Public visibility supplies no lawful basis, and the largest penalties sit here. France's CNIL fined Clearview AI 20 million euro in deliberation SAN-2022-019 of 17 October 2022 for building a face database from images scraped across the open web, finding breaches of Article 6 for want of a legal basis and of Articles 12, 15 and 17 on data subject rights. The regulator held that public accessibility does not stop data being personal, and creates no blanket permission to reuse it.

Article 83(5) puts a lawful basis failure in the top penalty tier, where the ceiling is a formula:

Maximum fine = the higher of EUR 20,000,000 and 4% of total worldwide annual turnover in the preceding financial year

A company turning over EUR 300 million takes 4%, or EUR 12 million, so the fixed figure wins and its ceiling is EUR 20 million. A company turning over EUR 1.2 billion takes 4% and lands at a EUR 48 million ceiling. Work out which fields in your dataset are personal data and which are anonymous records, then document a lawful basis under Article 6 before the first request goes out.

How do you check a target before you scrape it?

Read four things: the terms of service for a clause banning automated collection, the robots file for the paths the operator marked off, the API documentation for a licence covering your use, and a sample record for fields that identify a person. The terms carry the risk hiQ lost on, so an account your company holds at the target matters more than the scraper design. Note that a robots.txt disallow rule is a convention, not a statute, and that taking the documented API moves you to a contract you negotiated.

Budget that API path first, because rate limits decide how many keywords one client id covers. RedReplier watches Reddit, X, Bluesky, Facebook and Hacker News in one place, grades mentions by buying intent, and exposes an API and an MCP server for Claude Code, Cursor and any standard MCP client. The trade against running the platform API yourself sits in RedReplier next to the Reddit API.

Frequently asked questions

Is scraping publicly available data legal?

Public availability answers the computer access question and none of the other three. The hiQ opinion of 18 April 2022 made public visibility the reason the CFAA's "without authorization" clause may not apply; the CNIL's Clearview decision of 17 October 2022 made it irrelevant to lawful basis under the GDPR.

Did hiQ win or lose?

Both, on different claims. hiQ won the CFAA point at the Ninth Circuit on 18 April 2022 and kept its injunction, then lost the contract point before Judge Chen on 27 October 2022. A $500,000 consent judgment against hiQ followed on 8 December 2022.

Do terms of service bind a scraper who never clicked "I agree"?

The hiQ order did not reach it, because hiQ had accepted LinkedIn's User Agreement by signing up for accounts and advertising there. An account or purchase at the target binds the company even when the scraper runs logged out, so audit what your company already holds.

Does the GDPR apply to a company outside the EU?

Article 3(2) applies the GDPR to a controller with no EU establishment that offers goods or services to people in the Union, or monitors their behaviour within it. Collecting posts by identifiable people in the EU falls into the monitoring limb, the footing the CNIL fined Clearview AI on.

Does training an AI model on scraped content change the copyright answer?

Not by itself. Judge Bibas granted Thomson Reuters summary judgment on fair use on 11 February 2025 over Westlaw headnotes copied into a competing product. Facts stay unprotected under Feist, so the exposure sits in the expression you reproduce.

What separates the CFAA question from the contract question?

The CFAA asks whether a gate was up or down, in Van Buren's phrase, and its civil action needs $5,000 of loss in a year. The contract question needs no loss threshold. A scrape can pass the first and fail the second, as hiQ did.

What is the EU's sui generis database right?

It protects a database built with substantial investment in its contents, a right the United States has no equivalent for. Feist confines U.S. copyright to selection, coordination or arrangement, so facts stay unprotected there, while the EU layer covers the investment itself. Ryanair v. PR Aviation shows the two overlap: the CJEU let contractual limits stand precisely because Ryanair's database attracted neither copyright nor the sui generis right.

Does a robots.txt file have legal force?

A robots.txt disallow rule is a convention the site sets, not a statute a scraper can be sued under, so it carries no legal force by itself. Courts weigh the terms of service instead, the way Judge Chen weighed LinkedIn's User Agreement in the hiQ contract ruling. Reading the robots file before you scrape still tells you where the operator drew a line, even one without teeth.

Does scraping through an official API avoid legal risk?

Taking the documented API moves the relationship onto a contract you negotiated, rather than terms you agreed to just by using the site. It does not touch the other three questions: copyright still covers the expression you copy, and the GDPR still applies once a record identifies a person. Budgeting for the API path first also matters in practice, since rate limits decide how many keywords one client id can cover.

How large can a GDPR fine get for scraping personal data?

Article 83(5) puts a lawful basis failure in the top penalty tier, with a ceiling set at the higher of EUR 20 million or 4% of worldwide annual turnover. A company turning over EUR 300 million stays at the EUR 20 million floor, since 4% comes to only EUR 12 million, while a company turning over EUR 1.2 billion faces a EUR 48 million ceiling. France's CNIL used this framework to fine Clearview AI EUR 20 million for scraping face images without a lawful basis.