Web Data Extraction: What You Can Lawfully Collect, and How to Document It

The short answer: You can generally collect publicly available facts, such as product prices, business listings and published filings, when you respect site terms, technical access controls and copyright, avoid personal data without a lawful basis and do not overload the source. The legal picture differs by country and is still developing, so every program needs a documented source assessment and legal review. Teams that run data mining and processing at scale build that assessment into the scoping of every project.

Publicly visible is not the same as unrestricted

A common assumption is that anything visible on the public web can be collected. In practice, several layers of rules apply at once: the site’s terms of use, copyright and database rights in the content, privacy law where personal data is involved, computer misuse and anti-hacking laws where access controls are bypassed, and contract law where an account or API agreement was accepted.

Courts in the US have reached different conclusions in different cases about scraping public pages, and the EU and UK add database rights and strict data protection rules. None of this means web data cannot be used; it means each program needs a deliberate assessment rather than a default yes. This article is general information, not legal advice, and counsel should review any significant program.

Facts are generally safer than creative content

Facts, such as prices, product specifications, opening hours, company addresses and published regulatory filings, are generally not protected by copyright, although a database of them may be protected in some jurisdictions. Creative content, such as articles, reviews, photos and product descriptions, usually is protected, and copying it wholesale for republication or commercial use carries risk.

Many lawful programs therefore extract specific facts into a structured dataset rather than copying pages. Where content itself is needed, for example for research or AI training, licensing or using sources with permissive terms reduces risk. Keep the extraction narrow to the fields the project actually needs.

Personal data needs a lawful basis

Collecting information about identifiable people, such as names, emails, phone numbers, social profiles or reviews linked to a person, brings privacy law into play. Under GDPR and UK GDPR, you need a lawful basis, must inform people in many cases and must respect their rights. US state privacy laws add their own obligations, and several regulators have acted against companies that scraped personal data at scale.

The safest approach is to avoid personal data unless the project truly needs it, to minimize what is collected when it does, and to document the basis and retention period. See data cleansing and enrichment for the rules that apply when web data is appended to customer records, and CRM data decay for why appended data needs ongoing upkeep.

Respect access controls and technical limits

Bypassing logins, paywalls, CAPTCHAs or IP blocks to reach data is where legal risk rises sharply, because it can engage computer misuse laws and breach contract terms. Logged-in data accessed under an account agreement is governed by that agreement. Robots.txt files are not law, but they express the site owner’s wishes and are increasingly treated as relevant evidence of permission.

Technical courtesy matters too. Rate-limit requests, identify your crawler where appropriate, avoid peak hours for small sites and stop if a site asks you to. Where an official API exists, it is usually the better route.

Document every source

A source register is the single most useful control. For each source, record the URL or provider, what data is collected, the purpose, the legal basis, relevant terms of use, whether personal data is involved, the collection method and frequency and the retention period. Record the date of review and who approved it.

The register protects the business when a site owner, regulator or customer asks how data was obtained. It also makes it easy to remove a source if terms change. See information services and data operations for how data product companies keep sources and update cycles documented.

Quality: extracted data is only as good as its checks

Web sources change layouts, contain errors and go out of date. Extraction programs need field-level validation, regular accuracy sampling against sources, change detection and alerts when a source breaks. Without these checks, datasets degrade quietly and decisions built on them go wrong.

Report accuracy and completeness by field and source with every delivery, so users know where to trust the data and where to be careful. See research outsourcing for analysis programs that depend on extracted data.

What to put in a data collection contract

When an external team collects data for you, the contract should state which sources are approved, what may and may not be collected, how personal data is handled, who owns the resulting dataset and who is responsible if a source owner complains. It should require the source register to be maintained and shared, and give you the right to audit methods and stop collection from any source at short notice.

Include quality terms as well: accuracy and completeness targets by field, delivery formats and schedules, and how errors are corrected. Clear terms protect both sides and make it simple to change sources when rules or business needs change.

A pre-launch checklist for any extraction program

Before collection starts, confirm the purpose and the fields needed, review each source’s terms and access rules, decide whether personal data is involved and document the lawful basis if it is, check whether an API or licensed feed is available, set rate limits and crawler identification, define validation and accuracy sampling, and record retention and deletion rules.

Get sign-off from legal and the data owner, then pilot on a small sample and review the output before scaling. Repeat the review whenever a source changes its terms or the project’s purpose changes.

Web data for AI training

Collecting web data to train AI models has become the most contested use, with litigation over copyright, new opt-out mechanisms and disclosure obligations emerging in several jurisdictions. Developers increasingly favor licensed datasets, permissively licensed content and synthetic data, with documentation of provenance for every training source.

Our data mining and processing outsourcing page explains how vetted Philippine data teams scope sources, extract and validate data and maintain source registers, and where to put the human checkpoint in AI agent workflows covers the governance side of AI programs.