Ethics and robots.txt

Public data only, and we ask first.

The content on a website belongs to its owner. SitesGraph reads what a site publishes for everyone, and stops where the owner says stop.

How SitesGraphBot behaves

• robots.txt first. It is read before any other page, including Crawl-delay. A site that asks bots not to visit is not read; pages it disallows are skipped.
• One request at a time to your site, with a pause between requests.
• Public pages only. No logins, no forms, no paywalls. At most 10 pages per site.
• No bypassing bot protection. If a firewall or Cloudflare shows a challenge, it stops.
• It says who it is: the user agent is SitesGraphBot, and /bot explains it.
• It comes back about once a month to keep records fresh.

Opt out

Add this to your robots.txt:

User-agent: SitesGraphBot
Disallow: /

Or slow it down with Crawl-delay: 10. Want to be listed with full details despite bot protection? Allow the user agent SitesGraphBot in your firewall. More on the bot page.

Removal and corrections

A site owner can ask for their record to be removed or corrected. Write to [email protected] with the domain name. Blocking us in robots.txt stops new crawls; ask for removal to delete what we stored.

Personal data

Business websites sometimes list people (a team page, a personal email address). We store that only as published, for the legitimate interest of describing the business, and we do not build profiles of people. If you use the data for direct marketing, GDPR and e-privacy rules apply to you: check consent rules for your country, honour opt-outs, and do not spam. See privacy.

Where requests go

Besides the websites themselves, a lookup contacts the official domain and IP registries (RDAP/WHOIS). For sites that block crawlers, only the domain name is sent to Wikidata. Nothing else is sent to third parties.