A European initiative to structure the web

SitesGraphBot

You probably found this page in your server logs. Here's what our crawler does, and how to control it.

What it does

SitesGraphBot reads the public pages of business websites (the homepage and up to 10 pages such as contact, about or opening hours) to build a structured directory of European companies: what a company does, where it is and how to reach it.

How it behaves

• It respects robots.txt, including Crawl-delay.
• It makes one request at a time to your site, with a pause in between.
• It only reads public pages: no logins, no forms, no paywalls.
• It never gets around bot protection: if your firewall or Cloudflare shows it a challenge, it stops and only keeps what's public elsewhere (DNS, domain registry, Wikidata).
• It comes back about once a month to keep the data up to date.

Block or limit it

Add this to your robots.txt to keep it away completely:

User-agent: SitesGraphBot
Disallow: /

Or slow it down:

User-agent: SitesGraphBot
Crawl-delay: 10

Let it in

Does your site use bot protection (Cloudflare, DataDome, a firewall) and do you want to be listed with your full details? Allow the user agent SitesGraphBot in your protection's settings. In Cloudflare: Security → WAF → Custom rules, a rule "User Agent contains SitesGraphBot" with the action Skip.

Questions or removal

Want your company's record removed or corrected, or think the bot misbehaves? Contact the operator of the server it came from. See Data & GDPR for how we handle data.