A European initiative to structure the web

Docs

What you can do with SitesGraph, the API, the npm package, and how we handle data.

Why use SitesGraph

Find companies by what they are and where they are

"All hotels in Barcelona", "restaurants within 10 km of Antwerp", "Shopify shops in the Netherlands", "dentists in Ghent with an online booking tool". Every crawled website gets a category (Hospitality › Accommodation › Hotel), a location with coordinates, and a verdict (is it a real business?). Search them on Explore or with /v1/search, and export to CSV.

Turn a website into structured data

One call gives you a company's name, legal name, VAT number (checked), address, opening hours, emails, phone numbers, social profiles, languages and more, as JSON. Use it to fill a CRM, enrich sign-ups, build a directory, or give an AI agent clean facts instead of raw HTML.

Prepare social media posts from a website

Every record lists the site's images with their alt text, its logo, brand colours, page titles, descriptions and keywords per page. That's the raw material for posts and visuals about a company (a restaurant's dishes, a hotel's rooms, a shop's products) without asking the owner for files. Check the image rights before you publish.

Audit a website and help its owner

About 55 checks across SEO, accessibility (useful with the European Accessibility Act in force since 2025), marketing, trust and security, a score, and concrete tips in plain language. Share the report with the owner.

Market research

Which booking tools do restaurants in Belgium use? How many webshops accept Bancontact? Which CMS do Dutch municipalities run? Technologies, payment methods, delivery options and review widgets are detected on every crawled page.

Monitor and keep data fresh

Records are crawled again periodically: see which businesses closed (parked or unreachable), changed address, or launched a webshop.

API

All endpoints return JSON. No key is needed to read. Admin endpoints (the crawler and the dataset downloads) need Authorization: Bearer <ADMIN_TOKEN>; in a browser, the crawler page asks for it as a password.

Search filters

curl "http://localhost:3000/v1/search?category=hotel&city=Barcelona"
curl "http://localhost:3000/v1/search?category=restaurant&near=Antwerpen&km=10&format=csv"

npm package: look up one domain

The crawler and all the analysis are a standalone package. No database or server needed.

npm install sitesgraph

const { lookup } = require("sitesgraph");
const record = await lookup("restaurantbink.be");
console.log(record.site_status.verdict);   // "business"
console.log(record.category.category.path); // Hospitality › Food & drink › Restaurant
console.log(record.location.city);         // "Turnhout"

npx sitesgraph restaurantbink.be --summary

The technology rules (Wappalyzer, GPL-3.0) are downloaded once on first use, into ~/.cache/sitesgraph.

Self-hosting

Run SitesGraph on your own server (AWS or any Linux machine) with install.sh or Docker. Crawl a country, export the data (gzip NDJSON or Parquet) and merge exports from several servers: the newest record wins. See docs/deploy.md in the repository.

The dataset

The SitesGraph dataset is managed by Autonomous BV, as the base of a Global Trends system: following companies over time to learn what works and what doesn't. Which sectors grow, which technologies and business models companies adopt or drop, which companies stay active and which disappear.

We share it for research and with partners who want to contribute (crawling capacity, data, corrections, expertise). Get in touch with Autonomous BV.

Download and query it

On your own server, download every summary (one row per domain: name, status, category path, city, province, region, country, continent, coordinates, contact details, score, technologies, keywords…). These downloads are admin-only:

curl -H "Authorization: Bearer $ADMIN_TOKEN" -o sitesgraph.csv     https://your-server/v1/export/summaries.csv
curl -H "Authorization: Bearer $ADMIN_TOKEN" -o sitesgraph.parquet https://your-server/v1/export/summaries.parquet

Or from the command line on the server (works while it runs): npm run cli -- csv (a .csv.gz), npm run cli -- parquet (summaries + full records), npm run cli -- export (full records as NDJSON). Then query it with DuckDB:

duckdb -c "SELECT province, count(*) AS bakeries
           FROM 'sitesgraph.csv'
           WHERE category_path LIKE '%Bakery%' AND country = 'BE'
           GROUP BY 1 ORDER BY 2 DESC"

In the CSV, lists (category path, technologies, keywords) are joined with "; ". The Parquet file keeps them as real lists.

Data limitations

SitesGraph reads public websites, without AI and without paid sources. That works for most business websites, but not for all of them, and not for every field. Know these limits before you rely on the data. The full list of edge cases is in docs/limitations.md.

"Found" means we have a value, not that it's right. A set of verified sites is checked on every change.

Sites we can't read

• Bot protection (Cloudflare, DataDome, Akamai…): about 7% of sites, often big brands, answer with a challenge or refuse the visit. Their verdict is Blocks crawlers. We don't get around bot protection: no pretending to be a browser, no solving challenges. We still know their DNS, domain age, hosting and e-mail provider. For larger organisations, Wikidata fills in the name, country, city, type and sometimes the VAT number.
• robots.txt: a site that asks bots not to visit isn't read at all. Pages it disallows are skipped.
• JavaScript-only sites: their content is built in the browser, and we only read the HTML, so they look nearly empty (Website (JavaScript only)).
• Not read: text in images and PDFs (menus, opening hours), the inside of iframes (booking widgets), pages behind a login or a cookie wall, content that loads on click or scroll.
• At most 10 pages per site, 5 MB and 30 seconds per page. Big sites are mostly unseen.
• One point of view: sites that redirect by country show us the version for our server's country.

Where we guess

• Business or not: worked out from signals (address, VAT number, opening hours, shop…), not from a company register. A hobby site can look like a business, and a one-page business site can look like a plain website.
• Category: keywords in 6 languages, schema.org types, page types and technologies. One category per site, from a tree of about 60 (not NACE). Some sites end up in the wrong category; we keep those as test cases.
• Location: the city list covers places with 15,000+ inhabitants, so villages get only a country or the nearest bigger place. Coordinates are often the city centre, so radius searches are approximate. For chains, the main place may be the head office. A .com says nothing about the country.
• Contact details: emails hidden from bots (images, "at" spelled out, contact forms only) are missed; a number can be a fax.
• VAT numbers: checksum-checked for 10 countries, but not checked against VIES, and the number in a footer can belong to the web agency or the parent company.
• Technologies: rule-based. Unknown tools aren't detected, generic rules can misfire, and DNS-based finds (Microsoft 365, Google Workspace) mean the company has an account, not that the site uses it.
• Domain age: some registries hide it, and it can be the last registration date rather than the first. Behind a CDN we see the CDN, not the real host.
• Freshness: an automatic copyright year or sitemap date makes an old site look fresh.
• Score and tips: automated checks, not an audit, and not a European Accessibility Act compliance statement.

Coverage

• The crawler finds sites through links, so sites nobody links to are only found when someone looks them up. European domains go first.
• Social networks, marketplaces and website-builder subdomains are skipped: a business with only a Facebook page isn't in SitesGraph.
• One record per domain, not per company. Subdomains belong to their main domain.
• Records are crawled again after 30 days by default; each one shows its crawl date. Search updates within 10 minutes.

Data & GDPR

• Public business information only. SitesGraph reads what a website publishes for everyone: company name, address, opening hours, general contact details, technology. It doesn't log in, fill in forms, or bypass paywalls.
• robots.txt is respected. A site that asks crawlers not to visit isn't crawled. We crawl one page at a time per site, with a pause between requests, and identify ourselves as SitesGraphBot.
• Personal data. Business websites sometimes list people (a team page, an email like [email protected]). That's personal data under the GDPR. We store it only as it's published, for the legitimate interest of describing the business, and we don't build profiles of people.
• Your responsibility as a user. If you use the data for direct marketing, the GDPR and e-privacy rules apply to you: check consent rules for emails in your country, honour opt-outs, and don't spam.
• Removal. A site owner can ask for their record to be removed, or block us in robots.txt (User-agent: SitesGraphBot / Disallow: /).
• Where the data lives. On the servers you run SitesGraph on. Nothing is sent to third parties, except lookups to the official domain registries (RDAP/WHOIS), the websites themselves, and Wikidata for sites that block crawlers (only the domain name is sent).