---
title: "Crawler"
description: "What the whichcoding.tools agent reads on vendor sites, how often, how it identifies itself and how to reach a person about it."
canonical_url: "https://whichcoding.tools/crawler"
---
# Crawler

> What the whichcoding.tools agent reads on vendor sites, how often, how it identifies itself and how to reach a person about it.

## What it reads

Public pages only: the pricing page each tool file cites as a source, the homepage, the docs, the changelog, and the ACP agent registry. It requests the pages the data already cites and does not follow links beyond them. It never signs in and never reads a page behind a login.

The text it reads is data, never instructions. Anything on a page addressed to agents is ignored and noted in the run report.

## How often

The pricing sweep runs once a day at 06:15 UTC and fetches the pricing pages each tool cites, usually one. The rename watch (Monday), discovery (Tuesday), ACP watch (Wednesday) and stale sweep (Thursday) each run once a week. Any one page sees about one request a day, with a 20 second timeout and no retries on a page that refuses.

## How it identifies itself

Every request carries the user agent `whichcodingtools-agent/1.0 (+https://whichcoding.tools/crawler)`. Before fetching a page it reads the origin's robots.txt and skips any page a Disallow for `whichcodingtools-agent` or `*` covers. The weekly rename watch is the exception: it sends one HEAD request to each homepage to see where it redirects, and reads no robots.txt for it. A reserved page is reported in the run summary and nothing else happens to it. To keep it out, add that Disallow and the next run honours it.

A page that only renders in a browser is read in one the sandbox runs, which sends a stock Chromium user agent. The robots.txt check happens before that fallback too.

## What happens to the text

An excerpt of the page is kept in the public repository under content/snapshots/<slug>/ so a price on this site can be checked against the page it came from. That excerpt stays under the vendor's own terms and is not part of the data license.

A change to a price goes out as a pull request that shows the before and after next to the page it came from. A change to one tool that has an excerpt on file is merged once the checks pass, which include finding every price in that excerpt. Anything wider waits for a person, except a batch that only adds excerpts and refreshes dates.

## Contact

For anything about the crawler, a page to exclude, an excerpt to remove, or a request that looks wrong, open an issue on GitHub. It reaches Benjamin Canac, who maintains the site, not an abuse form. [Open an issue](https://github.com/benjamincanac/whichcodingtools/issues/new).

## Sitemap

See the full [sitemap](https://whichcoding.tools/sitemap.md) for all pages.