About Our Crawler

What it does

ClientPadBot only ever fetches public, unauthenticated pages — it never logs in, and it never sees anything a normal visitor to your site couldn't also see. It's used for two things, both triggered by a ClientPad customer who already has your business on file in their own CRM:

• Finding a general contact or team page — looking for a public mailto:link or a team/about page listing names and titles, the same way a person would if they visited your site looking for who to talk to.
• Reading public business signals — a careers page (to see if you're hiring), your homepage's own technology footprint, a press/about page for funding or leadership announcements, and your pricing page (to notice if it's changed since a prior visit). None of this is presented to our customers as confirmed fact — it's always labeled as a signal worth a human look, not a verified claim about your business.

What it respects

• robots.txt — checked before every single request, including Disallow,Allow, and Crawl-delay directives targeted at this User-Agent or at*.
• Rate limiting — at least a 2-second gap between requests to the same domain (longer if your own Crawl-delay asks for it), and a small global concurrency cap across every site ClientPad crawls at once, regardless of how many customers are running searches at the same time.
• No authentication, ever — this crawler has no login credentials for any site and never attempts to access anything behind a login wall.

How to block it

Add the following to your site's robots.txt and we'll stop crawling it entirely:

User-agent: ClientPadBot
Disallow: /

We check this before every request, so it takes effect immediately — no need to contact us first, though you're welcome to either way.

Questions

If you have a question about a specific request, or anything else about this crawler, reach out through our contact form.