OwlwakeBot

If you found OwlwakeBot/1.0 (+https://owlwake.com/bot) in your logs, this page is about us.

What it is

OwlwakeBot is how Owlwake's website checks name themselves. Owlwake watches websites for the businesses that own them: whether a site is up, its security certificate, its domain and DNS, its email records, and signs that someone else has changed it. We watch a site on a schedule only when someone who owns or controls it has asked us to, and now and then we look at a site once, when someone asks us about it.

Our checks are passive. They never log in, never scan ports, never guess passwords, and never fill in a form unless someone who owns or controls the site has agreed a test of it with us in writing.

What it asks for, and how often

• A site's homepage, and any pages its owner names, about once or twice a minute, and a few more times when a daily or weekly check reads it too.
• Its robots.txt, its sitemap and a few of the pages the sitemap lists, and its http:// and www addresses, from once a day to once a week.
• Once a week, up to 500 pages of the site, taken from its homepage and its sitemap, and the web links (http:// and https://) on those pages, including links to other sites. On the site itself it reads only the pages its robots.txt allows, and skips links into the paths named by its first 50 Disallow rules, following a User-agent: OwlwakeBot group if there is one and User-agent: * if not; if its robots.txt does not answer, answers with a server error or asks us to slow down, or redirects somewhere we do not follow, it reads the homepage alone. If one of the pages it reads answers with an error, too slowly or not at all, it requests each of those pages once, a fifth of a second apart, following their redirects after checking that each address they lead to is public, and reads them again without the ones that failed, unless the homepage failed or the site asked us to slow down. Links to other sites are checked without reading those sites' robots.txt. This is one way OwlwakeBot can visit a site we do not watch: a site we watch links to it.
• A short, fixed list of addresses that should never be public, so that we can warn a site's owner if one is. We ask for them about once a week, and on a schedule only on sites we have been asked to watch by someone who owns or controls them. Each run first asks for a random address that cannot exist, such as /owlwake-control-3f9a1c2b7d4e, to learn how the site answers a missing page. Where a file is there, we read its first few kilobytes to recognise it, and we never keep what it holds. The list:
  • /.env
  • /.git/HEAD
  • /.git/config
  • /wp-config.php.bak
  • /wp-config.php~
  • /backup.zip
  • /backup.sql
  • /phpinfo.php
  • /.DS_Store
  • /composer.json
  • /wp-content/uploads/
  • /wp-content/plugins/
  • /uploads/
  • /images/
  • /files/
  • /backup/

Any of these can run again sooner: up to three more times, ten seconds apart, to confirm a failure or a finding; after a run that could not finish, within the hour, or within a day for the list of addresses, while the crawl keeps its weekly turn; and more often while a problem it found is still open.

Our weekly speed test loads the homepage the way a mobile browser does, and does not name us. The connections that read a site's security certificate from the site itself, its DNS records and its mail server's greeting are not web requests, so they carry no user agent. We also name ourselves as OwlwakeBot to the public services we look sites up in: domain registries and IANA's list of them, Cert Spotter's certificate search and WordPress.org.

Where it comes from

Every request comes from 138.199.226.86, our server at Hetzner in Falkenstein, Germany. If a firewall blocks the list of addresses above, allowing this address and OwlwakeBot lets the checks through; the site's owner can also ask us to switch those checks off.

Reaching us

If OwlwakeBot is causing your site trouble, or it visits a site of yours that you did not ask us to watch, write to hello@owlwake.com with the site's address, and we will answer. If we are watching a site that nobody who owns or controls it asked us to watch, we stop.

How we handle personal data is in our Privacy Policy, and what we promise the owners who ask us to watch their sites is in our Terms of Service.