YallaCrawlerBot

What it is

YallaCrawlerBot is Yalla's web crawler. It checks which business software, such as CRM and email-marketing tools, public business websites use, so that we can understand the market for our product.

It reads public pages only. Most of the time it doesn't visit your site at all: it first reads copies of public pages from public web archives such as Common Crawl, and only visits a live site to confirm what it found there.

How to recognise it

Every request carries this User-Agent:

Mozilla/5.0 (compatible; YallaCrawlerBot/1.0; +https://yallacrm.ai/techscan-bot; techscan@yallacrm.ai)

How it behaves

• It obeys robots.txt. It reads the robots.txt of each site (each scheme and host) and refreshes it every day. If your robots.txt returns a server error or can't be reached, it doesn't fetch your pages on that visit.
• It sends one request at a time to your site, with at least 2 seconds between requests.
• It honours Crawl-delay up to 60 seconds. If your Crawl-delay is longer than that, it doesn't crawl your site.
• It fetches a small number of pages per visit, at most 8: usually your home page, up to two contact pages, and your privacy and terms pages.
• If your site answers with 429 or 503, it waits at least 10 minutes (or longer if you send Retry-After). After three errors in a row it waits a day. If your site answers 403, it stops.
• It never logs in, never submits forms and never keeps cookies.
• It only reads web pages (HTML), with size and time limits on every request.

How to block it

Add this to your robots.txt:

User-agent: YallaCrawlerBot
Disallow: /

We pick up the change within a day. If you later remove the rule, we may visit your site again.

How to have data about your site removed

Blocking stops future visits. To also delete what we hold about your site, email us with the website address:

We stop crawling your site and delete what we hold about it within 7 days, and reply to confirm.

What we store

• Which business tools we found on your site, with a short excerpt of the evidence and when we saw it.
• A copy of each page we fetched, with email addresses, phone numbers, email and phone links, and URL query strings removed. We delete these copies after 180 days.
• A record of each fetch (the address without its query string, the date, and a small set of technical response headers), kept for up to 2 years.
• Public DNS answers for your domain and, for UK sites, public facts from Companies House.

What we don't store

• Email addresses or phone numbers
• Details about people
• Images
• Anything behind a login

A stored page copy may still contain text that appears publicly on your page, such as a name; those copies are deleted after 180 days.

Who operates it

YallaCrawlerBot is operated by Yalla Technologies LLC, which is responsible for the data it collects. We collect it on the basis of our legitimate interest in researching the market for business software.