SofaarBot

What is SofaarBot?

SofaarBot is an automated crawler operated by Sofaar. It downloads publicly available news pages so that they can be indexed and analyzed for Sofaar's media-monitoring service.

About Sofaar

Sofaar is a media-intelligence platform. It follows news coverage across many publishers and languages, so that organizations can find the articles that mention them and understand how they are covered, much as they would with a search engine.

What it means for your site

• Each article is shown to Sofaar's users with its headline, a short excerpt and a link to your page, so readers are sent to your site.
• Sofaar never republishes the full text of your articles publicly.

How SofaarBot behaves

• It identifies itself with this User-Agent: Mozilla/5.0 (compatible; SofaarBot/1.0; +https://sofaar.net/bot)
• It reads robots.txt before crawling and follows it on every request: feeds, sitemaps, section pages and articles.
• It prefers the RSS feeds and news sitemaps your site publishes, then fetches the articles listed in them.
• It makes one request at a time to your site, at least one second apart. It slows down on 429 and 503 responses and honors Retry-After.
• It only visits public pages. It does not log in, and does not access paywalled or CAPTCHA-protected content.
• It does not try to get past bot challenges, does not pretend to be a web browser, and does not switch identities or IP addresses.
• For pages that need JavaScript, it uses an ordinary headless browser with the same User-Agent and the same rules. It does not download images, video or fonts.

How to block or limit SofaarBot

To block SofaarBot from your whole site, add this to your robots.txt:

User-agent: SofaarBot
Disallow: /

To keep it out of part of your site only:

User-agent: SofaarBot
Disallow: /private-section/

SofaarBot reads your robots.txt again at least once every 24 hours, so changes take effect within a day. To ask for a lower crawl rate instead of a block, email us.

How much it crawls

• It visits a site only to pick up new articles: feeds and sitemaps a few times an hour, then the new articles they list. For a busy news site this is usually tens to a few hundred requests a day; for most sites far fewer.
• It never fetches the same article twice on purpose, and it does not crawl archives, search pages, tag pages or user profiles.

Verifying SofaarBot

Every SofaarBot request, including those made by its headless browser, comes from one fixed address that is used by nothing else:

• IP address: 65.98.29.112
• Reverse DNS: crawler.sofaar.net — forward-confirmed, the name resolves back to the same address
• Machine-readable list: https://sofaar.net/bot/ips.json, updated before any address is added or changed
• Signed requests (Web Bot Auth): every request carries an RFC 9421 HTTP Message Signature with Signature-Agent: "https://sofaar.net"; the Ed25519 public key is published at https://sofaar.net/.well-known/http-message-signatures-directory

A request that carries the SofaarBot name but comes from another address is not from us. If you prefer, email us the IP address and time of a request and we will check it for you.

Who operates it

SofaarBot is operated by Sofaar (sofaar.net), a media-intelligence platform that monitors news coverage for organizations in Saudi Arabia and the wider region. The crawler runs from Sofaar's own server in London; it is not shared with any other service.

Contact

To report a problem, ask for a lower crawl rate, or ask any question about SofaarBot, email bot@sofaar.net.