How to recognize it

User agents, as they reach our logs:

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
  • First string: the backlink crawler. ~bl marks backlinks.
  • SemrushBot-BA and SemrushBot-SI: Backlink Audit and the On Page SEO Checker. They are separate bots with their own tokens.

Semrush’s pages give tokens, not full strings. Match on SemrushBot, and on SiteAuditBot and SplitSignalBot if you mean the whole family.

  • robots.txt tokens: SemrushBot, plus SiteAuditBot, SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, SplitSignalBot, SemrushBot-OCOB, SemrushBot-FT, SemrushBot-ESI and RyteBot for the other tools.
  • IP list: none for SemrushBot; Semrush says its addresses are not in blocks and not to block by IP. Site Audit uses 85.208.98.128/25 and SemrushBot-SI 85.208.98.0/24, given in Semrush’s help articles.
  • Reverse DNS: none published.

What it does

SemrushBot builds the web graph behind Semrush’s Backlink Analytics, and its data feeds other Semrush tools. The other bots crawl URLs that customers ask about: site audits, backlink audits, on-page checks, writing checks and A/B tests.

What you gain by allowing it

  • Current Semrush data about your site and the links you give. Anyone vetting your site with Semrush sees the real picture.

What you lose

By allowing it

  • Crawl load. Crawl-delay slows it, up to 10 seconds.
  • Competitors can study your pages and outgoing links in Semrush.

By blocking it

  • Semrush’s view of your pages and outgoing links. Your backlinks stay visible, because they are found on other sites.
  • Blocking SemrushBot does not stop the other bots, and a firewall rule matching “Semrush” also breaks Site Audit for you.

How to verify it

Semrush offers no IP list or reverse DNS for the backlink crawler. Cloudflare flags it as a verified bot, in the category “Search Engine Optimization”, except SemrushBot-SI, which Cloudflare files under “AI Crawler”.

Gotchas and history

  • Site Audit and OpenAI-Search: Site Audit can crawl with an OpenAI-Search user agent, to test how OpenAI’s search bot sees a site. Requests that look like OpenAI can therefore be a Semrush audit, which an IP check against OpenAI’s list exposes.
  • Owners can override robots.txt: a verified owner can make SiteAuditBot ignore robots.txt and meta robots on their own site.
  • SemrushBot-SI and Crawl-delay: it accepts only a value of 1; higher values make pages show as “not accessible” in the tool.
  • robots.txt errors: a 4xx on robots.txt means no restrictions for Semrush, while a 5xx stops the whole crawl.

What we observed

Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).

  • 156 requests from Semrush’s crawlers, mostly from Semrush’s own network (AS209366), plus 7 from a single test address. Cloudflare flagged all but one.
  • Firewall rules on both of our sites answered every Semrush request with a 403 at the edge, robots.txt included (14 times). According to Semrush’s own rules, a 4xx on robots.txt means “no restrictions”, so the bot kept trying.
  • 151 used the ~bl backlink string, 9 Backlink Audit and 2 the On Page SEO Checker. Cloudflare filed the 2 SI requests under “AI Crawler”.

About Semrush

Semrush runs about ten bots, each with its own token:

  • SemrushBot crawls for the backlink index;
  • SiteAuditBot runs customers’ site audits, and can crawl under an OpenAI-Search user agent;
  • SemrushBot-BA runs backlink audits;
  • SemrushBot-SI powers the On Page SEO Checker;
  • SemrushBot-SWA checks URLs for the writing assistant;
  • SplitSignalBot runs A/B tests;
  • SemrushBot-OCOB serves the Content Toolkit;
  • SemrushBot-FT powers the plagiarism checker;
  • SemrushBot-ESI serves Enterprise Site Intelligence;
  • RyteBot serves Ryte’s tools.

Blocking one token does not block the others. Semrush publishes ranges only for Site Audit (85.208.98.128/25) and SemrushBot-SI.

Sources