Cloudflare can block SEMrush crawlers from accessing your site by setting a Cloudflare rule. The problem occurs when the Cloudflare rules are set to “Block All” bots and not specific IPs. This is an issue for legitimate crawlers like SEMrush, but it’s also an issue for those that scrape content without permission or scraping malware. How do you fix this?
Here is a quick tutorial on how to fix Cloudflare blocking SEMrush crawler. This usually happens when Cloudflare detects an attack from the IP address of your server or if Cloudflare thinks you are using too much bandwidth. Today we will show you two steps that will help SEMrush crawl your website again and get started with SEO keyword tracking!
SEMrush Audit Error
If you run SEMrush Audits, you might have experienced SEMrush being blocked from auditing a website, you will see an error code like this:
Sorry, auditing domain.com has failed
We encountered an error that stopped us from crawling your website:
Network connection was interrupted.
We couldn’t crawl your homepage due to connectivity issues on your site.
How to fix it
This issue is typically caused by your hosting provider blocking our crawler. Please download a log file and contact your website administrator to fix this error. See how to whitelist SEMrushBot on different hosting providers.
How to fix SEMrush bot from being blocked
- Check your error code by downloading the SEMrush log (shown above).
- The second step is to identify that Cloudflare is blocking the scan since numerous things are blocking your crawl.
- After identifying it was a SEMrush blockage, you will need to Whitelist the SEMrush’s IP range in your Hosting:
Find out how to whitelist your IP range in CloudFlare; see below.
How to Whitelist SEMrushBot in Cloudflare:
Log in to your Cloudflare account.
Select your domain.
Click the Firewall >> Tools sub-menu settings.
Under IP Access Rules, enter the following details:
- Enter the SEMrush Values. (184.108.40.206, 220.127.116.11, 18.104.22.168)
- Select an Action >> Allow.
- Select whether the rule applies to This website or All websites in the account.
- Add a note (i.e., SEMrush).
- Click Add.
- Purge your cache and wait at least an hour before trying to audit your website again.
This can be done by clicking on “Purge Everything” under the Caching tab.
After the cache has cleared for SEMrush, your audits should have no issues running.
I hope this was helpful in understanding how Cloudflare can block SEMrush crawlers from accessing your site. If you have any questions about this, please email me.