Your Link Previews Are Not Broken, Your Firewall Is Answering the Crawlers With a 403

Blocking social media crawlers is what broke your link previews, and it happened the moment you turned bot protection on. Those preview cards are built by fetchers that read your page and pull the Open Graph tags out of it, and every one of those fetchers is a bot. Your new rule cannot tell them apart from the scrapers you meant to stop, so it hands them a 403 and logs a win. Nothing errors. Nothing warns you. Your shared links go out as bare URLs with no image, and the referral traffic that used to follow them quietly stops arriving.

The settings that usually did it

Five changes account for almost all of these cases, and it is common to have made more than one of them in the same afternoon.

  • An AI crawler block. Cloudflare announced on 1 July 2025 that it was changing the default to block AI crawlers, and plenty of site owners switched the same thing on manually elsewhere. The managed lists behind these controls get updated on the vendor’s schedule rather than yours, so a rule you set once can start catching new user agents months later.
  • A managed WAF ruleset or bot fighting mode. These score every request for how bot-like it looks. A well behaved preview fetcher scores badly, because it is a script with no mouse movement and no browsing history behind it.
  • A JavaScript challenge or an under attack setting. Crawlers do not execute JavaScript and cannot solve a challenge, so they receive the interstitial page and read that instead of your content. Some platforms then cache the interstitial as your preview.
  • Hotlink protection. This one produces the half broken preview. The crawler reads your HTML fine, finds your image URL, requests it from a referrer the rule does not recognise, and gets refused. You end up with a text only card.
  • A robots.txt copied from somewhere else. AI blocklists get shared around and pasted in wholesale. They frequently carry entries that go well beyond AI training crawlers.

Why nothing in your dashboard told you

From the firewall’s point of view this is not a failure. It is the feature working. Blocked requests are counted as threats mitigated, which is exactly the number the product is designed to make look impressive.

Your analytics will not flag it either. Analytics counts people who arrived, so a drop in referrals looks like a quiet fortnight rather than a fault. Search Console only covers Google, so it stays green while every other crawler is being turned away. The one place the evidence actually lives is your firewall’s event log, and almost nobody opens it unless something is on fire.

Which fetchers have to stay allowed

Cloudflare separates verified bots into categories, and two of them matter here. Page preview covers the fetchers that generate the cards on social platforms. AI crawler covers the ones collecting content for model training. Blocking the second category while allowing the first is the whole trick, and a blanket rule does not make that distinction for you.

Platform What it fetches What breaks when it is blocked
Pinterest Pinterestbot reads the landing page to index content and keep Pin metadata current Rich Pin data stops syncing, so Pins lose the title and site information that carries people to your page. Pinterest also states that large crawl delays affect the distribution and recommendation of your content
Facebook and Instagram facebookexternalhit reads your Open Graph tags Shared links lose the image, the headline, the description and the site attribution on both platforms
VKontakte The VK share scraper reads the same Open Graph tags and caches the result Posts render as a plain URL, and the failed read stays cached until it is cleared
LinkedIn, X, Slack, Discord, WhatsApp, Telegram Each runs its own unfurling fetcher against your page Preview cards collapse to raw links, which people click far less often

Two details from Meta’s own crawler documentation are worth knowing before you start debugging. Your Open Graph properties have to appear inside the first megabyte of the page or they get cut off. And Meta caches robots.txt for up to twenty-four hours, so a fix there will not show up instantly.

Pinterest has a separate trap that has nothing to do with your firewall. Pinterest does not support robots.txt files on subdomains, and only reads the one on your main domain. If your shop or your blog lives on a subdomain with its own robots.txt, Pinterest never sees those rules at all.

How to tell a real crawler from something wearing its name

The obvious fix is to write an allow rule matching the user agent string and get on with your day. You were expecting me to recommend that. Please do not. A user agent is a text field that any script can type anything into, and an allow rule that skips your security checks for anyone claiming to be Pinterestbot is a door with the word Pinterestbot written on it and no lock.

The proper method is forward-confirmed reverse DNS, and it takes about thirty seconds per address. Take the IP that made the request. Run a reverse DNS lookup on it to get a hostname. Check that the hostname belongs to the company it claims to. Then run a forward lookup on that hostname and confirm it resolves back to the same IP. An impostor controls the user agent and fails the second half of that check every time.

Pinterest documents this directly and tells you to use the host command for the reverse lookup, then confirm the domain in the response ends with pinterest.com or pinterestcrawler.com. Meta publishes IP ranges instead and asks you to allow either the user agents or the addresses. Where a platform offers a published range, use the range. Where it does not, use the reverse DNS check rather than trusting the string.

Getting your previews back in about twenty minutes

  1. Request one of your own pages from a terminal while setting a crawler user agent, and look at the status code you get back. A 403 or a challenge page is your answer. Repeat for two or three of the fetchers in the table above.
  2. Open the firewall event log and filter the last thirty days by user agent. You will usually find the blocked fetchers stacked up in a neat pile, with a timestamp that matches the day you changed the setting.
  3. Add skip rules by verified bot category rather than by user agent string, and scope them to your public content paths only.
  4. Force each platform to read the page again. Meta’s Sharing Debugger triggers a fresh crawl, LinkedIn has a post inspector, and Pinterest will resync Rich Pin data on its own within about twenty-four hours once it can read the page.
  5. Reduce any crawl delay in robots.txt. Pinterest honours delays of up to one and treats anything larger as one, so a long delay buys you nothing and costs you distribution.

What you should keep blocking

This is the part where I stop being entertaining, because swinging the other way will cost you more than broken previews ever did. Turning bot protection off entirely to fix a preview card leaves your login page open to credential stuffing and your checkout open to card testing, with comment spam and content scraping filling in whatever gaps are left. Those are account problems and money problems, and they do not announce themselves either.

Scope the protection instead of removing it. Public article and product pages need to be readable by crawlers, so run those with skip rules for verified bots and a sensible rate limit. Login paths, admin paths, search endpoints and anything that writes to your database need every challenge you can point at them, and no crawler has any business there. If you run WordPress, that means the login and admin paths stay locked while your posts stay open. A narrow, boring rule beats a sweeping one every time here.

Give it a week, then check your referral numbers again. Links that were shared while the crawlers were locked out will not repair themselves, because the platform cached a failed fetch, so push the important ones back through each debugger or reshare them. If referrals are still flat once the fetchers are provably getting a 200 back, the firewall was never your problem and you are looking at an ordinary website traffic shortfall instead. Those are two different jobs and it is worth knowing which one you actually have.

Leave a Reply

Your email address will not be published. Required fields are marked *