A growing number of Singapore businesses have, without ever meaning to, told the AI tools that could recommend them to stay away. It usually happens through a single line in a robots.txt file nobody on the team has opened in years, or a one-click security setting that blocks “bad bots” by default. Either way, the outcome is the same: a brand that has invested in brand citation and entity authority becomes invisible to the one audience it was trying to reach.
AI crawlers are not optional infrastructure. They are how ChatGPT, Gemini, Perplexity and Claude discover that a business exists at all, long before any model can cite it, recommend it or summarise what it offers. A site that blocks the wrong crawler is not being cautious about data privacy. It has opted out of AI search entirely, often without anyone noticing until a competitor appears in an answer and the business does not.
What AI Crawlers Actually Do
Every major AI platform runs automated crawlers to build and refresh its view of the web, much like Googlebot has done for traditional search for two decades. Some of these crawlers gather training data for future model versions. Others fetch a specific page in real time because someone just asked a question and the model needs current information to answer it well.
A business page that answers a shopper’s question about delivery times in Tampines, or a clinic’s page explaining a procedure, only gets used in an AI-generated answer if the right crawler could reach it in the first place. Brand citation work, content structure and entity authority all assume that basic access exists. When it does not, none of the rest of an AI search strategy gets a chance to work.
The Crawlers Worth Knowing By Name
Most site owners only ever encounter these names inside a robots.txt file or a security log, which is part of why they get blocked without anyone noticing. The differences between training and retrieval are explained directly in Google’s crawler documentation, and the crawlers that matter most for AI visibility in Singapore right now include:
- GPTBot: OpenAI’s crawler for gathering AI training data
- OAI-SearchBot: OpenAI’s separate crawler for ChatGPT’s search and citation features
- ClaudeBot: Anthropic’s crawler supporting Claude
- PerplexityBot: indexes pages specifically so Perplexity can cite them in answers
- Google-Extended: a robots.txt control covering both Gemini training and whether Google’s search index can ground AI answers
How Singapore Businesses End Up Blocking Crawlers by Accident
Nobody sets out to block ChatGPT or Perplexity from citing their business. It happens sideways, through a handful of common routes that rarely get reviewed once they are switched on.
A security or firewall plugin installed to stop scraper abuse often ships with a “block known bots” setting turned on by default, and that setting frequently does not distinguish between a bot scraping for spam and one fetching a page to answer a real question. A migration to a new host or content delivery network can also quietly reset robots.txt to a stricter template, undoing access that existed for years.
Cloudflare, which sits in front of a large share of Singapore business websites, now offers a single toggle that blocks these crawlers across the board. That default makes sense for a site mainly worried about its content being scraped to train models without permission. It makes far less sense for a site also hoping to appear in the answers those same models give, since several of the crawlers caught by that one switch are doing the citing, not only the training.
What the Data Actually Shows
The scale of this is larger than most marketing teams assume. Coverage in Search Engine Journal in June 2026 found that 79 percent of top news publishers now block at least one AI training bot by default, a shift that accelerated after major outlets including Reuters and Time changed their default settings. The same reporting cited an industry estimate that roughly three in ten AI bot requests did not actually comply with the access rules a site had published, which cuts in the opposite direction: some crawlers disregard robots.txt regardless of what a business sets.
For a Singapore SME, the useful takeaway is not to copy a global publisher’s defensive stance. Large newsrooms are weighing AI training against licensing negotiations worth real money, a different calculation from a local business trying to be found when someone asks an AI assistant where to get something done. The more relevant question is narrower: whether blocking a specific crawler costs more in lost visibility than it protects, given what that crawler is actually used for.
Checking Your Own Site in Five Minutes
A check takes a few minutes and needs no developer. Start with these steps:
- Open yourdomain.com/robots.txt directly in a browser and read the full file, not just the first few lines
- Look for “Disallow: /” sitting under a named AI user-agent such as GPTBot, ClaudeBot, PerplexityBot or Google-Extended
- Check the security or firewall plugin’s settings for a general “block AI bots” or “block known scrapers” toggle, since this can override robots.txt entirely
- Ask the web host or agency managing the site whether any server-level or CDN rule blocks these user-agents before a request even reaches robots.txt
Is llms.txt Worth Bothering With?
A newer proposal called llms.txt has attracted attention as a kind of robots.txt for AI, meant to point models toward a site’s most important pages in plain language. It sits in the root folder the same way robots.txt does, but describes the site rather than controlling crawler access to it.
No major AI platform has publicly confirmed that it actually reads or acts on these files yet. That makes llms.txt a reasonable low-priority addition for a site that already has one ready, but not something to prioritise ahead of fixing an actual crawler block or strengthening the structured data for AI search that underpins most real citations.
Getting This Right
None of this requires a developer on retainer or a full technical audit. It needs a few minutes with a robots.txt file, a look at whatever security plugin sits in front of the site, and a willingness to treat bot access as a visibility decision and not only a security one. Singapore businesses that get this basic access right give every other investment in brand citations and content structure an actual chance to be seen. If checking your own setup feels like one more thing to find time for, get in touch and we can take a look together.