EN
CN
Training large-scale AI models relies heavily on massive amounts of data. Major AI companies deploy web crawlers to scrape data from the internet indiscriminately; the sheer volume and frequency of these crawlers rival DDoS attacks. If your VPS hosts websites, API services, or AI applications, sensitive information—such as server details, business data, and even API keys—could inadvertently be scraped or leaked by these AI crawlers.
This article outlines a comprehensive strategy for protecting VPS information in the AI era, covering everything from risk identification and preventive measures to practical implementation steps.
Key Risks Facing VPS in the AI Era
Indiscriminate scraping of website content by AI crawlers. AI companies deploy vast numbers of crawlers to scrape internet data for model training. These crawlers operate at high volumes and frequencies, and many ignore the `robots.txt` protocol entirely. Originating from tens of thousands of global IP addresses and using randomized User-Agent strings, they are difficult to block effectively.
Data leaks due to exposed self-hosted AI assistants. Popular AI assistant tools like Clawdbot can be deployed on VPS; however, some users fail to configure security policies, leaving instances directly exposed to the public internet. Security researchers have identified over 1,000 exposed Clawdbot instances, at least 300 of which lack any authentication mechanism. Once discovered by attackers, these assistants can be exploited to access sensitive user data, including conversation logs, API keys, and messaging platform credentials.
Exploitation of misconfigured AI infrastructure. In January 2026, security vendors reported a large-scale attack campaign targeting misconfigured proxy servers, aiming to gain free access to paid LLM services like OpenAI and Google. Attackers scanned the internet for Ollama instances with vulnerabilities; successful exploitation allowed them to force the server to connect to external infrastructure under their control. Over 175,000 Ollama AI servers were found to be publicly exposed on the internet.
Preventive Measures: Blocking AI Crawlers from Scraping Your Content
Method 1: Use Cloudflare to block AI crawlers with a single click
This is currently the simplest and most effective method. Cloudflare offers free protection features specifically designed to block AI crawlers.
As of September 15, 2026, all websites using Cloudflare block general-purpose AI crawlers by default. Cloudflare categorizes AI traffic into three independently controllable behaviors: Search, Agent, and Training. You can navigate to Security Settings → Bot traffic in the Cloudflare dashboard and select the option to block training crawlers. Once enabled, Cloudflare maintains an automatically updated `robots.txt` file that explicitly instructs known AI crawlers to stay away from your content.
Method 2: Configure `robots.txt` to block AI crawlers
Create or modify the `robots.txt` file in your website's root directory and add the following rules:
User-agent: GPTBot
Disallow: /
User-agent: Chat GPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Bytespider
Disallow: /Note: `robots.txt` serves as a request rather than a mandate; some AI crawlers do not comply with it. Therefore, it is recommended to use this in conjunction with other methods.
Method 3: Block AI crawlers at the Nginx level
For scenarios where Cloudflare is not used, you can directly block the User-Agents of known AI crawlers within Nginx.
Add a `map` directive within the `http` block of `/etc/nginx/nginx.conf`:
map $http_user_agent $block_ai_bot {
default 0;
~*GPTBot 1;
~*ClaudeBot 1;
~*Bytespider 1;
~*CCBot 1;
~*PerplexityBot 1;
~*Google-Extended 1;
}
```Then, add the following to your `server` block:
if ($block_ai_bot) {
return 403;
}Method 4: Use WordPress plugins
If your site is built on WordPress, you can install plugins such as WindCodex ScraperBlock to enable AI crawler blocking with a single click from the dashboard.
Security configuration for self-hosted AI assistants
If you have deployed an AI assistant (such as Clawdbot, OpenClaw, or Hermes Agent) on a VPS, the following measures are crucial:
Do not expose the AI assistant directly to the public internet. By default, the AI assistant might be accessible directly via the public internet. An authentication mechanism must be configured.
Run in isolated Docker containers. Host the AI assistant service in a dedicated Docker container. If a container is compromised, the attack surface remains contained and does not spread to the entire VPS. Use a private Docker network for internal communication to prevent backend services from being directly exposed.
Configure a reverse proxy and firewall. Handle HTTPS traffic using a reverse proxy like Nginx or Caddy, and open only necessary ports (e.g., port 443 for HTTPS). Avoid exposing Docker ports, databases, or MCP services publicly.
Restrict access permissions. Use SSH keys for login, disable password authentication, and change the default SSH port. Restrict administrative access using IP whitelisting. Consider using a tool like Tailscale for network isolation rather than exposing services directly to the public internet.
Protect sensitive credentials. Do not store sensitive information, such as API keys, in environment variables; instead, use Docker secrets (tmpfs mounts) or the operating system's keyring.
Hardening LLM Infrastructure Security
If you are running LLM services like Ollama on a VPS, consider the following:
Restrict model sources. Limit Ollama model pulls to trusted registries. Implement strict egress filtering to block known malicious callback domains at the DNS level.
Monitor for anomalous access. Apply rate limiting to suspicious Autonomous System Numbers (ASNs). Monitor for network fingerprints associated with automated scanning tools.
Use network segmentation and IP whitelisting. Restrict access through network segmentation and IP whitelisting. Ensure all connections between clients and the server are encrypted using the latest TLS version.
Zero-Trust Architecture: Enhanced Security
For scenarios requiring high security, consider a zero-trust architecture:
Use Tailscale to establish a private network. Once the VPS joins the Tailscale network, services are exposed only via private IPs, with no public-facing ports. This prevents attackers from even having the opportunity to scan for open ports.
Implement the principle of least privilege. Run AI services using a dedicated non-root account. Assign separate containers to different users. Limit the resources the AI assistant can access and the actions it can perform.
Deploy sensitive data protection tools. Use tools like ai-noleak to ensure sensitive credentials are not leaked through AI assistant prompts or HTTP calls. To protect your VPS server information from being harvested by AI models in the AI era, you need to take a three-tiered approach:
| Protection Layer | Core Measures | Protection Goal |
| Blocking AI crawlers | Cloudflare protection, robots.txt, Nginx blocking, WordPress plugins | Prevent website content from being scraped for training |
| Self-hosted AI assistant security | Container isolation, reverse proxy, authentication mechanisms | Prevent AI assistants from being exploited to leak data |
| LLM infrastructure hardening | Restricting pull sources, egress filtering, anomaly monitoring | Prevent AI services from being abused by attackers |
A quick tip: Don't leave your VPS "exposed" in the AI era. Whether for website content or AI services, multi-layered protection is essential—from using Cloudflare to block crawlers and isolating AI assistants in containers to implementing zero-trust network architecture, every layer counts. Jtti provides high-performance VPS solutions and professional technical support to help you securely deploy various business operations and AI applications in the AI era. Visit the Jtti official website to learn more about our cloud server solutions.
