When using a Singapore server, a sudden monitoring alert appears, indicating that bandwidth usage consistently exceeds 95%, causing your website or application to become extremely slow—a situation that most operations and maintenance personnel have experienced. How can you quickly find the root cause and resolve it? When faced with full bandwidth, the key is to quickly determine whether it's "normal traffic" or an "abnormal attack" and take appropriate measures. An e-commerce website experiencing bandwidth strain due to a surge in users during a promotional period is fundamentally different from bandwidth exhaustion caused by a DDoS attack; the underlying logic and solution are completely different.
When bandwidth is critically low, the first step is not blindly upgrading, but precise diagnosis. You need to answer three questions: How much bandwidth is being used? Which process is using it? Where is the traffic flowing?
Usually, `vnstat` is used to view historical trends to determine whether it's a sudden spike or a sustained high level.
vnstat -d
However, if you need real-time monitoring, `iftop` is a more intuitive tool. It displays the IP connections consuming bandwidth in real time, like a "task manager."
sudo iftop -P
After running, the screen will dynamically refresh. Pay close attention to the total bandwidth utilization displayed at the top, and the IP addresses and traffic volumes on either side of the `=>` and `<=` symbols in the list below. This visually tells you who your Singapore server is communicating with and the direction of the traffic.
If `iftop` shows a large number of connections from unknown IPs, especially with a single target port and huge traffic, it's likely under attack. In this case, you need to use the `nethogs` command to pinpoint the culprit from a process perspective.
sudo nethogs eth0
`nethogs` will list the real-time bandwidth generated by each process (such as nginx, php-fpm, mysql). If you find an unknown process or one that shouldn't be consuming a lot of bandwidth at the top, it's likely malware or a misconfigured program.
If the traffic is confirmed to be from legitimate users (e.g., new product launches, marketing campaigns), a short-term emergency measure is temporary capacity expansion. Almost all cloud platforms support elastic bandwidth, which takes effect within minutes with a few clicks in the console. However, this is only a stopgap measure and requires long-term optimization.
Compress everything compressible: Ensure your website has Gzip or the more efficient Brotli compression enabled. In Nginx, you can configure it like this:
nginx gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml image/svg+xml;
gzip_min_length 1024;
gzip_comp_level 6;
This can reduce the size of text resources (HTML, CSS, JS) by more than 60%.
Utilize CDN: Host static resources (images, videos, CSS/JS libraries) on a CDN. This not only shifts bandwidth pressure to the CDN provider but also allows users to obtain resources from the nearest node, resulting in faster speeds. Simply change your website resource links to point to CDN addresses.
If diagnostics reveal a large amount of traffic originating from a few IPs or pointing to non-web ports, an attack is highly likely.
Quickly stop the bleeding: Enable cloud firewall/security groups. Log in to your cloud service provider's console immediately, configure security group rules, and block suspicious IP ranges. For example, if the attack originates from the `123.123.123.0/24` network segment, add a deny rule.
Application-layer rate limiting: For CC attacks targeting web ports, rate limiting can be implemented at the Nginx level.
nginx
# Define the rate limiting zone in the http block
limit_req_zone$binary_remote_addr zone=one:10m rate=10r/s;
# Apply in the server or location block
location/{ limit_req zone=one burst=20 nodelay;
#...other configurations
}
This configuration limits the request rate per IP to 10 requests per second, with a burst limit of 20 requests per second. Requests exceeding this limit are dropped to protect backend resources.
Seeking assistance with DDoS protection services: For large-scale DDoS attacks, self-defense is often insufficient. In this case, immediately activate the cloud provider's DDoS protection IP service. Its principle is that all traffic first passes through the DDoS cleaning center, filtering out malicious traffic before forwarding normal traffic to your origin server in Singapore. Sometimes the problem lies internally. A common cause is that the Singapore server has been infected with a mining program or backdoor, constantly sending data out. After finding the suspicious process using `nethogs`, use `ps aux|grep <process name>` to locate its path and completely remove it.
Another possibility is a network traffic storm caused by misconfiguration, such as incorrect mirror synchronization or log push settings. Check the scheduled task `crontab -l` and application configuration files.
After resolving the crisis, it's crucial to establish mechanisms to prevent recurrence.
1. Set up monitoring and alerts: Use cloud monitoring or tools like Zabbix or Prometheus to set threshold alerts for bandwidth usage (e.g., >80%). This will allow you to receive warnings before the bandwidth reaches its maximum capacity.
# A simple example of a custom monitoring script, which can be added to crontab for scheduled execution.
bandwidth_usage=$(vnstat--oneline|awk-F';''{print$11}')
threshold=95
if[[${bandwidth_usage%.*}-gt$threshold]];then
echo"Warning: Bandwidth utilization has reached ${bandwidth_usage}%"|mail-s"Bandwidth alert"admin@example.com
fi
2. Implement capacity planning: Regularly analyze business growth curves and plan bandwidth upgrades in advance. For example, proactively conduct stress tests and reserve bandwidth before planning large-scale promotional activities.
3. Optimize application architecture: In the long term, consider microservice architecture for applications and allow non-core, bandwidth-intensive services (such as file downloads and video streaming) to use independent object storage and CDN, thus architecturally preventing core business from being hampered by bandwidth issues.
When your Singapore server bandwidth is critically low again, remember this clear action plan: Don't panic. Use `iftop` and `nethogs` for quick diagnosis; distinguish between a "welcome business peak" and a "malicious traffic attack"; then take appropriate measures: optimize where necessary, limit traffic where necessary, and immediately activate DDoS protection if needed.
EN
CN