Solution for traffic limit exceeded caused by heavy access to files such as pdf, mp4, zip, rar, and jpg.

Published: 2024-07-26

1. Symptoms

Website traffic usage suddenly surges by tens or hundreds of times, and the number of visits also increases sharply. Daily traffic usage reaches tens or hundreds of GB, causing the virtual host to be suspended due to excessive traffic. It may be caused by AI robots scraping frantically.

As shown in the figure:

2. Analysis

After analyzing the access statistics report , it is mainly because one or several large files are accessed heavily, thereby causing a traffic surge, as shown in the figure:

As shown in the images above, it is usually a large file such as pdf, mp4, zip, rar, or jpg that is accessed heavily by different IPs.

After analyzing the web logs, the visiting IPs are generally in the same Class C segment or from a single IP address. The underlying reason may be that a competitor is using tools to consume normal traffic, exhausting the website traffic, or other similar reasons.

As shown in the figure:

3. Solution

For computers, accesses come from different IPs, so it cannot be determined whether they are abnormal.

However, through manual analysis, this explosive access is clearly abnormal and can be resolved through the following three methods:

1. First, analyze whether the access is from the same IP or IP range. If so, you can use the "Request Restriction" feature provided by the virtual host to block access from these IPs. However, if the visitor changes IP, you need to promptly identify and block the new IP. As shown in the figure:

2. A more thorough solution: upgrade to our cloud server. Cloud servers are billed by bandwidth and do not limit traffic, so the website will not be suspended due to excessive traffic.

On this basis, you can also use security groups and other features to block access from abnormal IP ranges.

3. If these large files such as mp3, rar, etc. are not necessary, you can also delete or rename them, or provide downloads only to logged-in users to hide the real addresses of the large files and prevent the file addresses from being exposed.