Abuse Rant

Introduction

For several months, I have been attacking my bandwidth problem from a totally different angle than I recently have. I was optimizing images, cutting down on all “extra” content, and compressing content for faster download time. However, after looking through my log files, that was not enough.

The issue

Many legitimate people use different tools to make their personal internet experience a positive one, namely:

• The use of “download managers”
• “Offline browsers” or synchronizing a page (or complete domain) on a schedule

However, these type of tools put stress on a server for really no purpose.

I have witnessed in my log files, particular readers downloading my complete domain (all 700+ pages) and all images every 24 hours. Unfortunately, this is also the exact “signature” that “content thieves” have. These particular instances have also downloaded all content, regardless if it has changed from the previous access.

Another point to make is that these particular readers could be coming from the United States, or from a variety of other countries.

The solution

I have made a conscious decision to ban all download managers and offline browsers, due to abuse of my network.

I have also banned access to my network from various foreign countries at a very low level.

Furthermore, I have banned access by IP addresses that use automated spidering software that choose to ignore robots.txt file. In only one hour, I captured an IP address from the Netherlands spidering all of my content while masquerading as a “valid user.”

Those three items have resulted in a large decrease in bandwidth usage. This also has the positive result of allowing my “legitimate” readers greater bandwidth and faster access while they surf around my content.

The backlash

I understand the implications of my decision. I know that I could possibly ban legitimate users of my content. However, the extreme measures I took was based on an overwhelming amount of abuse. No actions that any web master can take to solve the many issues that plague a public network, such as the internet, comes without risk.

I, however, am willing to take that chance.

On the positive side

Initially, I was wanting to block access to my web server as, under normal conditions, this is the server that uses the most bandwidth. However, I also was wanting to ban particular countries access to my E-Mail servers. This would require a large amount of time to create access “rules” and requirements for too many different situations.

Why, you may ask? The main reason is several countries tend to “generate” much more spam than others (Note: Even though the United States is a very large generator of spam E-Mails, many of them are “bounced” off of servers located in a different country). Sifting through my previous E-Mail archive, I have never received a “legitimate” E-Mail from any person located in several of them. However, a large amount of spam is relayed through these particular foreign countries.

I have also witnessed many attempts at sucking my complete domain, including images, from particular countries. The only thing that I can conclude is these are actions related to attempts at Copyright violations and bandwidth theft.

I have since blocked several countries access to my DNS servers. This has several “good” effects:

• Sending an E-Mail to any address requires a DNS lookup to “discover” the mail exchangers address for the target domain.
  • Blocking DNS queries from an IP address range stops the query and the E-Mail cannot be delivered.
    • That will effectively ban all spam being relayed from unsecured servers located in a country because of the lookup failure.
• I have recently modified my web server to “change” the address by which my domain is accessible from.
  • If a particular user accesses my domain directly by IP address, it will redirect to the default domain, which will require a DNS lookup.
    • That redirection, if stemming from a banned country, will result in no further access to my content because of the lookup failure.

Conclusion

Extreme measures are not even a solution for everyone. Some people and companies require relationships with countries around the world.

I am not one of them.

I have some rather dedicated readers in several locations that contact me often, to include:

• Australia
• Canada
• Europe
• Germany
• Mexico
• United States

Those locations listed above, will never be banned completely.

Black Viper
December 7, 2003

Revision History

December 7 , 2003: Initial release