I have had several complains from readers about my web server automatically banning their IP address because of "abuse." This is due to my recently implemented configuration to stop "bad" robots from sucking too much bandwidth. More information on this practice is located in my Abuse Rant. I implemented a "hidden link" which all "good" robots, including all major search engines, would ignore. Some proxy servers, pre-fetchers and firewall’s chose to ignore the robots.txt standard. This is an issue to take up with the creators of those programs, not my web site.
I have temporarily removed that particular function from the server. All access is currently available with the following exceptions:
- I still block all "offline browser" access. This is due to many people synchronizing the entire domain (740+ pages) every day, which is entirely not needed.
- Most "download managers" remain blocked. Please use the "normal" means of downloading my files. No user name and password is required to do so. If such information is requested, it could be due to the use of a download manager.
- Access by "page editors" are not authorized. I hope the reasons are obvious.
- When a "bad" robot hits, I will still immediately get a report via E-Mail and will selectively disable IP addresses instead of automatically banning them.
I appreciate everyone’s feedback while I fine tune the domain to provide everyone rapid content while not alienating others.