How to prevent unauthorized spidering - asp.net

Question

I want to prevent automated html scraping from one of our sites while not affecting legitimate spidering (googlebot, etc.). Is there something that already exists to accomplish this? Am I even using the correct terminology?
EDIT: I'm mainly looking to prevent people that would be doing this maliciously. I.e. they aren't going to abide by robots.txt
EDIT2: What about preventing use by "rate of use" ... i.e. captcha to continue browsing if automation is detected and the traffic isn't from a legitimate (google, yahoo, msn, etc.) IP.

Answer 1

This is difficult if not impossible to accomplish. Many "rogue" spiders/crawlers do not identify themselves via the user agent string, so it is difficult to identify them. You can try to block them via their IP address, but it is difficult to keep up with adding new IP addresses to your block list. It is also possible to block legitimate users if IP addresses are used since proxies make many different clients appear as a single IP address.
The problem with using robots.txt in this situation is that the spider can just choose to ignore it.
EDIT: Rate limiting is a possibility, but it suffers from some of the same problems of identifying (and keeping track of) "good" and "bad" user agents/IPs. In a system we wrote to do some internal page view/session counting, we eliminate sessions based on page view rate, but we also don't worry about eliminating "good" spiders since we don't want them counted in the data either. We don't do anything about preventing any client from actually viewing the pages.

Answer 2

One approach is to set up an HTTP tar pit; embed a link that will only be visible to automated crawlers. The link should go to a page stuffed with random text and links to itself (but with additional page info: /tarpit/foo.html , /tarpit/bar.html , /tarpit/baz.html - but have the script at /tarpit/ handle all requests with the 200 result).
To keep the good guys out of the pit, generate a 302 redirect to your home page if the user agent is google or yahoo.
It isn't perfect, but it will at least slow down the naive ones.
EDIT: As suggested by Constantin, you could mark the tar pit as offlimits in robots.txt. The good guys use web spiders that honor this protocol will stay out of the tar pit. This would probably get rid of the requirement to generate redirects for known good people.

Answer 3

If you want to protect yourself from generic crawler, use a honeypot.
See, for example, http://www.sqlite.org/cvstrac/honeypot. The good spider will not open this page because site's robots.txt disallows it explicitly. Human may open it, but is not supposed to click "i am a spider" link. The bad spider will certainly follow both links and so will betray its true identity.
If the crawler is created specifically for your site, you can (in theory) create a moving honeypot.

Answer 4

I agree with the honeypot approach generally. However, I put the ONLY link to the honeypot page/resource on a page blocked by "/robots.txt" - as well as the honeypot blocked by such. This way, the malicious robot has to violate the "disallow" rule(s) TWICE to ban itself. A typical user manually following an unclickable link is likely only to do this once and may not find the page containing the honeypot URL.
The honeypot resource logs the offending IP address of the malicious client into a file which is used as an IP ban list elsewhere in the web server configuration. This way, once listed, the web server blocks all further access by that client IP address until the list is cleared. Others may have some sort of automatic expiration, but I believe only in manual removal from a ban list.
Aside: I also do the same thing with spam and my mail server: Sites which send me spam as their first message get banned from sending any further messages until I clear the log file. Although I implement these ban lists at the application level, I also have firewall level dynamic ban lists. My mail and web servers also share banned IP information between them. For an unsophisticated spammer, I figured that the same IP address may host both a malicious spider and a spam spewer. Of course, that was pre-BotNet, but I never removed it.

Answer 5

robots.txt only works if the spider honors it. You can create a HttpModule to filter out spiders that you don't want crawling your site.

Answer 6

You should do what good firewalls do when they detect malicious use - let them keep going but don't give them anything else. If you start throwing 403 or 404 they'll know something is wrong. If you return random data they'll go about their business.
For detecting malicious use though, try adding a trap link on search results page (or the page they are using as your site map) and hide it with CSS. Need to check if they are claiming to be a valid bot and let them through though. You can store their IP for future use and a quick ARIN WHOIS search.

Answer 7

1 install iptables and tcpdump (for linux)
2 detect and autorize good traffic, for example googlebot
in perl
$auth="no";
$host=`host $ip`;
if ($host=~/.googlebot.com\.$/){$auth="si";}
if ($host=~/.google.com\.$/){$auth="si";}
if ($host=~/.yandex.com\.$/){$auth="si";}
if ($host=~/.aspiegel.com\.$/){$auth="si";}
if ($host=~/.msn.com\.$/){$auth="si";}
Note: host googlebot is 55.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-55.googlebot.com.
3 Create schedule or service capture traffic and count packet by host and insert in database or insert in you site a sql query insert ip for count ip traffic
for example in perl
$ip="$ENV{'REMOTE_ADDR'}";
use DBI;
if ($ip !~/^66\.249\./){
my $dbh = DBI->connect('DBI:mysql:database:localhost','user','password') or die print "non connesso:";
my $sth = $dbh->prepare("UPDATE `ip` SET totale=totale+1, oggi=oggi+1, dataUltimo=NOW() WHERE ip ='$ip'");
$sth ->execute;
$rv = $sth->rows;
if ($rv < 1){
my $sth = $dbh->prepare("INSERT INTO `ip` VALUES (NULL, '$ip', 'host', '1', '1', 'no', 'no', 'no', NOW(), 'inelenco.com', oggi+1)");
$sth ->execute;
}
$dbh->disconnect();
}
Or sniff traffic by service for example in perl
$tout=10;
$totpk=3000;
$tr= `timeout $tout tcpdump port $porta -nn -c $totpk`;
#trSplit=split(/\n/,$tr);
undef %contaUltimo;
foreach $trSplit (#trSplit){
if ($trSplit=~/IP (.+?)\.(.+?)\.(.+?)\.(.+?)\.(.+?) > (.+?)\.(.*?)\.(.+?)\.(.+?)\.(.+?): Flags/){
$ipA="$1.$2.$3.$4";
$ipB="$6.$7.$8.$9";
if ($ipA eq "<SERVER_IP>"){$ipA="127.0.0.1";}
if ($ipB eq "<SERVER_IP>"){$ipB="127.0.0.1";}
$conta{$ipA}++;
$conta{$ipB}++;
}
4 block host if traffic is > $max_traffic
for example in perl
if ($conta->{$ip} > $maxxDay){block($conta->{$ip});}
sub block{
my $ipX=shift;
if ($ipX =~/\:/){
$tr= `ip6tables -A INPUT -s $ipX -j DROP`;
$tr= `ip6tables -A OUTPUT -s $ipX -j DROP`;
print "Ipv6 $ipX blocked\n";
print $tr."\n";
}
else{
$tr= `iptables -A INPUT -s $ipX -j DROP`;
$tr= `iptables -A OUTPUT -s $ipX -j DROP`;
print "iptables -A INPUT -s $ipX -j DROP";
print "Ipv4 $ipX blocked\n";
print $tr."\n";
}
}
Another method is read log traffic server.
for example in linux /var/log/apache2/*error.log
contain all query error
/var/log/apache2/*access.log contains all web traffic
create a Bash script read log and block bad spider.
For block attack read all log, for example for block ssh attack read log ssh error and block ip. iptables -A INPUT -s $ip -j DROP