[ Sponsored Links ]

Advertise here »

Excessive Bandwidth/MS Search

reply to message » post a new message » e-mail to a friend »
Subject:  Excessive Bandwidth/MS Search
Author:   Alex Chudnovsky:   view profile | all posts by this author | add to favourites
Date:   18:43:29 3 December 2004
 
The Times has a cheek to compare search engines while at the same time banning any robots from their own site apart from Google! Here is so-called robots.txt that defines which robots (that support this standard) can not access it - http://business.timesonline.co.uk/robots.txt
 
« Back to Internet Marketing Forum
  • Excessive Bandwidth/MS Search, Dan Zambonini, 30 Nov 12:05
    Just a quick note of caution - if anyone’s noticing a huge increase in their bandwidth over the last month, it’s probably due to the new Microsoft search spider. We’ve had a 10x ...
    • Excessive Bandwidth/MS Search, Loz, 30 Nov 15:37
      This is quite amazing. The spider must have been very active, like some trapped animal gone crazy! If all this traffic had a (considerably) negative impact on your server's perf ...
      • Excessive Bandwidth/MS Search, Dan Zambonini, 30 Nov 17:28
        It is a bit like a trapped animal - something extremely dangerous...  Possibly something rabid, or something with lots of bugs all over it. We're still looking into it - but the ...
        • RE: Excessive Bandwidth/MS Search, Ashley , 1 Dec 10:45
          Mmm... I'm intrigued by the 'denial of service' angle. Could it be that there will be court cases coming againts search engines sometime soon for excessive site crawling? Loss of r ...
      • Excessive Bandwidth/MS Search, Alex Chudnovsky, 2 Dec 18:46
        > If all this traffic had a (considerably) negative impact on your > server's performance then could it be classified as a denial > of service attack? Unlikely - DoS attack ...
    • Excessive Bandwidth/MS Search, DeriJones, 3 Dec 18:21
      Well, I hate to say 'told you so', but on Nov 11th I did...see the Times newspaper: http://business.timesonline.co.uk/article/0,,9075-1354322,00.html But hey, don't you just ...
      • Excessive Bandwidth/MS Search, Dan Zambonini, 3 Dec 18:32
        No, that's a good point.  It's using the     MS Search 4.0 Robot user agent, but I'll check the IP address range - could well be a spoofer.  I'll let you know - good thinking. ...
        • Excessive Bandwidth/MS Search, DeriJones, 6 Dec 11:27
          The IP address is vital - can you let us know ASAP? See the discussion here from a year ago, of some dodgy guys using the MS search bot User Agent on their bot: http://www. ...
      • Excessive Bandwidth/MS Search, Alex Chudnovsky, 3 Dec 18:43
        The Times has a cheek to compare search engines while at the same time banning any robots from their own site apart from Google! Here is so-called robots.txt that defines which ro ...
« Back to Internet Marketing Forum Post a new message »
Subscribe for only €299