Better to implement the rel=nofollow attribute to prevent us giving the nasty ad sites PageRank from the "please blacklist this" links we post!
See Google's blog for more info: http://googleblog.blogspot.com/2005/01/ ... -spam.html
rel=nofollow for links
.. I just searched for "Google doesn't deep-crawl this site anyway (don't know why). " and we came up 5th on Google.rick752 wrote:Google doesn't deep-crawl this site anyway (don't know why).
Afaict, the private area here cannot be crawled as it is hidden and password protected .... Gogglebot probably wouldn't even try to get in.
I also seen that "Don't get drunk by the pool" in Community Laughter, Fun and Comedy
returns this site at about #15 on Google.
I alos block a few IP addresses due to the excessive number of hits. I'll have to go look at those stats again. Right now it looks like MSN is hitting like crazy...
MSN always hits like crazy.rbsterli wrote:Right now it looks like MSN is hitting like crazy...
The former URL of my adblock list on a university server was answering with HTTP 301
for more than a year and a half, and MSN came back at least twice daily.
I then changed to plain 404 - and still see at least one MSN hit per day for that URL.
Their crawler seems to be completely bonkers.
To prevent this forum from being crawled, you should look into creating a robots.txt -
at least google respects those settings and permissions.