Googlebot Is Getting Caught in robots.txt Spider Traps
on September 13th, 2011 at 5:29 am
we am subsidy his claim. we also got strike by it, some 4 months after environment adult a spider trap that has until now been operative fine.
The couple to a spider trap is rel=Nofollowed, a folder is criminialized in robot.txt. baltimore storage . addiction diagnosis services .
The spider trap works by banning by ip address, not user representative so a not caused by a phony – and of march robots.txt was setup adult rightly and prior, it was in place days before a spider trap was incited on, and a run with no problems for months. mr handyman . portland plumber .
My logs show, it was a genuine google, from a genuine google ip residence that abandoned my robots.txt, abandoned rel-nofollow and fundamentally killed my site.