|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|
Re: [PLUG] Do no robots texts work?
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Wednesday 05 May 2004 15:59, George Theall wrote:
| I maintain a few small sites, monitor my logs pretty closely, and have a
| couple of traps for bad robots, including a bogus setting in my
| robots.txt files telling robots not to visit a non-existent area of my
| webs. While I do find plenty of examples of 'bots that completely
| ignore restrictions in robots.txt, I can't recall the last time I saw
Specifically, Yahoo!'s Slurp and MSN's crawler _WILL_ ignore robots.txt
sometimes, but its not clear when (not clear to me, anyway).
- --
Tobias DiPasquale
202A 04C4 2CE6 B985 8520 88D6 CD25 1A6C B9B5 1595
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFAmVAHzSUabLm1FZURAqRvAKCYNJDl9losswRk0VmazIMDhH13agCfa3b3
pYuozaErDXBVIwgf6Qc9fPI=
=9TyK
-----END PGP SIGNATURE-----
___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
|
|