17 Jan
2011
17 Jan
'11
4:21 p.m.
Thanks for at least trying. Yes, I'm aware of robotstxt.org. My question was not regarding user-agents, but about whether a string such as /foo blocks only http://website.org/foo, or if it would block http://website.org/blah/foo. I think it would not, but I'm looking for documentation, references, or examples that specifically address that.
From reading Google's reference on robots.txt as extended by Google,
Microsoft, and Yahoo, the most obvious interpretation would be that I need to block /*foo, but again, no specific confirmation.
As for the trailing slash, again, no specific reference to that format so far.