User-agent: * Visit-time: 0100-0400
I went to recheck the the robots.txt standard at http://www.robotstxt.org/. Visit-time wasn’t in the standard. What the heck was happening here…
Google to the rescue, and I find out that robots.txt is evolving somewhere else: http://www.conman.org/people/spc/robots2.htm
Several new tags are being used in the wild, including the sitemap tag. Check the end of http://google.com/robots.txt.
So, for those that didn’t know (like me), seems like the robots.txt is getting a new life.