Posted on Quarta-feira, 4th Fevereiro 2009 by mestrejoao
I was looking at www.dre.pt while exploring certain functionality for QuickLex and noticed that Diário da República’s robots.txt had some information that I had never seen before:
User-agent: *
Visit-time: 0100-0400
I went to recheck the the robots.txt standard at http://www.robotstxt.org/. Visit-time wasn’t in the standard. What the heck was happening here…
Google to the rescue, and I find [...]
Tags: Add new tag, Diário da República, Google, Robots, Robots exclusion standard, Search Engines
Posted in Web | Comments (0)