Commit graph

5 commits

Author SHA1 Message Date
rugk
a5730b701c Revert order change
According to Wikipedia this is indeed right.. https://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive

Then there are only many wrong examples out there.
2016-07-03 13:09:47 +02:00
rugk
0567a720fd Change order of allow rule in robots.txt
I think this is the proper order. Don't know if it matters, but it is not bad to change it in any case.
2016-07-03 13:05:26 +02:00
ic0nic
c918a9fe2e Incorrect structure
The structure for robots.txt is incorrect for some/most search engines.
2013-11-01 01:22:16 +01:00
ic0nic
50d4fb9bf3 Allow discovery
To allow search engines to at least find the website.
2013-01-29 19:33:31 +00:00
Greg Knaddison
f37303d858 For server-load and privacy reasons, disallow robots. 2012-08-10 18:21:26 +02:00