Christian Whiting
Using Robots.txt Files To Feed The Spiderbots
It's a Thursday evening. You are looking at your website logs to determine where your hits are coming from. You notice you are getting a ton of 404 errors records for a robots.txt file.You might not even know what a robots.txt file is, let...
Date Posted: August 11, 2005