Summary: I've heard it is a good idea to have a "robots.txt" file. Is it really all that important and if so, just how important is it?
Summary: I've heard it is a good idea to have a "robots.txt" file. Is it really all that important and if so, just how important is it?