# robots.txt file to handle crawlers and spiders User-agent: * Disallow :/files/