# /robots.txt for webcrawler.com frontend machines User-agent: * Disallow: /YHS/Profile/ Disallow: /YHS/Messages/ Disallow: /YHS/Reunion/