# Preventing robots ( like indexer swish ) from entering certain directories # Good info on this : info.webcrawler.com/mak/projects/robots/norobots.html # Memo: To exclude a directory only your Disallow should end in a / User-agent: * Disallow: /~archive/ Disallow: /access/