# Disallow particular paths for all user agents User-agent: * Disallow: / Crawl-delay: 30 # Disallow entire user agents User-agent: Semrush Disallow: /