# Crawlers Setup User-agent: * Allow: / #Crawl-delay: 60 #Visit-time: 2330-0530 #Request-rate: 1/1m # Directories Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /downloader/ Disallow: /lib/ Disallow: /magento/ Disallow: /pkginfo/ Disallow: /report/ Disallow: /stats/ Disallow: /shell/ Disallow: /var/ # Paths (clean URLs) Disallow: /index.php/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Disallow: /catalog/product/gallery/ Disallow: /catalogsearch/ Disallow: /search/ Disallow: /checkout/ Disallow: /control/ Disallow: /contacts/ Disallow: /customer/ Disallow: /customize/ Disallow: /newsletter/ Disallow: /poll/ Disallow: /review/ Disallow: /sendfriend/ Disallow: /tag/ Disallow: /wishlist/ Disallow: /productalert/ Disallow: /onestepcheckout/ Disallow: /catalog/seo_sitemap/ Disallow: /productquestion/ Disallow: /gomage_social/ # Files Disallow: /cron.php Disallow: /cron.sh Disallow: /error_log Disallow: /install.php Disallow: /LICENSE.html Disallow: /LICENSE.txt Disallow: /LICENSE_AFL.txt Disallow: /STATUS.txt # Paths (no clean URLs) Disallow: /*?p=*& Disallow: /*?SID= Disallow: /errors/ #Disallow: /*mode= #Disallow: /*limit= #Disallow: /*dir= #Disallow: /*order= # Google User-agent: Googlebot Disallow: User-agent: Googlebot-image Disallow: User-agent: MJ12bot Disallow: / User-agent: Vagabondo Disallow: / User-agent: BaiDuSpider Disallow: / User-agent: Exabot Disallow: / # # Yandex tends to be rather aggressive, may be worth keeping them at arms lenght User-agent: YandexBot Crawl-delay: 120 User-agent: BSpider Disallow: / Sitemap: https://www.mifarma.es/media/mf_sitemap_ssl.xml Sitemap: https://www.mifarma.pt/sitemap_pt.xml Sitemap: https://www.mifarma.fr/sitemap_fr.xml Sitemap: https://www.mifarma.co.uk/sitemap_uk.xml Sitemap: https://www.mi-farma.it/sitemap_it.xml Sitemap: https://www.nutrix.es/nutrix_sitemap.xml