## Robots.txt for Magento Community and Enterprise ## GENERAL SETTINGS # Enables robots.txt rules for all crawlers User-agent:* ## Crawl-delay parameter: the number of seconds you want to wait between successful requests to the same server. ## Set a crawl rate, if your server's traffic problems. Please note that Google ignore crawl-delay setting in Robots.txt. You can set up this in Google Webmaster tool # Crawl-delay: 30 Crawl-delay: 30 ## Magento sitemap: URL to your sitemap file in Magento # Sitemap: https://vn.toto.com/sitemap.xml ## Settings that relate to the UNDER CONSTRUCTION ## Do not allow indexing files and folders that are required during development: CVS, SVN directory and dump files Disallow: /CVS Disallow: /*.Svn$ Disallow: /*.Idea$ Disallow: /*.Sql$ Disallow: /*.Tgz$ ## GENERAL SETTINGS For MAGENTO ## CUSTOM RULES GOES HERE Disallow: /index.php/ Disallow: /checkout/cart/ Disallow: /onepage/ Disallow: /customer/account/create Disallow: /control/ Disallow: /contacts/ Disallow: /customize/ Disallow: /newsletter/ Disallow: /poll/ Disallow: /review/ Disallow: /sendfriend/ Disallow: /tag/ Disallow: /wishlist/ Disallow: /du-an-tham-khao/* Disallow: /author/ Disallow: /2017/* Disallow: /2016/* Disallow:*/filter/* Allow:*/filter/technology/* Disallow:*/filter/technology/*,* Disallow:*/filter/technology/*/* Disallow:*/filter/cat/*,* Disallow:*/filter/cat/*/* Allow:*/filter/functionality/* Disallow:*/filter/functionality/*,* Disallow:*/filter/functionality/*/* ## Do not index the page Magento admin Disallow: /admin/ ## Do not index the general technical Magento directory Disallow: /app/ Disallow: /downloader/ Disallow: /errors/ Disallow: /includes/ Disallow: /lib/ Disallow: /pkginfo/ Disallow: /shell/ Disallow: /var/ ## Do not index the shared files Magento Disallow: /api.php Disallow: /cron.php Disallow: /cron.sh Disallow: /error_log Disallow: /get.php Disallow: /install.php Disallow: /LICENSE.html Disallow: /LICENSE.txt Disallow: /LICENSE_AFL.txt Disallow: /README.txt Disallow: /RELEASE_NOTES.txt ## MAGENTO SEA IMPROVEMENT ## Do not index the page subcategories that are sorted or filtered. Disallow: /*?Dir* Disallow: /*?Dir=desc Disallow: /*?Dir=asc Disallow: /*?Limit=all Disallow: /*?Mode* ## Do not index the second copy of the home page (example.com/index.php/). Un-comment only if you have activated Magento SEO URLs. ## Disallow: /index.php/ ## Do not index the link from the session ID Disallow: /*?SID= ## Do not index the page checkout and user account Disallow: /checkout/ Disallow: /onestepcheckout/ Disallow: /customer/ Disallow: /customer/account/ Disallow: /customer/account/login/ ## Do not index the search page and CEO, non-optimized link categories Disallow: /catalogsearch/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ ## Server Settings ## Do not index the general technical directories and files on a server Disallow: /cgi-bin/ Disallow: /cleanup.php Disallow: /apc.php Disallow: /memcache.php Disallow: /phpinfo.php ## SETTINGS Image indexing ## Optional: If you do not want to Google and Bing to index your images # User-agent: Googlebot-Image # Disallow: / # User-agent: msnbot-media # Disallow: / ## Do not index pointofsale Disallow: /pointofsales Disallow: /wp-admin/ Disallow: /search?q=* Disallow: *?replytocom Disallow: */attachment/* Disallow: /wp-admin/admin-ajax.php Disallow: /wp-admin/images/* Disallow: /contents/* Disallow: /portfolio_entries/* Disallow: /category/* Disallow: /portfolio_entries/* Disallow: /2016/* Disallow: /index.php?l=en&m=don-dat-hang* Disallow: /storelocator/* Disallow: /media/revision/* #User-agent: Googlebot #Disallow: / #User-agent: coccoc #Disallow: / #User-agent: Googlebot/2.1 #Disallow: / #User-agent: coccocbot-shopping/1.0 #Disallow: / #User-agent: coccocbot-shopping #Disallow: / #User-agent: coccocbot #Disallow: / #User-agent: coccocbot-web #Disallow: / #User-agent: coccocbot-image #Disallow: / #User-agent: coccocbot-fast #Disallow: / #User-agent: coccocbot-ads #Disallow: / #User-agent: Googlebot #Disallow: / #User-agent: Googlebot-News #Disallow: / #User-agent: Googlebot-Image #Disallow: / #User-agent: Googlebot-Video #Disallow: / #User-agent: Mediapartners-Google #Disallow: / #User-agent: AdsBot-Google-Mobile-Apps #Disallow: / #Update-Dec15 Disallow: /index.html Disallow: /contents/ Disallow: */? Disallow: /index.php Disallow: /*?* Disallow: /wp-content/themes/ Disallow: /wp-content/plugins/