You might also want to read: How does your crawler work?
Since not all websites have an XML sitemap, the default crawler setting is "full". However, if you have an XML sitemap to your website, you might want to consider using smart crawling.
Using smart crawl instead of the traditional full crawl has several advantages:
- By only crawling your XML sitemap, you'll have more control over what content from your website is crawled, since only the links in your sitemap will be indexed
- Crawling is way more effective because all links are accessed from your sitemap, whereas the traditional crawler will go through all available links on all pages on your website
= less pressure on your servers
- The above allows you to do more frequent crawls. If you use the <lastmod> tag, the smart crawler will usually look for updates every 6 hours*
Enabling smart crawling
Currently, it's not possible to change your crawler from full to small directly from MyCludo.
To enable smart crawling, submit a support ticket and tell us which crawler you want to change to Smart. A support agent will then be in touch to let you know it's done and inform you about the updated crawl frequency. There will be no downtime during this change.
Is smart crawling included in my subscription?
Whether you're an Essential or Enterprise customer, you'll be able to use smart crawling.
*This can vary from contract to contract.