AJAX generated content allows the web page and web browser to process data without having to reload the page. For example, if you hit a “Submit” button on the page, AJAX processes the information and updates the content without the browser reloading. This can often occur when pagination occurs and is loaded using AJAX.
With Cludo’s standard crawlers, it indexes the page before any AJAX technology is applied. As a result, it is not able to see content loaded with it.
In cases where AJAX technology is being used, Cludo can use an Async crawler to index the content at an additional price, or you can provide a sitemap for links located behind AJAX content.
How do I know if I have AJAX generated content?
There are a few ways to identify if a page uses AJAX to load content.
Check the HTML source code for content
If the content is visible on the page but not in the HTML source code, the content is being loaded Asynchronously. For example:
What are the options for crawling AJAX content?
Depending on what is being loaded using AJAX, there are some different methods for indexing the content.
AJAX with Pagination
In instances where additional links and content are behind ajax buttons, you can create a sitemap with the links to index or crawl the page using the Async crawler.
Content is loaded using AJAX
For instances where specific content is being loaded using AJAX, you may choose to insert that content into a meta tag for Cludo to crawl with a standard crawler or use the Async crawler.
Crawling your web pages using the Async crawler requires an additional fee. Contact our team for more information.
If you have any questions about the best methods for indexing your content, do not hesitate to reach out to support.