Hi everyone,
I’m setting up an Elasticsearch index using the Elastic Web Crawler to power an internal search feature for my site: tiktokio cloud .
The crawler connects and indexes the main page titles and basic body text without throwing any HTTP errors, and robots.txt isn't blocking anything. However, it seems to miss certain dynamically rendered elements and custom data-* / meta attributes that I want mapped as separate keyword fields in my index.
Here is the basic extraction rule structure I’m testing in my crawl configuration:
JSON
{
"fields": {
"page_category": {
"extract": {
"css": "meta[name='category']",
"attribute": "content"
}
}
}
}
Does the Elastic Web Crawler execute client-side JavaScript before applying CSS extraction rules, or do I need to enable a headless rendering option / serve pre-rendered HTML for those fields to get indexed properly?
Any pointers would be greatly appreciated!