Elastic Web Crawler skipping dynamic content / custom meta fields when indexing my site

Hi everyone,

I’m setting up an Elasticsearch index using the Elastic Web Crawler to power an internal search feature for my site: tiktokio cloud .

The crawler connects and indexes the main page titles and basic body text without throwing any HTTP errors, and robots.txt isn't blocking anything. However, it seems to miss certain dynamically rendered elements and custom data-* / meta attributes that I want mapped as separate keyword fields in my index.

Here is the basic extraction rule structure I’m testing in my crawl configuration:

JSON

{
  "fields": {
    "page_category": {
      "extract": {
        "css": "meta[name='category']",
        "attribute": "content"
      }
    }
  }
}

Does the Elastic Web Crawler execute client-side JavaScript before applying CSS extraction rules, or do I need to enable a headless rendering option / serve pre-rendered HTML for those fields to get indexed properly?

Any pointers would be greatly appreciated!

Extraction rules run against the HTML the crawler fetches, not the DOM after client-side JS runs (unless your crawler is explicitly configured for JavaScript rendering). If page_category is injected at runtime, it won't be seen. Serve that meta tag in the initial server-rendered HTML or prerender the page; your selector/attribute pair is fine for a static <meta name="category" content="...">. Verify with curl/view-source rather than DevTools, and do the same for any data-* attributes you need.