# Elastic Web Crawler skipping dynamic content / custom meta fields when indexing my site

**URL:** <https://discuss.elastic.co/t/elastic-web-crawler-skipping-dynamic-content-custom-meta-fields-when-indexing-my-site/391647>\
**Category:** Elasticsearch\
**Created:** [November 16, 2026, 4:46pm UTC](https://discuss.elastic.co/t/elastic-web-crawler-skipping-dynamic-content-custom-meta-fields-when-indexing-my-site/391647 "2026-11-16T16:46:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielcarter890](https://avatars.discourse-cdn.com/v4/letter/d/df705f/32.png) [@danielcarter890](https://discuss.elastic.co/u/danielcarter890)\
**Post date:** [November 16, 2026, 4:46pm UTC](https://discuss.elastic.co/t/elastic-web-crawler-skipping-dynamic-content-custom-meta-fields-when-indexing-my-site/391647/1 "2026-11-16T16:46:41Z")

</div>

Hi everyone,

I’m setting up an Elasticsearch index using the **Elastic Web Crawler** to power an internal search feature for my site: tiktokio cloud .

The crawler connects and indexes the main page titles and basic body text without throwing any HTTP errors, and `robots.txt` isn't blocking anything. However, it seems to miss certain dynamically rendered elements and custom `data-*` / meta attributes that I want mapped as separate keyword fields in my index.

Here is the basic extraction rule structure I’m testing in my crawl configuration:

JSON

```auto
{
  "fields": {
    "page_category": {
      "extract": {
        "css": "meta[name='category']",
        "attribute": "content"
      }
    }
  }
}

```

Does the Elastic Web Crawler execute client-side JavaScript before applying CSS extraction rules, or do I need to enable a headless rendering option / serve pre-rendered HTML for those fields to get indexed properly?

Any pointers would be greatly appreciated!

###

---

<div class="post-metadata">

**Author:** ![alexanderdavis](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@alexanderdavis](https://discuss.elastic.co/u/alexanderdavis)\
**Post date:** [November 16, 2026, 9:18pm UTC](https://discuss.elastic.co/t/elastic-web-crawler-skipping-dynamic-content-custom-meta-fields-when-indexing-my-site/391647/2 "2026-11-16T21:18:55Z")

</div>

Extraction rules run against the HTML the crawler fetches, not the DOM after client-side JS runs (unless your crawler is explicitly configured for JavaScript rendering). If `page_category` is injected at runtime, it won't be seen. Serve that meta tag in the initial server-rendered HTML or prerender the page; your selector/attribute pair is fine for a static `<meta name="category" content="...">`. Verify with `curl`/view-source rather than DevTools, and do the same for any data-\* attributes you need.
