Data attributes for inclusion and exclusion are not working

Elastic Stack Elasticsearch
1

I've implemented the "data-elastic-exclude" attribute in the body tag and "data-elastic-include" in specific content sections to ensure that only those parts are indexed by Elasticsearch web crawler. However, it appears that this functionality isn't functioning as expected. Despite these attributes, the web crawler is gathering all content within the body, including navigation, sidebar, and footer elements. Are there any specific requirements or considerations I should be aware of when using "data-elastic-exclude" and "data-elastic-include"? Can I exclude header and footer at least from body_content?

Here's the relevant code snippet:

<body data-elastic-exclude data-paywall-content-type="premium">
   ...
   <section data-elastic-include class="content article-content">
        {{ article.content|safe }}
   </section>
   ...
</body>
2

@Dongmyoung_Kim Did you resolve this, or find a workaround? We are seeing the same issue.

I have a data-elastic-include attribute on a div that wraps my main content, which is the only thing on a page we want to index.

Expected behavior: The global navigation on my site, which is not in this div, should not be indexed.

From the docs:

For all pages that contain HTML tags with a data-elastic-include attribute, the crawler will only index content within those tags.

Actual behavior: The global navigation is indexed and is included in the body_content field.

This negatively impacts both our results and the snippets shown alongside them.

Edit: This seems like a bug to me. I've opened an issue here: Crawler attributes like `data-elastic-include` are ignored · Issue #152 · elastic/crawler · GitHub