# Elastic Web crawler not able to parse complete html of a page, other Search engine able to crawl , elastic misses / ignore many sections of page

**URL:** <https://discuss.elastic.co/t/elastic-web-crawler-not-able-to-parse-complete-html-of-a-page-other-search-engine-able-to-crawl-elastic-misses-ignore-many-sections-of-page/297654>\
**Category:** Elastic Search\
**Created:** [February 19, 2022, 4:56am UTC](https://discuss.elastic.co/t/elastic-web-crawler-not-able-to-parse-complete-html-of-a-page-other-search-engine-able-to-crawl-elastic-misses-ignore-many-sections-of-page/297654 "2022-02-19T04:56:02Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sanjeev\_kalwar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanjeev_kalwar/32/98831_2.png) [@sanjeev\_kalwar](https://discuss.elastic.co/u/sanjeev_kalwar)\
**Post date:** [February 19, 2022, 4:56am UTC](https://discuss.elastic.co/t/elastic-web-crawler-not-able-to-parse-complete-html-of-a-page-other-search-engine-able-to-crawl-elastic-misses-ignore-many-sections-of-page/297654/1 "2022-02-19T04:56:02Z")

</div>

We are trying to crawl a site using elastic enterprise app search web crawler. We are facing a problem that full content of the page is not getting indexed. In most of the pages the indexing stops after a certain point and rest of html is ignored, In some pages it crawls some section , skips some then again crawls rest of the webpage sections. This is happening for the body content. We do not know if this is due to a malformed html, how do we identify that. Is there any option or config to fix this.

While the existing search solution that uses Adobe S&P is able to search and index all contents.

we are using 7.16.1 version

We are facing this issue with many pages in this site.

Need urgent help

---

<div class="post-metadata">

**Author:** ![oleksiy-elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiy-elastic/32/49074_2.png) [@oleksiy-elastic](https://discuss.elastic.co/u/oleksiy-elastic)\
**Post date:** [February 20, 2022, 6:48pm UTC](https://discuss.elastic.co/t/elastic-web-crawler-not-able-to-parse-complete-html-of-a-page-other-search-engine-able-to-crawl-elastic-misses-ignore-many-sections-of-page/297654/2 "2022-02-20T18:48:55Z")

</div>

Hello, Sanjeev

Sorry you are experiencing the issues with your web crawler. Can you please share a link to the pages that fail to index? It’d help us troubleshoot content extraction for you.

Additionally, you can use the content extraction API to see what the crawler sees on a specific page: [Web crawler API reference | Elastic App Search Documentation [8.0] | Elastic](https://www.elastic.co/guide/en/app-search/current/web-crawler-api-reference.html#web-crawler-apis-post-crawler-extract-url)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 8:34am UTC](https://discuss.elastic.co/t/elastic-web-crawler-not-able-to-parse-complete-html-of-a-page-other-search-engine-able-to-crawl-elastic-misses-ignore-many-sections-of-page/297654/3 "2022-11-04T08:34:18Z")

</div>