# Kibana - ES scroll returned fewer total hits than expected!

**URL:** https://discuss.elastic.co/t/kibana-es-scroll-returned-fewer-total-hits-than-expected/300931
**Category:** Kibana
**Tags:** elastic-stack-reporting
**Created:** [March 29, 2022, 8:44am UTC](https://discuss.elastic.co/t/kibana-es-scroll-returned-fewer-total-hits-than-expected/300931 "2022-03-29T08:44:03Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![hocho](https://avatars.discourse-cdn.com/v4/letter/h/8e7dd6/32.png) [@hocho](https://discuss.elastic.co/u/hocho)
#### Post date: [March 29, 2022, 8:44am UTC](https://discuss.elastic.co/t/kibana-es-scroll-returned-fewer-total-hits-than-expected/300931/1 "2022-03-29T08:44:03Z")

</div>

Hello,

I have run into an issue where all the events that are displayed in Discover are not exported to the CSV file.

When this happens Kibana generates the following log message in kibana.log:

```auto
ES scroll returned fewer total hits than expected! Search result total hits: 530. Row count: 373.

```

This happens when I perform KQL queries that takes 30+ seceonds to be completed in Discover. Example of an query:

```auto
<field1> : abc123 OR <field2> : *abc456* OR <field3> : abc789

```

**What I think the problem is**  
I am assuming this is caused by some kind of timeout or timer if no response is recieved within in X amount of time from Elasticsearch.

**Workarounds**

1. Storing the query in Saved session before exporting seems make the query faster and therefore the CSV exports the same data as displayed in Discover.

2. Performing the query 5 - 7 times before exporting also seems to improve the repsonse time and therefore the CSV exports the same data as displayed in Discover.

**Question**  
Is there any paramater in kibana.yml that I can configure to increase the amount of time that kibana should wait for responses? I have increased the `elasticsearch.requestTimeout` to `600000` and restarted kibana but does not seem to help.

I am using the following reporting config in kibana.yml

```auto
    # Reporting ---
    xpack.reporting.csv.maxSizeBytes: 209715200
    elasticsearch.requestTimeout: 600000 
    xpack.reporting.queue.timeout: 3000000
    xpack.reporting.csv.scroll.size: 1500
    xpack.reporting.csv.scroll.duration: 1m
    xpack.reporting.roles.enabled: false

```

Running Kibana version 7.17.0

---

<div class="post-metadata">

### Author: ![tsullivan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tsullivan/32/31077_2.png) [@tsullivan](https://discuss.elastic.co/u/tsullivan)
#### Post date: [March 30, 2022, 8:13pm UTC](https://discuss.elastic.co/t/kibana-es-scroll-returned-fewer-total-hits-than-expected/300931/2 "2022-03-30T20:13:47Z")

</div>

> [@hocho](#):
>
> ```auto
> xpack.reporting.csv.scroll.size: 1500
> 
> ```

I suggest lowering this config value. Discover returns a "page" of the first 500 results. CSV export keeps paging through the results using page sizes from this setting. It takes 30+ seconds for Discover to return 500, so you could see an improvement by lowering this back to the default of 500 or even lower.

> 1. Storing the query in Saved session before exporting seems make the query faster and therefore the CSV exports the same data as displayed in Discover.

Good to know this is working! This is something for the Kibana team to look further into and see if we can provide this automatically.

Questions for you:

1. Which version are you using?
2. How long does it take to export the CSV when you use a workaround?

---

<div class="post-metadata">

### Author: ![tsullivan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tsullivan/32/31077_2.png) [@tsullivan](https://discuss.elastic.co/u/tsullivan)
#### Post date: [March 30, 2022, 8:15pm UTC](https://discuss.elastic.co/t/kibana-es-scroll-returned-fewer-total-hits-than-expected/300931/3 "2022-03-30T20:15:06Z")

</div>

> [@hocho](#):
>
> Is there any paramater in kibana.yml that I can configure to increase the amount of time that kibana should wait for responses?

Yes, you are already using it in your config: [`xpack.reporting.csv.scroll.duration`](https://www.elastic.co/guide/en/kibana/current/reporting-settings-kb.html#reporting-csv-settings)

---

<div class="post-metadata">

### Author: ![hocho](https://avatars.discourse-cdn.com/v4/letter/h/8e7dd6/32.png) [@hocho](https://discuss.elastic.co/u/hocho)
#### Post date: [March 31, 2022, 2:16pm UTC](https://discuss.elastic.co/t/kibana-es-scroll-returned-fewer-total-hits-than-expected/300931/4 "2022-03-31T14:16:05Z")

</div>

I implemented your advice to decrease `xpack.reporting.csv.scroll.size` to `500` and I also did increased the following setting `elasticsearch.shardTimeout` to `60000` (ms).

So far this seems to work and I get complete CSV exports.

I am a bit uncertain if Saved session is helping Kibana/Elasticsearch to generate the CSV export faster, it seems that way but could you confirm? When using the Saved session I do the following steps

1. Perform the query and wait until it completes
2. Save the query (top right corner save)
3. Click Save session marker and goto the "Manage Session" page
4. Then I click the Saved session which opens up the Saved session which contains the query etc
5. Generate CSV export

I do the above steps because the current view in Discover does not seem to update with the Session that I just saved.

If I do above steps you will get into the Saved session, and by looking at the URL you can see `&searchSessionId=88e37cf3-6526-4305-asdsdasd`. So I am uncertain if this saved session is used when Kibana starts generating the CSV?

If you dont do above steps you will not be in the Saved session that you saved and the URL will end with (for example) `sort:!(!('@timestamp',desc)))` , might be intented I dont know.

To answer your questions:

1. 7.17
2. 90 MB export took about 4 min

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 28, 2022, 2:16pm UTC](https://discuss.elastic.co/t/kibana-es-scroll-returned-fewer-total-hits-than-expected/300931/5 "2022-04-28T14:16:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
