Kibana - ES scroll returned fewer total hits than expected!

Elastic Stack Kibana
1

Hello,

I have run into an issue where all the events that are displayed in Discover are not exported to the CSV file.

When this happens Kibana generates the following log message in kibana.log:

ES scroll returned fewer total hits than expected! Search result total hits: 530. Row count: 373.

This happens when I perform KQL queries that takes 30+ seceonds to be completed in Discover. Example of an query:

<field1> : abc123 OR <field2> : *abc456* OR <field3> : abc789

What I think the problem is
I am assuming this is caused by some kind of timeout or timer if no response is recieved within in X amount of time from Elasticsearch.

Workarounds

  1. Storing the query in Saved session before exporting seems make the query faster and therefore the CSV exports the same data as displayed in Discover.

  2. Performing the query 5 - 7 times before exporting also seems to improve the repsonse time and therefore the CSV exports the same data as displayed in Discover.

Question
Is there any paramater in kibana.yml that I can configure to increase the amount of time that kibana should wait for responses? I have increased the elasticsearch.requestTimeout to 600000 and restarted kibana but does not seem to help.

I am using the following reporting config in kibana.yml

    # Reporting ---
    xpack.reporting.csv.maxSizeBytes: 209715200
    elasticsearch.requestTimeout: 600000 
    xpack.reporting.queue.timeout: 3000000
    xpack.reporting.csv.scroll.size: 1500
    xpack.reporting.csv.scroll.duration: 1m
    xpack.reporting.roles.enabled: false

Running Kibana version 7.17.0

2

I suggest lowering this config value. Discover returns a "page" of the first 500 results. CSV export keeps paging through the results using page sizes from this setting. It takes 30+ seconds for Discover to return 500, so you could see an improvement by lowering this back to the default of 500 or even lower.

  1. Storing the query in Saved session before exporting seems make the query faster and therefore the CSV exports the same data as displayed in Discover.

Good to know this is working! This is something for the Kibana team to look further into and see if we can provide this automatically.

Questions for you:

  1. Which version are you using?
  2. How long does it take to export the CSV when you use a workaround?
3

Yes, you are already using it in your config: xpack.reporting.csv.scroll.duration

4

I implemented your advice to decrease xpack.reporting.csv.scroll.size to 500 and I also did increased the following setting elasticsearch.shardTimeout to 60000 (ms).

So far this seems to work and I get complete CSV exports.

I am a bit uncertain if Saved session is helping Kibana/Elasticsearch to generate the CSV export faster, it seems that way but could you confirm? When using the Saved session I do the following steps

  1. Perform the query and wait until it completes
  2. Save the query (top right corner save)
  3. Click Save session marker and goto the "Manage Session" page
  4. Then I click the Saved session which opens up the Saved session which contains the query etc
  5. Generate CSV export

I do the above steps because the current view in Discover does not seem to update with the Session that I just saved.

If I do above steps you will get into the Saved session, and by looking at the URL you can see &searchSessionId=88e37cf3-6526-4305-asdsdasd. So I am uncertain if this saved session is used when Kibana starts generating the CSV?

If you dont do above steps you will not be in the Saved session that you saved and the URL will end with (for example) sort:!(!('@timestamp',desc))) , might be intented I dont know.

To answer your questions:

  1. 7.17
  2. 90 MB export took about 4 min
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.