Data synchronization time in elasticsearch

Elastic Stack Elasticsearch
1

Hi,

This is a query regarding data synchronization in elasticsearch.

We encountered a problem where we insert some data into elasticsearch at a regular interval. Now, immediately after data insertion, we run a search query for that particular data but the search query fails to find it even when the data is visible on kibana dashboard. The same query, when ran after a minute or so, is able to find the data.
I am not sure, but maybe this is because of data synchronization takes some time?If yes, is there any parameter that could be set/unset to reduce data synchronization time?
If no, what can we do in order to succefully be able to run search query?

Best Regards,
Akshat

2

Check refresh_interval here

3

"the search query fails to find it even when the data is visible on kibana dashboard"

But refresh_interval makes no sense when Kibana shows it. But be SURE you see it in Kibana and not just some similar data, as I've made this mistake.

No way Kibana can show the data unless it's already refreshed, so makes me wonder if you have time sync issue, the timestamps and your query may not align so your query and Kibana using slightly different windows (like from 11:23-11:44) but after a few minute you see data - we see this a lot in our system with wrong timezones, though of course that's after 1 or more hours.

If you can repeat this, suggest using a direct cURL or Devtools query instead, which will have a lot fewer parameters and added 'stuff' from Kibana - just pull last docs sorted by descending timestamp to see what ES really has, compare to your query, etc.

Of course, just for testing, you can temporarily adjust referesh_interval shorter (watch loads if you have high index rates) down to the default of 1s to see if helps, but it should not matter if Kibana or cURL really can see the data.

4

What does the query look like?

5

@ylasri @Steve_Mushero thanks for your inputs, I'll look into what you have suggested.

6

@Christian_Dahlqvist Sorry for the delayed response.
here is the query:

curl -k -u admin:kibadmin -H "Content-Type: application/json" https://elasticsearch.logging.svc.cluster.local:9200/alarms*/_search --insecure -d '
{
"query": {
"bool": {
"must": [
{
"match": {
"specificProblem":

{ "query": "1597240364" }

}
}
]
}
}
}' |jq
8

Hi @Christian_Dahlqvist
Any suggestions ?

9

What is your refresh interval for the index? How many nodes do you have in the cluster?

10

Hi @Christian_Dahlqvist,
refresh_interval is not set manually, it should take the default value for it i.e. 1 second.
We have 3 Master, 3 client and 2 data nodes, overall usually we have 13-14 nodes in the kubernetes cluster.

11

Which version of Elasticsearch are you running?

12

We are using both 7.0.1 and 7.8.0.

13

@Christian_Dahlqvist any suggestions ?

14

@Christian_Dahlqvist Any update on this?

15

No. I do not have any idea what is going on.

16

How can we find what might have caused the delay?
Can anyone from elastic community help with this?

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.