Data Drops

Elastic Stack Elasticsearch
1

We are seeing data drops intermittently while being written into Elastic. However, when we check for bulk exceptions it doesn't correlate to the data drop times.

is there any other place or logs to look for? any help to determine and analyze this situation would be helpful.

2

Seeing them how?
How is data being sent to Elasticsearch?
Do you have Monitoring enabled?

3

we do not see anything in the logs. However we do notice some data being lost.
We have monitoring enabled and it also doesn't show any signs of data loss. Which is strange, hence would be good to know if anyway to get this information.

4

It would help if you provided some additional detail on how you are getting your data into Elasticsearch. Without this it is impossible to speculate about what is happening.

5

data sent through bulk upserts from our upstream system.

6

Are you checking the bulk request response for any errors and retrying if you encounter any? How do you identify that data is being dropped?

7

there is no issues in the response to bulk request. However when we see the chart we see gaps making us believe the data being dropped. However none of the logs shows otherwise. Hence not sure how to confirm there are no issues on the Elastic that could cause data drop

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.