Data stream update information

[Svetlo (Svetlomir Balevski)]

Hello,

I am trying to troubleshoot disappearing logs. Logs that are associated with a particular data stream have disappeared (I am not able to find under the data stream) outside of office hours. As they have disappeared in the middle of the night, no changes from our side have been done to elastic. The only clue I have is that the data stream have been updated in the time logs disappeared. Health of data stream is green.

How I can see more information for this updated (I can see Last updated date in the data Streams bot nothing more)? If not I was thinking of rollover the stream but not sure if this is going to make any difference.

Thank you.

[leandrojmp (Leandro Pereira)]

Elasticsearch does not delete anything unless it is told to by a DELETE request or using an ILM policy.

Also, you can't updated documents in the backing indices of a data streams.

Does your data stream have any ILM policy attached?

Is your cluster exposed to the public internet? Does it have security enabled?

What do you have in the logs for the master node? Any delete request will be shown in the logs of the current master node.

[Svetlo (Svetlomir Balevski)]

Thank you for the answer.

Sorry, probably my explanation was misleading (as I am new to elastic). By disappearing I meant - no new logs have appeared under this data stream (I have checked logs are sent). Old logs can still be seen under Analytics/Discover.
Yes, there is a ILM policy but it haven't been touched when the logs stopped appearing.

Is there a way to see what have been updated into the data stream?

[Svetlo (Svetlomir Balevski)]

It seems that rollover fixed the problem. I am still curious what actually have happened and will really like to see some explanation hot to tshoot such problems in the future.