# Losing logs in ELK
**URL:** https://discuss.elastic.co/t/losing-logs-in-elk/275356
**Category:** Elasticsearch
**Created:** [June 8, 2021, 11:02pm UTC](https://discuss.elastic.co/t/losing-logs-in-elk/275356 "2021-06-08T23:02:59Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@devopscanada](https://discuss.elastic.co/u/devopscanada)
#### Post date: [June 8, 2021, 11:02pm UTC](https://discuss.elastic.co/t/losing-logs-in-elk/275356/1 "2021-06-08T23:02:59Z")
- Setup: Kubernetes cluster with filebeat-7.12.0 -\> logstash-7.9. -\> elasticsearch-7.9.1 -\> kibana-7.9.1
- Problem: we lose many logs often and sometimes in a massive way, as in we stop seeing logs from one or more pods (confirmed fact from other logging/alert mechanisms), we don't know where/how. There is no particular event that we can relate to logs from some k8s pods not starting to appear in ES; it's not tied to a particular server of particular type of pod or anything we can isolate or reproduce.
- Settings and resource utilization:
- Nothing I can see indicates a direct capacity/resource issue in ES (eg `GET /_nodes/hot_threads` etc) or in any other component.
- I don't know if the issue is in filebeat not sending logs, logstash dropping them or somehow in ES. Looking at error logs:
- I've search all the errors and I haven't found anything actionable or anything that matches our situation. A weak hypothesis is that if we get a lot of quick log rotations, we may lose some logs because k8s moves rotated logs outside of reach of filebeat. Still this wouldn't explain a massive number of logs missing.
- As for the Logstash dropping logs, not sure how to confirm one way or another; I don't have errors about that and querying the stats I get almost the same amount of events in/filtered/out and if it was dropping logs I'd expect the numbers for in and out to be different.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 9, 2021, 12:26am UTC](https://discuss.elastic.co/t/losing-logs-in-elk/275356/2 "2021-06-09T00:26:12Z")
> [@devopscanada](#):
>
> ```auto
> Harvester could not be started on new file: ... Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: file info is not identical with opened file. Aborting harvesting and retrying file later again
>
> ```
These seem pretty relevant.
Can you enable debug on one Filebeat pod and provide more logging?
---
### Author:  [@devopscanada](https://discuss.elastic.co/u/devopscanada)
#### Post date: [June 9, 2021, 2:08am UTC](https://discuss.elastic.co/t/losing-logs-in-elk/275356/3 "2021-06-09T02:08:24Z")
Thanks (and sorry probably this would be more for the filebeat forum), I can try putting one filebeat in debug mode, although in K8s this is a bit of a PITA since they are DaemonSets (the same filebeat config running in each node), so I'll have to do a workaround like using taints in one node so that filebeat DaemonSet doesn't put a pod in it and separately and manually putting a filebeat pod in this host with debugging enabled.
I do have the filebeat pods with the stats endpoint enabled, so I can manually look at metrics like for ex for harverster in one pod:
```auto
kubectl exec filebeat-filebeat-fslqv -- curl -s -XGET 'localhost:5066/stats?pretty' |jq .filebeat.harvester
{
"closed": 3,
"open_files": 10,
"running": 10,
"skipped": 0,
"started": 13
}
```
What are the guidelines for those metrics to be "good"? I imagine something like:
open\_files = running
running + closed = started
Or what other metrics can I focus on that would indicate a problem (specifically with missing files/logs), like maybe registrar.write.fail \> 0 , pipeline.events.failed \> 0 or output.read|write.errors \> 0 ?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 7, 2021, 2:08am UTC](https://discuss.elastic.co/t/losing-logs-in-elk/275356/4 "2021-07-07T02:08:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.