- Pipeline is file on disk => Filebeat => Logstash => Elasticsearch .Further now my disk is full and the size of the file that was storing the logs till now is not changing . But still Logstash is parsing logs and those are getting pushed to ES . Further one more reason why I think ES is not the bottleneck is that when ever I search "*" in kibana after few seconds I can see change in number of hits . This indicates logs are getting pushed to ES by Logstash normally .But the other server logs are pretty behind in the queue , probably thats why its taking long for them to get pushed in the ES . What say ?
- Now after my disk space has gone full for the file my RAM of ELK machine is 2033/3764MB and is fluctuating upto 2164/3764MB . So do I still need to fear regarding use up of RAM ??
Further one more reason why I think ES is not the bottleneck is that when ever I search "*" in kibana after few seconds I can see change in number of hits .
That doesn't prove anything.
This indicates logs are getting pushed to ES by Logstash normally .
No, but it indicates that the pipeline hasn't stalled completely.
Now after my disk space has gone full for the file my RAM of ELK machine is 2033/3764MB and is fluctuating upto 2164/3764MB . So do I still need to fear regarding use up of RAM ??
I don't know how to respond. ES uses RAM, and the more data you push into ES the more RAM you need. You should set the ES heap size to about half the RAM, and if the heap utilization is constantly high you should probably consider expanding the cluster (with additional nodes and/or more powerful nodes).
Over and out.
@magnusbaeck now as the file to which the bash script was writing contents has taken all the disk space now my memory is not increasing that way . Any reason ???