I have been running 4 servers including ELK itself with topbeat and packetbeat installed to send data to elasticsearch. When I undergoes to look for the logs file in elasticsearch, it is just an average 10kb for a single log file of that day. It seems incredibly small. I would like to ask if it is normal or unusual?
That sounds normal. When things are working well ES doesn't log much.
So where would the data be stored in elasticsearch when the data are incoming from beats
Aren't we talking about the logs of Elasticsearch itself, typically stored in /var/log/elasticsearch?
I originally though all the incoming beats data will be stored as log data in elasticsearch, but seems that as like what you have said, the log directories only store elasticsearch log error. So I am wondering where the beats data have gone as I am wondering how much file space is needed to allocate for ELK server.
I have accidentally found that there are an index file in elasticsearch which also stored and saved data daily. I would like to know if the index directory is storing the data for beats packets.
Thanks.
Elasticsearch stores the data in its data directory. The location is configurable, but I think it's usually /var/lib/elasticsearch. ES can also report information about the size of indexes via APIs. You should run a dashboard plugin like Marvel, kopf, or ElasticHQ to track the state of your cluster.