Does Elastic Search store all log data sent from File Beat?


(Sameer Panicker) #1

I have configured a log path inside FB configuration which has data around 10GB. This is my server 1.

I have ES installed on my Server 2. FB is configured to send data to ES from Server 1 to Server 2.

ES is used to store data, will it actually pull/copy all 10GB of data onto my Server 2 OR it wont physically copy/pull 10GB of files from Server 1, rather it will just use those log files present on Server 1 to filter.

How does it actually works?


(Mark Walkom) #2

ES will store anything that FB sends to it.
So as long as you aren't filtering on the FB end, then it'll be everything you point it at.


(Sameer Panicker) #3

By default what filter value is set on FB. Where can i see those filter? Bcoz i have not configured any specific filter from my end.

Also, i see the same disk size on my server 2 and all my logs are also getting searched. How is that occurring ?


(Mark Walkom) #4

None.

Can you elaborate on this a bit more, I am not following what you are asking.


(Sameer Panicker) #5

The server disk space should go low when fb sends all the 10gb of data to my elastic search server. Currently, i dont see any degradation in my servers disk space.


(Mark Walkom) #6

Filebeat does not delete the files once it has read them, that is up to you to manage.


(Sameer Panicker) #7

May be i m missing a configuration here. Can u tell me where can i see all the stored by ES ?


(Mark Walkom) #8

Are you using Kibana? That's the most visual way.
Otherwise maybe something like _cat/indices.


(Sameer Panicker) #9

Yes i m using kibana.


(Mark Walkom) #10

Using the Discover page in Kibana will show you all the data.


(system) #11