This will be a bit long, but hopefully have enough information to get sorted out..
Yesterday, I set up Elasticsearch, Logstash and Kibana on a clean CentOS 7 VM (hostname kp3).
I have two other servers (Both CentOS 6.. hostnames kp1, kp2) in which I set up with filebeat to send /var/log/secure and /var/log/messages to this server.
I followed this 'howto':
Surprisingly, it worked right off the bat, then, it didn't.
I can see data being added in /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
[root@kp3 ~]# du -sh /var/lib/elasticsearch/elasticsearch/nodes/0/indices/* 452K /var/lib/elasticsearch/elasticsearch/nodes/0/indices/filebeat-2018.05.19 7.5M /var/lib/elasticsearch/elasticsearch/nodes/0/indices/filebeat-2018.10.14 5.5M /var/lib/elasticsearch/elasticsearch/nodes/0/indices/filebeat-2018.10.15 322M /var/lib/elasticsearch/elasticsearch/nodes/0/indices/filebeat-2018.10.16 21M /var/lib/elasticsearch/elasticsearch/nodes/0/indices/filebeat-2018.10.17 [root@kp3 ~]# ls -alrt /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ total 0 drwxr-xr-x. 4 elasticsearch elasticsearch 29 Oct 16 18:56 .kibana drwxr-xr-x. 8 elasticsearch elasticsearch 65 Oct 16 19:18 filebeat-2018.10.16 drwxr-xr-x. 8 elasticsearch elasticsearch 65 Oct 16 19:24 filebeat-2018.10.14 drwxr-xr-x. 8 elasticsearch elasticsearch 65 Oct 16 19:52 filebeat-2018.05.19 drwxr-xr-x. 8 elasticsearch elasticsearch 65 Oct 16 19:52 filebeat-2018.10.15 drwxr-xr-x. 8 elasticsearch elasticsearch 156 Oct 17 04:00 . drwxr-xr-x 8 elasticsearch elasticsearch 65 Oct 17 04:00 filebeat-2018.10.17 drwxr-xr-x. 4 elasticsearch elasticsearch 52 Oct 17 19:09 ..
Going further into today's filebeat directory, I can see files like _2in.cfs, etc... that have data in them and were just written to this last minute..
In Kibana, using the sample dashboards, it says to use 'filebeat-*' index pattern. I see some data from yesterday, but only for about a half hour or so and nothing aftrewards.
I even looked through my history on the server to see what I must have changed/edited in that time that it stopped working and can't figure it out.