Not getting the filebeat updated in real-time on the kibana dashboard

Elastic Stack Kibana
1

Hi

I have installed filebeat on different nodes to send /var/log/messages to logstash which is in turn sends to elasticsearch and then to Kibana for display. What I do not see is that on the Kibana Discover and Dashboard, no updates happen in realtime. What I mean is that when /var/log/messages file adds messages, nothing is shown on Kibana. I have to reset the time to Now and then it shows. What I am looking for is for this data to show up in realtime with out me reloading or resetting anything.

Sorry forgot to add that I seem to have to restart filebeat for the updates to showup on kibana.

Any help will be greatly appreciated.

Thanks and Regards

Vardhan

2

Hi Sundara,

Can you check a few things to see if your filebeat data is updating in real-time in Elasticsearch? You could check this a couple of different ways.

  1. Use this in your browser http://localhost:9200/_cat/indices replacing the localhost and port if needed to reach your cluster. This will show you the indices and their doc counts. Refresh that page a few times (as often you expect you would be getting filebeat updates) and see if the counts change for that index.

  2. Use this in your browser to get the count of docs in an index.
    http://localhost:9200/logstash-2016.08.24/_count
    Again changing the host and port if needed. Check the _cat/indices above to make sure you find the right index name.

If you don't see the document count changing in Elasticsearch then of course Kibana won't see those changes either. Please let me know what you find and we can debug further.

Regards,
Lee

3

Hi Lee

Thank you very much for the information. I tried both and I keep getting that site cannot be reached. I will try some more and update you.

Regards

Vardhan

4

Hi Lee

I was able to get the indices listed. I do not see any changes after refresh. However I also see that the /var/log/messages file does not change at all, perhaps that is why filebeat on that node does not send any data. I am still looking into this. I would appreciate if you can point me to how to ensure that realtime data from filebeats is pushed through to logstash - elasticsearch - eventually to kibana.

Thanks and Regards

Vardhan

5

Hi Sundara,

You should ask a question in the Beats or Logstash discuss channels for those products. I don't know that much about those products.

But one tip I can give you is that you could insert messages into some log files with logger. For example, on RedHat/CentOS logger test will put the message test in /var/log/messages.

Another tip is that I know logstash can output to stdout so you can see the messages right in your console. I don't know exactly how to configure that.

Regards,
Lee

6

Hi Lee

thanks much for your information. Appreciate it much. I will check on what you had suggested.

Regards

Vardhan