Filebeat 5.0 unable to send/publish logs to Logstash 5.0

[VIJAYP (VIJAYA K PEDDAREDDY)]

Hi,

I am trying to monitor logs using Filebeat 5.0 from same host where ELK 5.0 is configured.

I see i/o timeout errors in /var/log/filebeat/filebeat:

This is happening on some hosts. I did verify the connectivity and it looks ok.

My filebeat.yml is as follows

filebeat.prospectors:
input_type: log
paths:

• /var/log/message
  document_type: log

filebeat.registry_file: /var/lib/filebeat/registry

output.logstash:
hosts: ["10.50.50.139:5044"]
index: filebeat
bulk_max_size: 2048

Logstash Configurations is as shown below:

I really appreciate all your help.

Thank you.
Vj.

[ruflin]

Why do you use an external IP if filebeat and LS are running on the same host? Do you see the error constantly or only from time to time?

[VIJAYP (VIJAYA K PEDDAREDDY)]

Hi,

Thanks for your reply.

You are correct. LS and filebeat are running on the same host. I changed the configuration back to localhost:

hosts: ["localhost:5044"]

NOTE: NGINX is installed in same host and it's up and running.

Now, I see the "filebeat-*" index got created in elastic search ( little progress) and able to see some logs in kibana. But this happens only after stopping and starting logstash and filebeat. Every time I restart them I see logs being pushed to Elastic search. Otherwise, nothing happening.

Here's the filebeat Log:

Here's the Logstash Log (logstash-plain.log):

Please let me know if you need any other information related to configuration.

Appreciate all your help.

Best regards
Vj.

[ruflin]

Can you please post your log files as text? Pictures are very hard to read and its not possible to search inside. If the log is too big, paste it into a gist and link it here.

[VIJAYP (VIJAYA K PEDDAREDDY)]

Hi,

Thanks for your reply.

After spending quite some time and doing multiple trail and errors I finally able to get logstash working.

In my logstash output plugin configuration I have "SNIFFING" set to true even though I don't have Elastic Cluster.

I set "sniffing => false" which fixed the issue.

Thank you
Vj.