Yes, that's the problem. Why isn't Filebeat able to connect to localhost:9200? Or should it be connecting to something else?
I think I make some mistake.
Filebeat is installed on a remote server, so it's normal that there is nothing on localhost.
I need to configure filebeat to listen on my remote server logstash?
Ok my mistake. When I reconfigure Filebeat I have deleted my old file
Will make modification and will see if all is ok
Hello,
Modification has been made. Now my elasticsearch (on a remote server) is listening on the good interface.
Then I configure kibana to fetch the good server for Elasticsearch.
Also, modification has been made on filebeat. Here is the a part of the log file :
2016-01-26T10:48:33+01:00 DBG %!s(int=0) events out of %!s(int=200) events sent to logstash. Continue sending ...
2016-01-26T10:48:33+01:00 INFO Error publishing events (retrying): EOF
2016-01-26T10:48:33+01:00 INFO send fail
2016-01-26T10:48:33+01:00 INFO backoff retry: 1s
2016-01-26T10:48:34+01:00 DBG Try to publish %!s(int=200) events to logstash with window size %!s(int=10)
2016-01-26T10:48:34+01:00 DBG %!s(int=0) events out of %!s(int=200) events sent to logstash. Continue sending ...
2016-01-26T10:48:34+01:00 INFO Error publishing events (retrying): EOF
2016-01-26T10:48:34+01:00 INFO send fail
2016-01-26T10:48:34+01:00 INFO backoff retry: 2s
2016-01-26T10:48:38+01:00 DBG Start next scan
2016-01-26T10:48:38+01:00 DBG scan path /var/log/syslog
2016-01-26T10:48:38+01:00 DBG Check file for harvesting: /var/log/syslog
2016-01-26T10:48:38+01:00 DBG Update existing file for harvesting: /var/log/syslog
2016-01-26T10:48:38+01:00 DBG Not harvesting, file didn't change: /var/log/syslog
So I think for now Filebeat is Ok ?
Do you have info avout ELK ? Kibana is my web interface to see informations from logstash ? Logstash is the applicative that collect info from log ? And what about Elasticsearch ?
Thanks
So I think for now Filebeat is Ok ?
The log indicates that there's an error, at least a temporary one. Does the data eventually reach Logstash and Elasticsearch?
Do you have info avout ELK ? Kibana is my web interface to see informations from logstash ?
Yes.
Logstash is the applicative that collect info from log ?
Yes.
And what about Elasticsearch ?
ES stores the data and performs the queries.
Do you know How can I see if datas have reached Logstash or ES ?
From what I see in Kibana, some datas have been reached as I have some informations on Kibana but now no new data
Are there additional indications in the log that Filebeat succeeds or fails to send events to Logstash? As I said earlier, the log snippet you posted earlier that Filebeat had problems but that it would try again. Which version of Filebeat is this?
HEre is the version of Filebeat :
root@crinforecettevm01:/var/log/mybeat# filebeat --version
filebeat version 1.0.1 (amd64)
root@crinforecettevm01:/var/log/mybeat#
When looking into log of filebeat, here is an error :
2016-01-26T15:07:48+01:00 INFO Connecting error publishing events (retrying): read tcp 172.31.0.168:44959->37.59.235.27:5044: i/o timeout
May be I need to configure something on my remote server ( logstash ?)
Just to be sure, here is my infra :
Server 1 : Elasticsearch + Logstash + Kibana
Server 2 : Filebeat
Filebeat send info on my remote server.
May be I need filebeat on my remote server ? As in the error below, I see
read tcp 172.31.0.168:44959->37.59.235.27:5044: i/o timeout
But nothing running on port 5044 !
37.59.235.27 is server 1
172.31.0.168 is server 2
2016-01-26T15:07:48+01:00 INFO Connecting error publishing events (retrying): read tcp 172.31.0.168:44959->37.59.235.27:5044: i/o timeout
This is a network connectivity issue. Is there e.g. a firewall blocking access from your Filebeat host (172.31.0.168) to your Logstash host (37.59.235.27)?
Ok thanks. I will see this with the right service.
Will let you know ASAP
Thank you so much for all your help
Just a question, on the remote server nothing is running on port 5044. Is it normal?
Juste check my conf and it seems it's ok on logstash side.
And no issue from my firewall as when making "telnet 37.59.235.27 5044" all is ok
May be I need to configure my logstash a little bit better ?
firewall/nat/whatever having timeout on connections? Sometimes firewalls silently close connections without sending FIN which kinda breaks communication.
Hello
Here is the error on filebeat :
2016-01-28T13:19:03+01:00 INFO Error publishing events (retrying): EOF
I make the necessary on firewall side