Setup ELK 5.0 but Metribeat Not Working

Elastic Stack Beats
1

Hello,

I have just installed a new ELK 5.0 Server on Ubuntu 15.04. Kibana/Elasticsearch/Logstash 5.0
I am able to login to Kibana and for a few minutes I saw some logs coming in but then they stopped. At the moment it's only really getting it's own logs.

I installed topbeat/filebeat/metricbeat and configured it to send to Elasticsearch and Logstash. Perhaps I made a mistake in the topbeat/filebeat/metribeat.yml files as when I run this:

$ curl -XGET 'http://localhost:9200/metricbeat-*/_search?pretty'

I get 0 hits:

{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : 0.0,
"hits" : [ ]
}
}

Metribeat is running just fine but I am not sure if I configured it correctly. Also since I am setting the output to Elasticsearch do I don't need to also set the output to Logstash correct? This is a stand alone server that has all ELK installed. I imported the Dashboards which don't works 'cause there is no logs.
I keep seeing this in /var/log/metricbeat

2016-10-26T18:08:37-07:00 ERR Connecting error publishing events (retrying): Get http://kibana.domain.com:5044: dial tcp 127.0.1.1:5044: getsockopt: connection refused

Also Logstash is installed but I get these errors in /var/log/logstash:

[2016-10-26T17:47:09,514][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}

Please help. Thank You

2

You normally only want to enable either the Elasticsearch or the Logstash output, but not both. So for Metricbeat, just enable the elasticsearch output to localhost:9200 to get started. If you running ES on the same machine then you should be able to just use the default Metricbeat configuration.

3

hey Andrew

I removed filebeat and topbeat so that I can just make metricbeat work, I dpkg it, it shows in /etc/metricbeat/ and I edited the metricbeat.yml to point to itself. I then start metricbeat which starts ok, no errors and then I run this and I get no hits so kibana has nothing

elk@kibana:/etc/metricbeat$ sudo service metricbeat status

  • metricbeat.service - LSB: Sends metrics to Elasticsearch.
    Loaded: loaded (/etc/init.d/metricbeat)
    Active: active (exited) since Thu 2016-10-27 10:17:49 PDT; 32s ago
    Docs: man:systemd-sysv-generator(8)
    Process: 24091 ExecStart=/etc/init.d/metricbeat start (code=exited, status=0/SUCCESS)
    Main PID: 23918 (code=exited, status=203/EXEC)
    CGroup: /system.slice/metricbeat.service

Oct 27 10:17:49 kibana systemd[1]: Starting LSB: Sends metrics to Elasticsearch....
Oct 27 10:17:49 kibana systemd[1]: Started LSB: Sends metrics to Elasticsearch..

elk@kibana:/etc/metricbeat$ curl -XDELETE 'http://localhost:9200/metricbeat-*'
{"acknowledged":true

}

elk@kibana:/etc/metricbeat$ curl -XGET 'http://localhost:9200/metricbeat-*/_search?pretty'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : 0.0,
"hits" :
}
}
elk@kibana:/etc/metricbeat$

4

Oh and I tried setting up metricbeat on another CentOS box, pointing to my ELK server but I keep getting a"Connection refused" when I try to to test Metricbeat so this might explain also why I can't get any metricbeats to the ELK server itself.

[cockpit@cockpit sysconfig]$ curl -XGET 'http://kibana.dhxmedia.com:9200/metricbeat-*/_search?pretty'

curl: (7) Failed connect to kibana.dhxmedia.com:9200; Connection refused

I already tried stopping the firewall in the ELK server but that didn't help

5

Hey Andrew,

I removed metricbeat package as well, I re-configured it again and this time it worked : S
Maybe when I tried to configure all the other beats something happened? Ok so the ELK server itself is working just fine but.... now the other server(s) can't send logs to my ELK server. I keep seeing that error: "curl: (7) Failed to connect to kibana.domain.com:9200; Connection refused

Again, the ELK server (kibana) has the Firewall Disabled. I'm able to ping the ELK server from that other box but can't send metric logs to it. Any idea?

6

Can you telnet to kibana.domain.com:9200 from the host running the Beat? What does netstat -an | grep 9200 show on the server running ES?

7

Hey Andrew,

This is the result:

elk@kibana:/var/lib/elasticsearch$ netstat -an |grep 9200

tcp 0 0 127.0.0.1:51848 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:53116 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:51948 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:51850 127.0.0.1:9200 ESTABLISHED
tcp6 0 0 127.0.0.1:9200 :::* LISTEN
tcp6 0 0 ::1:9200 :::* LISTEN
tcp6 0 0 127.0.0.1:9200 127.0.0.1:51948 ESTABLISHED
tcp6 0 0 127.0.0.1:9200 127.0.0.1:51850 ESTABLISHED
tcp6 0 0 127.0.0.1:9200 127.0.0.1:51848 ESTABLISHED
tcp6 0 0 127.0.0.1:9200 127.0.0.1:53116 ESTABLISHED

8

elk@kibana:/var/lib/elasticsearch$ sudo ufw status
Status: active

To Action From

5601/tcp ALLOW Anywhere
9200/tcp ALLOW Anywhere
80/tcp ALLOW Anywhere
5044/tcp ALLOW Anywhere
5601/tcp (v6) ALLOW Anywhere (v6)
9200/tcp (v6) ALLOW Anywhere (v6)
80/tcp (v6) ALLOW Anywhere (v6)
5044/tcp (v6) ALLOW Anywhere (v6)

9

by default shouldn't the elasticsearch.yml configure "network"? my elasticsearch.yml file shows all commented in the "Network" field:

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

#network.host: 192.168.0.1

Set a custom port for HTTP:

#http.port: 9200

10

Based on netstat, your ES server is only listening on localhost so you won't be able to connect to 9200 from a remote host.

You need to configure Elasticsearch to listen on something other than loopback. Set the network.host option in the /etc/elasticsearch/elasticsearch.yml file. This will trigger the bootstrap checks if you are using 5.0.

11

Hey Andrew,

Thanks! I appreciate your help, I originally changed it to network.host IP.of.ELK but that just basically crashed ElasticSearch all together. Changed it to 0.0.0.0 and it worked like a charm.

Thanks for helping me out Andrew! Couldn't have done it without you

Joey

12

This topic was automatically closed after 21 days. New replies are no longer allowed.