Filebeat monitoring not working

[wiliamauc85]

Hi,

I've been trying to set up filebeat monitoring and going through the docs, my filebeats.yml looks like this

- type: log
  enabled: true
  paths:
    - /var/log/secure
output.logstash:
  hosts: ["logstash:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/config/certs/ca.crt"]
xpack.monitoring:
  enabled: true
  elasticsearch:
    hosts: ["https://elastic1:9200"]
    username: beats_system
    password: password

When I start filebeats, i get this in the filebeats log
2019-04-10T15:51:13.624+0100 INFO [monitoring] elasticsearch/elasticsearch.go:234 Failed to connect to Elastic X-Pack Monitoring. Either Elasticsearch X-Pack monitoring is not enabled or Elasticsearch is not available. Will keep retrying.

Sample of the log is below

[root@FB certs]# tail -f /var/log/filebeat/filebeat
2019-04-10T15:51:13.592+0100    INFO    crawler/crawler.go:72   Loading Inputs: 1
2019-04-10T15:51:13.593+0100    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2019-04-10T15:51:13.594+0100    INFO    log/input.go:138        Configured paths: [/var/log/secure]
2019-04-10T15:51:13.594+0100    INFO    input/input.go:114      Starting input of type: log; ID: 4595524358270348772
2019-04-10T15:51:13.594+0100    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2019-04-10T15:51:13.594+0100    INFO    log/harvester.go:255    Harvester started for file: /var/log/secure
2019-04-10T15:51:13.594+0100    INFO    cfgfile/reload.go:150   Config reloader started
2019-04-10T15:51:13.594+0100    INFO    cfgfile/reload.go:205   Loading of config files completed.
**2019-04-10T15:51:13.624+0100    INFO    [monitoring]    elasticsearch/elasticsearch.go:234      Failed to connect to Elastic X-Pack Monitoring. Either Elasticsearch X-Pack monitoring is not enabled or Elasticsearch is not available. Will keep retrying.**
2019-04-10T15:51:14.595+0100    INFO    pipeline/output.go:95   Connecting to backoff(async(tcp://logstash:5044))
2019-04-10T15:51:14.641+0100    INFO    pipeline/output.go:105  Connection to backoff(async(tcp://logstash:5044)) established

So from the logs, my logstash output is working, but I but I can't see any stats/metrics for filebeats in Kibana>Monitoring

Is there something I'm missing?

[BKG]

beats_system appears to be a role, not a user... unless your user is called beats_system too?

[wiliamauc85]

Yup, it's a user too

{
  "beats_system" : {
    "username" : "beats_system",
    "roles" : [
      "beats_system"
    ],
    "full_name" : null,
    "email" : null,
    "metadata" : {
      "_reserved" : true
    },
    "enabled" : true
  }
}

I actually got it working, I was missing

ssl.certificate_authorities: ["/etc/filebeat/config/certs/ca.crt"]

added to my filebeats.yml

 xpack.monitoring:
   enabled: true
   elasticsearch:
     hosts: ["https://elastic1:9200"]
     ssl.certificate_authorities: ["/etc/filebeat/config/certs/ca.crt"]
     username: beats_system
     password: changed

Can now see beats under monitoring