Filebeat doesn't show in kibana monitoring

Elastic Stack Beats
1

I have two filebeat running.
But I just can see one filebeat instance in kibana monitoring.
Both filebeat are sending monitor metric:

{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":17},"total":{"ticks":40,"time":50,"value":40},"user":{"ticks":30,"time":33}},"info":{"ephemeral_id":"1443a781-a8c1-41ef-ab35-18c610562909","uptime":{"ms":30010}},"memstats":{"gc_next":4194304,"memory_alloc":2653984,"memory_total":5832112,"rss":16232448}},"filebeat":{"events":{"added":9,"done":9},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":6,"batches":6,"total":6},"read":{"bytes":3551},"type":"elasticsearch","write":{"bytes":6002}},"pipeline":{"clients":1,"events":{"active":0,"filtered":3,"published":6,"retry":4,"total":9},"queue":{"acked":6}}},"registrar":{"states":{"current":2,"update":9},"writes":9},"system":{"cpu":{"cores":16},"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.0031,"5":0.0006}}},"xpack":{"monitoring":{"pipeline":{"clients":1,"events":{"published":2,"retry":4,"total":2},"queue":{"acked":2}}}}}}}

Beat name from filebeat log:
Beat name: instance-n8ze37kj
Beat name: instance-m33rc65a

I just can see instance-m33rc65a in kibana monitoring.
Bost filebeat.yml is the same.

Any idea to find the problem? thanks!

2

Have you checked filebeat logs and configs correctly forwarding to the same Elasticsearch cluster?

Can you share logs and configuration of 'failing' filebeat?

3

Both filebeat.yml is the same:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /root/webappServer/log/webapp*.log
  json.keys_under_root: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.template.enabled: false
setup.kibana:
output.elasticsearch:
  hosts: ["192.168.0.15:9200", "192.168.0.13:9200"]
  loadbalance: true
  worker: 2
  username: "elastic"
  password: "password"
  index: "webapp-web-%{+yyyy.MM.dd}"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch:
4

'failing' filebeat log:

2018-04-10T19:41:48.257+0800    INFO    instance/beat.go:468    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-04-10T19:41:48.258+0800    INFO    instance/beat.go:475    Beat UUID: 53896df5-458c-4e6e-8fcc-97d8ec5138b1
2018-04-10T19:41:48.258+0800    INFO    instance/beat.go:213    Setup Beat: filebeat; Version: 6.2.3
2018-04-10T19:41:48.258+0800    INFO    elasticsearch/client.go:145     Elasticsearch url: http://192.168.0.15:9200
2018-04-10T19:41:48.259+0800    INFO    elasticsearch/client.go:145     Elasticsearch url: http://192.168.0.15:9200
2018-04-10T19:41:48.259+0800    INFO    elasticsearch/client.go:145     Elasticsearch url: http://192.168.0.13:9200
2018-04-10T19:41:48.259+0800    INFO    elasticsearch/client.go:145     Elasticsearch url: http://192.168.0.13:9200
2018-04-10T19:41:48.259+0800    INFO    pipeline/module.go:76   Beat name: instance-n8ze37kj
2018-04-10T19:41:48.259+0800    INFO    elasticsearch/client.go:145     Elasticsearch url: http://192.168.0.15:9200
2018-04-10T19:41:48.259+0800    INFO    elasticsearch/client.go:145     Elasticsearch url: http://192.168.0.15:9200
2018-04-10T19:41:48.259+0800    INFO    elasticsearch/client.go:145     Elasticsearch url: http://192.168.0.13:9200
2018-04-10T19:41:48.260+0800    INFO    elasticsearch/client.go:145     Elasticsearch url: http://192.168.0.13:9200
2018-04-10T19:41:48.260+0800    INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-04-10T19:41:48.260+0800    INFO    elasticsearch/elasticsearch.go:154      Start monitoring endpoint init loop.
2018-04-10T19:41:48.260+0800    INFO    instance/beat.go:301    filebeat start running.
2018-04-10T19:41:48.260+0800    INFO    registrar/registrar.go:108      Loading registrar data from /var/lib/filebeat/registry
2018-04-10T19:41:48.262+0800    INFO    registrar/registrar.go:119      States Loaded from registrar: 2
2018-04-10T19:41:48.262+0800    INFO    crawler/crawler.go:48   Loading Prospectors: 1
2018-04-10T19:41:48.263+0800    INFO    log/prospector.go:111   Configured paths: [/root/webappServer/log/webapp*.log]
2018-04-10T19:41:48.263+0800    INFO    crawler/crawler.go:82   Loading and starting Prospectors completed. Enabled prospectors: 1
2018-04-10T19:41:48.263+0800    INFO    cfgfile/reload.go:127   Config reloader started
2018-04-10T19:41:48.263+0800    INFO    cfgfile/reload.go:219   Loading of config files completed.
2018-04-10T19:41:48.263+0800    INFO    log/harvester.go:216    Harvester started for file: /root/webappServer/log/webapp.log
2018-04-10T19:41:48.264+0800    INFO    elasticsearch/elasticsearch.go:177      Stop monitoring endpoint init loop.
2018-04-10T19:41:48.265+0800    INFO    elasticsearch/elasticsearch.go:183      Start monitoring metrics snapshot loop.
2018-04-10T19:41:49.266+0800    INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.2.3
2018-04-10T19:41:49.266+0800    INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.2.3
2018-04-10T19:41:49.268+0800    INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.2.3
2018-04-10T19:41:49.268+0800    INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.2.3
2018-04-10T19:42:18.261+0800    INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":17},"total":{"ticks":40,"time":50,"value":40},"user":{"ticks":30,"time":33}},"info":{"ephemeral_id":"1443a781-a8c1-41ef-ab35-18c610562909","uptime":{"ms":30010}},"memstats":{"gc_next":4194304,"memory_alloc":2653984,"memory_total":5832112,"rss":16232448}},"filebeat":{"events":{"added":9,"done":9},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":6,"batches":6,"total":6},"read":{"bytes":3551},"type":"elasticsearch","write":{"bytes":6002}},"pipeline":{"clients":1,"events":{"active":0,"filtered":3,"published":6,"retry":4,"total":9},"queue":{"acked":6}}},"registrar":{"states":{"current":2,"update":9},"writes":9},"system":{"cpu":{"cores":16},"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.0031,"5":0.0006}}},"xpack":{"monitoring":{"pipeline":{"clients":1,"events":{"published":2,"retry":4,"total":2},"queue":{"acked":2}}}}}}}
5

Is the log complete or is there some more work being logged afterwards?

Check the registry file. The registry file contains the last processed file offset.

From logs I can tell:

  • 1 file is being processed
  • some events have bee published
  • the publishing queue is empty, as all events have been processed
  • registry file has 2 been updated 9 times already.
  • no more logs are picked up from the file: is file still actively written to? compare registry file with file size.
6 
2018-04-11T10:27:18.261+0800	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":13960,"time":13967},"total":{"ticks":43080,"time":43095,"value":43080},"user":{"ticks":29120,"time":29128}},"info":{"ephemeral_id":"1443a781-a8c1-41ef-ab35-18c610562909","uptime":{"ms":53130010}},"memstats":{"gc_next":4194304,"memory_alloc":3666712,"memory_total":4462814688}},"filebeat":{"events":{"active":1,"added":10,"done":9},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":9,"batches":7,"total":9},"read":{"bytes":2351},"write":{"bytes":7086}},"pipeline":{"clients":1,"events":{"active":1,"published":10,"total":10},"queue":{"acked":9}}},"registrar":{"states":{"current":2,"update":9},"writes":7},"system":{"load":{"1":0.16,"15":0.06,"5":0.05,"norm":{"1":0.01,"15":0.0038,"5":0.0031}}},"xpack":{"monitoring":{"pipeline":{"events":{"published":3,"total":3},"queue":{"acked":3}}}}}}}
2018-04-11T10:27:48.261+0800	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":13970,"time":13976},"total":{"ticks":43110,"time":43122,"value":43110},"user":{"ticks":29140,"time":29146}},"info":{"ephemeral_id":"1443a781-a8c1-41ef-ab35-18c610562909","uptime":{"ms":53160010}},"memstats":{"gc_next":4201152,"memory_alloc":2181408,"memory_total":4465319328}},"filebeat":{"events":{"added":7,"done":7},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":7,"batches":6,"total":7},"read":{"bytes":1996},"write":{"bytes":5734}},"pipeline":{"clients":1,"events":{"active":1,"published":7,"total":7},"queue":{"acked":7}}},"registrar":{"states":{"current":2,"update":7},"writes":6},"system":{"load":{"1":0.09,"15":0.05,"5":0.04,"norm":{"1":0.0056,"15":0.0031,"5":0.0025}}},"xpack":{"monitoring":{"pipeline":{"events":{"published":3,"total":3},"queue":{"acked":3}}}}}}}
2018-04-11T10:28:18.261+0800	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":13980,"time":13984},"total":{"ticks":43130,"time":43143,"value":43130},"user":{"ticks":29150,"time":29159}},"info":{"ephemeral_id":"1443a781-a8c1-41ef-ab35-18c610562909","uptime":{"ms":53190010}},"memstats":{"gc_next":4194304,"memory_alloc":2958944,"memory_total":4467846896}},"filebeat":{"events":{"added":8,"done":8},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":8,"batches":6,"total":8},"read":{"bytes":2021},"write":{"bytes":6272}},"pipeline":{"clients":1,"events":{"active":1,"published":8,"total":8},"queue":{"acked":8}}},"registrar":{"states":{"current":2,"update":8},"writes":6},"system":{"load":{"1":0.06,"15":0.05,"5":0.04,"norm":{"1":0.0038,"15":0.0031,"5":0.0025}}},"xpack":{"monitoring":{"pipeline":{"events":{"published":3,"total":3},"queue":{"acked":3}}}}}}}

/var/log/filebeat/filebeat logged monitoring metric works fine.

Where is the registry file?

7

Events are being send and ACKed. Try searching your events by beat.hostname. e.g.

curl -XGET -H "Content-Type: application/json" "http://elastic:password@192.168.0.15:9200/webapp-web-*/_search?q=beat.hostname=${hostname}"

Using a term filter you can see how many hostnames you have indexed.

8

Thanks for quickly reply.

instance-n8ze37kj events is correctly import to es.

1523454182568
1523454182568.jpg920×404 50 KB

The problem is instance-n8ze37kj doesn't show in kibana monitoring beats.

image
image.png1528×594 46.8 KB

9

Oh, it's the monitoring data not showing data up.

From logs I can see the reporter is being started. Can you run filebeat with debugging enabled + "monitoring" and "elasticsearch" debug selector. Either update the logger setting in the config file or run filebeat with -d "monitoring,elasticsearch".

Also check for presence of .monitoring indices.

10

I explore GET /.monitoring-beats-6-2018.04.12/_search, and found both instance-m33rc65a and instance-n8ze37kj events.

11

That's interesting. Can you check the event timestamps from both hosts being in the same range?

12

1523531236252
1523531236252.jpg2094×1222 222 KB

I use kibana discover to query .monitoring-beats-*.

13

I found the problem. I copy the filebeat from instance-m33rc65a to instance-n8ze37kj.
They have the same data/meta.json (uuid).

I delete data/meta.json, and restart instance-n8ze37kj filebeat, then the kibana monitoring show all filebeats.

1 Like
14

Oh. Yeah, never copy data. The data directory contains all data local to an instance.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.