Filebeat harvester dont started on files

version:6.2.3

i run a filebeat container ,but no harvester log

  • filebeat.yml
filebeat.config.prospectors:
  enabled: true
  path: service/*.yml
  reload.enabled: true
  reload.period: 10s
output.logstash: # 输出地址
  hosts: ["xxxxx:5044"]
  • service/demo.yml
- type: log
  paths:
    - /home/logs/sales-service.adapter.error*.log
  multiline:
    pattern: ^\d{4}
    negate: true
    match: after
  fields:
    doc_type: ssp
  • filebeay.log
2018-04-27T05:49:23.651Z	INFO	instance/beat.go:468	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2018-04-27T05:49:23.652Z	INFO	instance/beat.go:475	Beat UUID: b545f22e-938f-4783-8402-6d3434f62420
2018-04-27T05:49:23.652Z	INFO	instance/beat.go:213	Setup Beat: filebeat; Version: 6.2.3
2018-04-27T05:49:23.653Z	INFO	pipeline/module.go:76	Beat name: e9791d946c41
2018-04-27T05:49:23.654Z	INFO	elasticsearch/client.go:145	Elasticsearch url: http://116.62.6.161:9200
2018-04-27T05:49:23.657Z	INFO	elasticsearch/elasticsearch.go:154	Start monitoring endpoint init loop.
2018-04-27T05:49:23.658Z	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-04-27T05:49:23.658Z	INFO	instance/beat.go:301	filebeat start running.
2018-04-27T05:49:23.658Z	INFO	registrar/registrar.go:71	No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
2018-04-27T05:49:23.665Z	INFO	elasticsearch/elasticsearch.go:177	Stop monitoring endpoint init loop.
2018-04-27T05:49:23.665Z	INFO	elasticsearch/elasticsearch.go:183	Start monitoring metrics snapshot loop.
2018-04-27T05:49:23.680Z	INFO	registrar/registrar.go:108	Loading registrar data from /usr/share/filebeat/data/registry
2018-04-27T05:49:23.680Z	INFO	registrar/registrar.go:119	States Loaded from registrar: 0
2018-04-27T05:49:23.680Z	WARN	beater/filebeat.go:261	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-04-27T05:49:23.680Z	INFO	crawler/crawler.go:48	Loading Prospectors: 0
2018-04-27T05:49:23.681Z	WARN	[cfgwarn]	cfgfile/reload.go:77	BETA: Dynamic config reload is enabled.
2018-04-27T05:49:23.681Z	INFO	crawler/crawler.go:82	Loading and starting Prospectors completed. Enabled prospectors: 0
2018-04-27T05:49:23.681Z	INFO	cfgfile/reload.go:127	Config reloader started
2018-04-27T05:49:33.683Z	INFO	log/prospector.go:111	Configured paths: [/home/logs/sales-service.adapter.error*.log /home/logs/sales-service.adapter.warn*.log /home/logs/sales-service.adapter.info*.log]
2018-04-27T05:49:33.683Z	INFO	cfgfile/reload.go:258	Starting 1 runners ...
2018-04-27T05:49:53.660Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":29},"total":{"ticks":70,"time":88,"value":70},"user":{"ticks":50,"time":59}},"info":{"ephemeral_id":"1efac364-8834-4a29-a274-61d184477546","uptime":{"ms":30017}},"memstats":{"gc_next":4473924,"memory_alloc":3664320,"memory_total":3664320,"rss":15396864}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":2},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":1},"system":{"cpu":{"cores":2},"load":{"1":2.72,"15":2.85,"5":2.74,"norm":{"1":1.36,"15":1.425,"5":1.37}}},"xpack":{"monitoring":{"pipeline":{"clients":1,"events":{"published":2,"retry":1,"total":2},"queue":{"acked":2}}}}}}}
2018-04-27T05:50:23.660Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":35},"total":{"ticks":100,"time":109,"value":100},"user":{"ticks":70,"time":74}},"info":{"ephemeral_id":"1efac364-8834-4a29-a274-61d184477546","uptime":{"ms":60017}},"memstats":{"gc_next":4194304,"memory_alloc":2139880,"memory_total":4744608,"rss":724992}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1},"reloads":3},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":2.44,"15":2.83,"5":2.67,"norm":{"1":1.22,"15":1.415,"5":1.335}}},"xpack":{"monitoring":{"pipeline":{"events":{"published":3,"total":3},"queue":{"acked":3}}}}}}}

no harvester log

Hi,

Please try with below format :

filebeat.prospectors:

  • type: docker
    containers.ids:
    • '*'

or if you need more help about filebeat and container you can use auto-discover method below is the link for your reference.
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html

Hope this will help you!

Thanks,

oh i resolve it , i made a mistake
i change path then i can scan files.

- type: log
  paths:
    - /home/logs/*.log
  multiline:
    pattern: ^\d{4}
    negate: true
    match: after
  fields:
    doc_type: ssp

thank you

another question
should i specified registry file and mount registry file by using docker.

Hi,

I'm not sure about this as i'm not comfortable with docker.

You should read this article it may be helpful for you. Please check.

https://www.elastic.co/guide/en/beats/filebeat/6.0/running-on-docker.html

If you need any help related to filebeat configuration or logstash configuration then i can help. please let me know.

Thanks,
Harsh Bajaj

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.