Filebeat is not sending the logs to the logstash

Elastic Stack Beats
1

Hi All,
I am using filebeat and ELK stack. I am not getting any logs from the filebeat to Logstash. Can anyone help me. Its very urgent.
Filebeat version : 6.0.0
ELK stack(sebp/elk) : 6.0.0

Filebeat.config :--

filebeat.prospectors:
- type: log
enabled: true
paths:
- '/var/lib/docker/containers//.log'
ignore_older: 0
scan_frequency: 10s
json.message_key: log
json.keys_under_root: true
json.add_error_key: true
multiline.pattern: "^[[:space:]]+(at|\.{3})\b|^Caused by:"
multiline.negate: false
multiline.match: after
registry_file: usr/share/filebeat/data/registry
output.logstash:
hosts: ["14.315.344.174:5044"]

Filebeat logs :---

2018/07/23 07:43:48.883641 beat.go:426: INFO Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2018/07/23 07:43:48.883754 metrics.go:23: INFO Metrics logging every 30s
2018/07/23 07:43:48.885007 beat.go:433: INFO Beat UUID: 8f0f82c8-e460-46e9-8d96-a6fdc8364ca2
2018/07/23 07:43:48.885042 beat.go:192: INFO Setup Beat: filebeat; Version: 6.0.0
2018/07/23 07:43:48.885553 module.go:80: INFO Beat name: 1999258b635b
2018/07/23 07:43:48.886031 beat.go:260: INFO filebeat start running.
2018/07/23 07:43:48.886112 registrar.go:71: INFO No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
2018/07/23 07:43:48.887555 registrar.go:108: INFO Loading registrar data from /usr/share/filebeat/data/registry
2018/07/23 07:43:48.887605 registrar.go:119: INFO States Loaded from registrar: 0
2018/07/23 07:43:48.887630 filebeat.go:260: WARN Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018/07/23 07:43:48.887663 crawler.go:44: INFO Loading Prospectors: 1
2018/07/23 07:43:48.887795 registrar.go:150: INFO Starting Registrar
2018/07/23 07:43:48.888355 prospector.go:103: INFO Starting prospector of type: log; id: 2696038032251986622
2018/07/23 07:43:48.888387 crawler.go:78: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/07/23 07:44:18.884245 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=3036536 beat.memstats.memory_total=3036536 filebeat.harvester.open_files=0 filebeat.harvester.running=0 libbeat.config.module.running=0 libbeat.output.type=logstash libbeat.pipeline.clients=1 libbeat.pipeline.events.active=0 registrar.states.current=0 registrar.writes=1

2

Could you please format your using </>, so if you have configuration problems it can be detected?

3

Hi @kvch
Noémi Ványi Thanks for reply. Here my formatted code.

Filebeat.yml:-

- type: log
  enabled: true
  paths:
    - '/var/lib/docker/containers/*/*.log'
  ignore_older: 0
  scan_frequency: 10s
  json.message_key: log
  json.keys_under_root: true
  json.add_error_key: true
  multiline.pattern: "^[[:space:]]+(at|\\.{3})\\b|^Caused by:"
  multiline.negate: false
  multiline.match: after
  registry_file: usr/share/filebeat/data/registry
4

Hi,
I am facing simllar problem, it was working completely fine till last night, since then filebeat is not sending any input to logstash.

5

the only recent change which i did was taking a sample file and created index pattern to verify the multiline pattern settings, upon reading the messages, i deleted that index to create the actual one with production files. SInce then i have been facing this problem.

6

Could you please both share debug logs?

@siva1 What changed before Filebeat failed to send logs?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.