Filebeat issue in configuration file

amritesh_mishra · September 3, 2018, 11:43am

We have created this config file but file beat is sending only one file at a time it is not sending the second path file which i have put in the filebeat config file.

this is the config
filebeat:
prospectors:

  paths:
    - /talend/6.1.1/runtime/log/tesb.log

  input_type: log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

paths:
- /home/talendpp/runtime/wrapper.log
input_type: log
fileds: {log_type: wrapperpreprodruntime}
multiline.pattern: '^\d+'
multiline.negate: true
multiline.match: after

output:
logstash:
hosts: ["172.18.7.16:5044"]
index: filebeat

Please help on this is very urgent.

kvch · September 3, 2018, 12:36pm

Could you please format you entire config using </>?

amritesh_mishra · September 3, 2018, 12:38pm

<
filebeat:
prospectors:

  paths:
    - /talend/6.1.1/runtime/log/tesb.log

  input_type: log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

paths:
- /home/talendpp/runtime/wrapper.log
input_type: log
fileds: {log_type: wrapperpreprodruntime}
multiline.pattern: '^\d+'
multiline.negate: true
multiline.match: after

output:
logstash:
hosts: ["172.18.7.16:5044"]
index: filebeat
/>

kvch · September 3, 2018, 2:08pm

Could you try again? You might have a whitespace error and this way it's impossible to debug it.

You could try adding ``` to the beginning and to the end of your config file.

amritesh_mishra · September 3, 2018, 2:44pm

<```
filebeat:
prospectors:

paths:
- /talend/6.1.1/runtime/log/tesb.log
  input_type: log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after
paths:
- /home/talendpp/runtime/wrapper.loginput_type: log
  fileds: {log_type: wrapperpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

output:
logstash:
hosts: ["172.18.7.16:5044"]
index: filebeat

kvch · September 3, 2018, 5:31pm

This config should work:

filebeat.prospectors:
- input_type: log
  paths:
   - /talend/6.1.1/runtime/log/tesb.log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
   - /home/talendpp/runtime/wrapper.log
  fileds: {log_type: wrapperpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["172.18.7.16:5044"]
  index: filebeat

amritesh_mishra · September 4, 2018, 7:20am

It is sending only the first path which you mentioned not the second one "- /talend/6.1.1/runtime/log/tesb.log"

This is the path which is being send

kvch · September 4, 2018, 10:14am

Could you please share the debug logs? (Output of filebeat -e -d "*")

amritesh_mishra · September 4, 2018, 10:24am

2018-09-04T11:16:13.719+0100 2018-09-04T11:16:13.728+0100 2018-09-04T11:16:13.728+0100 2018-09-04T11:16:13.729+0100 2018-09-04T11:16:13.730+0100 2018-09-04T11:16:13.730+0100 2018-09-04T11:16:13.735+0100 2018-09-04T11:16:13.736+0100 2018-09-04T11:16:13.736+0100 2018-09-04T11:16:13.736+0100 2018-09-04T11:16:13.736+0100 2018-09-04T11:16:13.736+0100 2018-09-04T11:16:13.736+0100 2018-09-04T11:16:13.739+0100 2018-09-04T11:16:13.739+0100 2018-09-04T11:16:13.739+0100 2018-09-04T11:16:13.740+0100 2018-09-04T11:16:15.040+0100 2018-09-04T11:16:17.042+0100 2018-09-04T11:16:21.058+0100 2018-09-04T11:16:29.059+0100 2018-09-04T11:16:43.732+0100 2018-09-04T11:16:45.061+0100 2018-09-04T11:17:13.732+0100 2018-09-04T11:17:17.098+0100 2018-09-04T11:17:43.731+0100 2018-09-04T11:18:13.732+0100 2018-09-04T11:18:17.100+0100 INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
INFO instance/beat.go:475 Beat UUID: 1a45d01b-b76b-4f6e-8642-61f11e6e7bed
INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.3
INFO pipeline/module.go:76 Beat name: ukmr0xltmi002
INFO instance/beat.go:301 filebeat start running.
INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
INFO registrar/registrar.go:108 Loading registrar data from /var/lib/filebeat/registry
INFO registrar/registrar.go:119 States Loaded from registrar: 7
WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
INFO crawler/crawler.go:48 Loading Prospectors: 2
WARN [cfgwarn] prospector/config.go:25 DEPRECATED: input_type prospector config is deprecated. Use type instead. Will be removed in version: 6.0.0
INFO log/prospector.go:111 Configured paths: [/talend/6.1.1/runtime/log/tesb.log]
WARN [cfgwarn] prospector/config.go:25 DEPRECATED: input_type prospector config is deprecated. Use type instead. Will be removed in version: 6.0.0
INFO log/prospector.go:111 Configured paths: [/talend/6.1.1/runtime/data/log/wrapper.log]
INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 2
INFO log/harvester.go:216 Harvester started for file: /talend/6.1.1/runtime/log/tesb.log
INFO log/harvester.go:216 Harvester started for file: /talend/6.1.1/runtime/data/log/wrapper.log
ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":141},"total":{"ticks":800,"time":808,"value":800},"user":{"ticks":660,"time":667}},"info":{"ephemeral_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":30030}},"memstats":{"gc_next":49351216,"memory_alloc":41929928,"memory_total":196864576,"rss":62279680}},"filebeat":{"events":{"active":3459,"added":3466,"done":7},"harvester":{"open_files":2,"running":2,"started":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":2,"events":{"active":3458,"filtered":8,"published":3458,"retry":4096,"total":3466}}},"registrar":{"states":{"cleanup":2,"current":7,"update":7},"writes":7},"system":{"cpu":{"cores":8},"load":{"1":0.11,"15":0.11,"5":0.18,"norm":{"1":0.0138,"15":0.0138,"5":0.0225}}}}}}
ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":156},"total":{"ticks":830,"time":844,"value":830},"user":{"ticks":680,"time":688}},"info":{"ephemeral_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":60029}},"memstats":{"gc_next":49351216,"memory_alloc":46390504,"memory_total":201325152,"rss":2662400}},"filebeat":{"events":{"active":161,"added":161},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":3619,"published":161,"retry":1246,"total":161}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":0.07,"15":0.1,"5":0.16,"norm":{"1":0.0088,"15":0.0125,"5":0.02}}}}}}
ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":165},"total":{"ticks":880,"time":885,"value":880},"user":{"ticks":720,"time":720}},"info":{"ephemeral_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":90029}},"memstats":{"gc_next":25860592,"memory_alloc":14686280,"memory_total":203891576,"rss":872448}},"filebeat":{"events":{"active":181,"added":181},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":3800,"published":181,"retry":16,"total":181}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":0.11,"15":0.1,"5":0.16,"norm":{"1":0.0138,"15":0.0125,"5":0.02}}}}}}
INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":170,"time":174},"total":{"ticks":910,"time":918,"value":910},"user":{"ticks":740,"time":744}},"info":{"ephemeral_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":120029}},"memstats":{"gc_next":25860592,"memory_alloc":19599128,"memory_total":208804424,"rss":90112}},"filebeat":{"events":{"active":165,"added":165},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":3965,"published":165,"total":165}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":0.12,"15":0.1,"5":0.16,"norm":{"1":0.015,"15":0.0125,"5":0.02}}}}}}
ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused

kvch · September 4, 2018, 10:29am

Your output configuration seems to be incorrect. Could you double check it (correct, IP, port, SSL, credentials, etc.)? If that works and you still don't see the logs, please share the debug logs again.

Right now it seems that prospectors are started for both files.
Is it possible that there is no newline at the end of the last line in the second log file? Filebeat will not flush its buffers until it encounters a newline.

amritesh_mishra · September 4, 2018, 10:44am

then how one log file is going to logstash and the other is not going. In that case no log file should go if the ip is wrong

kvch · September 4, 2018, 10:50am

Indeed. The error messages might be caused by backpressure on Logstash. Could you please check the load of Logstash?

amritesh_mishra · September 4, 2018, 10:52am

How to check the load in logstash in linux system

kvch · September 5, 2018, 12:12pm

There is a guide on checking the performance of Logstash: https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html

system · October 25, 2018, 3:42pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Multiple input configuration Beats filebeat	9	3401	September 4, 2018
Filebeat is not starting when running the config file Beats	2	2186	May 25, 2018
Need help with setting up "Multiple prospectors" in filebeat Beats filebeat	12	3690	June 30, 2016
Multi-File configuration Beats filebeat	7	1482	September 10, 2018
Filebeat and Multiline Problem Beats filebeat	3	1194	September 14, 2016

Filebeat issue in configuration file

this is the config filebeat: prospectors:

< filebeat: prospectors:

Related Topics

this is the config
filebeat:
prospectors:

<
filebeat:
prospectors: