Filebeat issue in configuration file

Elastic Stack Beats
1

We have created this config file but file beat is sending only one file at a time it is not sending the second path file which i have put in the filebeat config file.

this is the config
filebeat:
prospectors:

  paths:
    - /talend/6.1.1/runtime/log/tesb.log

  input_type: log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

  • paths:

    • /home/talendpp/runtime/wrapper.log

    input_type: log
    fileds: {log_type: wrapperpreprodruntime}
    multiline.pattern: '^\d+'
    multiline.negate: true
    multiline.match: after

output:
logstash:
hosts: ["172.18.7.16:5044"]
index: filebeat

Please help on this is very urgent.

2

Could you please format you entire config using </>?

4

<
filebeat:
prospectors:

  paths:
    - /talend/6.1.1/runtime/log/tesb.log

  input_type: log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

  • paths:

    • /home/talendpp/runtime/wrapper.log

    input_type: log
    fileds: {log_type: wrapperpreprodruntime}
    multiline.pattern: '^\d+'
    multiline.negate: true
    multiline.match: after

output:
logstash:
hosts: ["172.18.7.16:5044"]
index: filebeat
/>

5

Could you try again? You might have a whitespace error and this way it's impossible to debug it.

You could try adding ``` to the beginning and to the end of your config file.

7

<```
filebeat:
prospectors:

  • paths:

    • /talend/6.1.1/runtime/log/tesb.log
      input_type: log
      fields: {log_type: tesbpreprodruntime}
      multiline.pattern: '^\d+'
      multiline.negate: true
      multiline.match: after

  • paths:

    • /home/talendpp/runtime/wrapper.loginput_type: log
      fileds: {log_type: wrapperpreprodruntime}
      multiline.pattern: '^\d+'
      multiline.negate: true
      multiline.match: after

output:
logstash:
hosts: ["172.18.7.16:5044"]
index: filebeat

8

This config should work:

filebeat.prospectors:
- input_type: log
  paths:
   - /talend/6.1.1/runtime/log/tesb.log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
   - /home/talendpp/runtime/wrapper.log
  fileds: {log_type: wrapperpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["172.18.7.16:5044"]
  index: filebeat
10

It is sending only the first path which you mentioned not the second one "- /talend/6.1.1/runtime/log/tesb.log"

This is the path which is being send

11

Could you please share the debug logs? (Output of filebeat -e -d "*")

12

2018-09-04T11:16:13.719+0100 INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-09-04T11:16:13.728+0100 INFO instance/beat.go:475 Beat UUID: 1a45d01b-b76b-4f6e-8642-61f11e6e7bed
2018-09-04T11:16:13.728+0100 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.3
2018-09-04T11:16:13.729+0100 INFO pipeline/module.go:76 Beat name: ukmr0xltmi002
2018-09-04T11:16:13.730+0100 INFO instance/beat.go:301 filebeat start running.
2018-09-04T11:16:13.730+0100 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-09-04T11:16:13.735+0100 INFO registrar/registrar.go:108 Loading registrar data from /var/lib/filebeat/registry
2018-09-04T11:16:13.736+0100 INFO registrar/registrar.go:119 States Loaded from registrar: 7
2018-09-04T11:16:13.736+0100 WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-09-04T11:16:13.736+0100 INFO crawler/crawler.go:48 Loading Prospectors: 2
2018-09-04T11:16:13.736+0100 WARN [cfgwarn] prospector/config.go:25 DEPRECATED: input_type prospector config is deprecated. Use type instead. Will be removed in version: 6.0.0
2018-09-04T11:16:13.736+0100 INFO log/prospector.go:111 Configured paths: [/talend/6.1.1/runtime/log/tesb.log]
2018-09-04T11:16:13.736+0100 WARN [cfgwarn] prospector/config.go:25 DEPRECATED: input_type prospector config is deprecated. Use type instead. Will be removed in version: 6.0.0
2018-09-04T11:16:13.739+0100 INFO log/prospector.go:111 Configured paths: [/talend/6.1.1/runtime/data/log/wrapper.log]
2018-09-04T11:16:13.739+0100 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 2
2018-09-04T11:16:13.739+0100 INFO log/harvester.go:216 Harvester started for file: /talend/6.1.1/runtime/log/tesb.log
2018-09-04T11:16:13.740+0100 INFO log/harvester.go:216 Harvester started for file: /talend/6.1.1/runtime/data/log/wrapper.log
2018-09-04T11:16:15.040+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
2018-09-04T11:16:17.042+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
2018-09-04T11:16:21.058+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
2018-09-04T11:16:29.059+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
2018-09-04T11:16:43.732+0100 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":141},"total":{"ticks":800,"time":808,"value":800},"user":{"ticks":660,"time":667}},"info":{"ephemeral_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":30030}},"memstats":{"gc_next":49351216,"memory_alloc":41929928,"memory_total":196864576,"rss":62279680}},"filebeat":{"events":{"active":3459,"added":3466,"done":7},"harvester":{"open_files":2,"running":2,"started":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":2,"events":{"active":3458,"filtered":8,"published":3458,"retry":4096,"total":3466}}},"registrar":{"states":{"cleanup":2,"current":7,"update":7},"writes":7},"system":{"cpu":{"cores":8},"load":{"1":0.11,"15":0.11,"5":0.18,"norm":{"1":0.0138,"15":0.0138,"5":0.0225}}}}}}
2018-09-04T11:16:45.061+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
2018-09-04T11:17:13.732+0100 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":156},"total":{"ticks":830,"time":844,"value":830},"user":{"ticks":680,"time":688}},"info":{"ephemeral_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":60029}},"memstats":{"gc_next":49351216,"memory_alloc":46390504,"memory_total":201325152,"rss":2662400}},"filebeat":{"events":{"active":161,"added":161},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":3619,"published":161,"retry":1246,"total":161}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":0.07,"15":0.1,"5":0.16,"norm":{"1":0.0088,"15":0.0125,"5":0.02}}}}}}
2018-09-04T11:17:17.098+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused
2018-09-04T11:17:43.731+0100 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":165},"total":{"ticks":880,"time":885,"value":880},"user":{"ticks":720,"time":720}},"info":{"ephemeral_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":90029}},"memstats":{"gc_next":25860592,"memory_alloc":14686280,"memory_total":203891576,"rss":872448}},"filebeat":{"events":{"active":181,"added":181},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":3800,"published":181,"retry":16,"total":181}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":0.11,"15":0.1,"5":0.16,"norm":{"1":0.0138,"15":0.0125,"5":0.02}}}}}}
2018-09-04T11:18:13.732+0100 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":170,"time":174},"total":{"ticks":910,"time":918,"value":910},"user":{"ticks":740,"time":744}},"info":{"ephemeral_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":120029}},"memstats":{"gc_next":25860592,"memory_alloc":19599128,"memory_total":208804424,"rss":90112}},"filebeat":{"events":{"active":165,"added":165},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":3965,"published":165,"total":165}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":0.12,"15":0.1,"5":0.16,"norm":{"1":0.015,"15":0.0125,"5":0.02}}}}}}
2018-09-04T11:18:17.100+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused

13

Your output configuration seems to be incorrect. Could you double check it (correct, IP, port, SSL, credentials, etc.)? If that works and you still don't see the logs, please share the debug logs again.

Right now it seems that prospectors are started for both files.
Is it possible that there is no newline at the end of the last line in the second log file? Filebeat will not flush its buffers until it encounters a newline.

14

then how one log file is going to logstash and the other is not going. In that case no log file should go if the ip is wrong

15

Indeed. The error messages might be caused by backpressure on Logstash. Could you please check the load of Logstash?

16

How to check the load in logstash in linux system

17

There is a guide on checking the performance of Logstash: https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.