# Filebeat issue in configuration file

**URL:** <https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 3, 2018, 11:43am UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062 "2018-09-03T11:43:02Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [September 3, 2018, 11:43am UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/1 "2018-09-03T11:43:02Z")

</div>

We have created this config file but file beat is sending only one file at a time it is not sending the second path file which i have put in the filebeat config file.

## this is the config filebeat: prospectors:

```
  paths:
    - /talend/6.1.1/runtime/log/tesb.log

  input_type: log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

```

- paths:

output:  
logstash:  
hosts: ["172.18.7.16:5044"]  
index: filebeat

Please help on this is very urgent.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 3, 2018, 12:36pm UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/2 "2018-09-03T12:36:57Z")

</div>

Could you please format you entire config using `</>`?

---

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [September 3, 2018, 12:38pm UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/4 "2018-09-03T12:38:52Z")

</div>

## \< filebeat: prospectors:

```
  paths:
    - /talend/6.1.1/runtime/log/tesb.log

  input_type: log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

```

- paths:

output:  
logstash:  
hosts: ["172.18.7.16:5044"]  
index: filebeat  
/\>

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 3, 2018, 2:08pm UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/5 "2018-09-03T14:08:40Z")

</div>

Could you try again? You might have a whitespace error and this way it's impossible to debug it.

You could try adding ``` to the beginning and to the end of your config file.

---

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [September 3, 2018, 2:44pm UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/7 "2018-09-03T14:44:36Z")

</div>

\<```  
filebeat:  
prospectors:

- paths:

- paths:

output:  
logstash:  
hosts: ["172.18.7.16:5044"]  
index: filebeat

```auto

```

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 3, 2018, 5:31pm UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/8 "2018-09-03T17:31:42Z")

</div>

This config should work:

```auto
filebeat.prospectors:
- input_type: log
  paths:
   - /talend/6.1.1/runtime/log/tesb.log
  fields: {log_type: tesbpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
   - /home/talendpp/runtime/wrapper.log
  fileds: {log_type: wrapperpreprodruntime}
  multiline.pattern: '^\d+'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["172.18.7.16:5044"]
  index: filebeat

```

---

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [September 4, 2018, 7:20am UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/10 "2018-09-04T07:20:55Z")

</div>

It is sending only the first path which you mentioned not the second one "- /talend/6.1.1/runtime/log/tesb.log"

This is the path which is being send

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 4, 2018, 10:14am UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/11 "2018-09-04T10:14:12Z")

</div>

Could you please share the debug logs? (Output of `filebeat -e -d "*"`)

---

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [September 4, 2018, 10:24am UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/12 "2018-09-04T10:24:08Z")

</div>

2018-09-04T11:16:13.719+0100 INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]  
2018-09-04T11:16:13.728+0100 INFO instance/beat.go:475 Beat UUID: 1a45d01b-b76b-4f6e-8642-61f11e6e7bed  
2018-09-04T11:16:13.728+0100 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.3  
2018-09-04T11:16:13.729+0100 INFO pipeline/module.go:76 Beat name: ukmr0xltmi002  
2018-09-04T11:16:13.730+0100 INFO instance/beat.go:301 filebeat start running.  
2018-09-04T11:16:13.730+0100 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s  
2018-09-04T11:16:13.735+0100 INFO registrar/registrar.go:108 Loading registrar data from /var/lib/filebeat/registry  
2018-09-04T11:16:13.736+0100 INFO registrar/registrar.go:119 States Loaded from registrar: 7  
2018-09-04T11:16:13.736+0100 WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.  
2018-09-04T11:16:13.736+0100 INFO crawler/crawler.go:48 Loading Prospectors: 2  
2018-09-04T11:16:13.736+0100 WARN [cfgwarn] prospector/config.go:25 DEPRECATED: input\_type prospector config is deprecated. Use type instead. Will be removed in version: 6.0.0  
2018-09-04T11:16:13.736+0100 INFO log/prospector.go:111 Configured paths: [/talend/6.1.1/runtime/log/tesb.log]  
2018-09-04T11:16:13.736+0100 WARN [cfgwarn] prospector/config.go:25 DEPRECATED: input\_type prospector config is deprecated. Use type instead. Will be removed in version: 6.0.0  
2018-09-04T11:16:13.739+0100 INFO log/prospector.go:111 Configured paths: [/talend/6.1.1/runtime/data/log/wrapper.log]  
2018-09-04T11:16:13.739+0100 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 2  
2018-09-04T11:16:13.739+0100 INFO log/harvester.go:216 Harvester started for file: /talend/6.1.1/runtime/log/tesb.log  
2018-09-04T11:16:13.740+0100 INFO log/harvester.go:216 Harvester started for file: /talend/6.1.1/runtime/data/log/wrapper.log  
2018-09-04T11:16:15.040+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused  
2018-09-04T11:16:17.042+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused  
2018-09-04T11:16:21.058+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused  
2018-09-04T11:16:29.059+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused  
2018-09-04T11:16:43.732+0100 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":141},"total":{"ticks":800,"time":808,"value":800},"user":{"ticks":660,"time":667}},"info":{"ephemeral\_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":30030}},"memstats":{"gc\_next":49351216,"memory\_alloc":41929928,"memory\_total":196864576,"rss":62279680}},"filebeat":{"events":{"active":3459,"added":3466,"done":7},"harvester":{"open\_files":2,"running":2,"started":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":2,"events":{"active":3458,"filtered":8,"published":3458,"retry":4096,"total":3466}}},"registrar":{"states":{"cleanup":2,"current":7,"update":7},"writes":7},"system":{"cpu":{"cores":8},"load":{"1":0.11,"15":0.11,"5":0.18,"norm":{"1":0.0138,"15":0.0138,"5":0.0225}}}}}}  
2018-09-04T11:16:45.061+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused  
2018-09-04T11:17:13.732+0100 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":156},"total":{"ticks":830,"time":844,"value":830},"user":{"ticks":680,"time":688}},"info":{"ephemeral\_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":60029}},"memstats":{"gc\_next":49351216,"memory\_alloc":46390504,"memory\_total":201325152,"rss":2662400}},"filebeat":{"events":{"active":161,"added":161},"harvester":{"open\_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":3619,"published":161,"retry":1246,"total":161}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":0.07,"15":0.1,"5":0.16,"norm":{"1":0.0088,"15":0.0125,"5":0.02}}}}}}  
2018-09-04T11:17:17.098+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused  
2018-09-04T11:17:43.731+0100 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":165},"total":{"ticks":880,"time":885,"value":880},"user":{"ticks":720,"time":720}},"info":{"ephemeral\_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":90029}},"memstats":{"gc\_next":25860592,"memory\_alloc":14686280,"memory\_total":203891576,"rss":872448}},"filebeat":{"events":{"active":181,"added":181},"harvester":{"open\_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":3800,"published":181,"retry":16,"total":181}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":0.11,"15":0.1,"5":0.16,"norm":{"1":0.0138,"15":0.0125,"5":0.02}}}}}}  
2018-09-04T11:18:13.732+0100 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":170,"time":174},"total":{"ticks":910,"time":918,"value":910},"user":{"ticks":740,"time":744}},"info":{"ephemeral\_id":"4b29e837-8761-4749-8def-e7fd01c3e7cc","uptime":{"ms":120029}},"memstats":{"gc\_next":25860592,"memory\_alloc":19599128,"memory\_total":208804424,"rss":90112}},"filebeat":{"events":{"active":165,"added":165},"harvester":{"open\_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":3965,"published":165,"total":165}}},"registrar":{"states":{"current":7}},"system":{"load":{"1":0.12,"15":0.1,"5":0.16,"norm":{"1":0.015,"15":0.0125,"5":0.02}}}}}}  
2018-09-04T11:18:17.100+0100 ERROR pipeline/output.go:74 Failed to connect: dial tcp 172.18.7.16:5044: getsockopt: connection refused

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 4, 2018, 10:29am UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/13 "2018-09-04T10:29:37Z")

</div>

Your output configuration seems to be incorrect. Could you double check it (correct, IP, port, SSL, credentials, etc.)? If that works and you still don't see the logs, please share the debug logs again.

Right now it seems that prospectors are started for both files.  
Is it possible that there is no newline at the end of the last line in the second log file? Filebeat will not flush its buffers until it encounters a newline.

---

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [September 4, 2018, 10:44am UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/14 "2018-09-04T10:44:27Z")

</div>

then how one log file is going to logstash and the other is not going. In that case no log file should go if the ip is wrong

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 4, 2018, 10:50am UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/15 "2018-09-04T10:50:52Z")

</div>

Indeed. The error messages might be caused by backpressure on Logstash. Could you please check the load of Logstash?

---

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [September 4, 2018, 10:52am UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/16 "2018-09-04T10:52:45Z")

</div>

How to check the load in logstash in linux system

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 5, 2018, 12:12pm UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/17 "2018-09-05T12:12:15Z")

</div>

There is a guide on checking the performance of Logstash: [https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html](https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 25, 2018, 3:42pm UTC](https://discuss.elastic.co/t/filebeat-issue-in-configuration-file/147062/19 "2018-10-25T15:42:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.