# Can't recieve filebeat logs via Logstash

**URL:** <https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893>\
**Category:** Logstash\
**Created:** [April 22, 2019, 7:18pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893 "2019-04-22T19:18:24Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 22, 2019, 7:18pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/1 "2019-04-22T19:18:24Z")

</div>

Hi,

I have a problem with recieving logs via logstash from filebeat. I can recieve them directly to elasticsearch but not via Logastash.

Logstash conf.d input file:

```
input {
    beats {
        type => "filebeat"
        port => "5044"
    }
}

```

Filebeat config:

```
#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
# output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: ["http://192.168.0.160:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.0.160:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

```

I don't find the problem in the logs. How can I test the connection?  
Later I want to add this filter: [https://gist.github.com/nagyv/398b0f9d0cb4061b6068](https://gist.github.com/nagyv/398b0f9d0cb4061b6068) for Odoo logs. But I'm far away from that. The netflow module is working fine this is also configured in Logstash.

---

<div class="post-metadata">

**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\
**Post date:** [April 22, 2019, 8:43pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/2 "2019-04-22T20:43:30Z")

</div>

Did you check both the filebeat and Logstash logs? Also you need an output from Logstash to Elasticsearch. [https://www.elastic.co/guide/en/logstash/current/output-plugins.html](https://www.elastic.co/guide/en/logstash/current/output-plugins.html)  
Example:

```auto
input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => ["https://abc.gov:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
}

```

Make sure that your plugins for Logstash are enabled as well.  
[https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html](https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html)

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 23, 2019, 4:28am UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/3 "2019-04-23T04:28:26Z")

</div>

Hi Ryan,

Thanks for your reply. I have added in the output the missing code. But id does not help.  
I found in the log the following:

```
Apr 23 06:23:52 SVGWMA-XXXX-04 filebeat: 2019-04-23T06:23:52.302+0200#011ERROR#011pipeline/output.go:100#011Failed to connect to backoff(async(tcp://192.168.0.160:5044)): dial tcp 192.168.0.160:5044: connect: connection refused
Apr 23 06:23:52 SVGWMA-XXXXX-04 filebeat: 2019-04-23T06:23:52.302+0200#011INFO#011pipeline/output.go:93#011Attempting to reconnect to backoff(async(tcp://192.168.0.160:5044)) with 736 reconnect attempt(s)
Apr 23 06:23:52 SVGWMA-XXXXX-04 filebeat: 2019-04-23T06:23:52.302+0200#011DEBUG#011[logstash]#011logstash/async.go:111#011connect
Apr 23 06:23:54 SVGWMA-XXXXX-04 filebeat: 2019-04-23T06:23:54.925+0200#011INFO#011[monitoring]#011log/log.go:144#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":6640,"time":{"ms":9}},"total":{"ticks":27590,"time":{"ms":11},"value":27590},"user":{"ticks":20950,"time":{"ms":2}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":8},"info":{"ephemeral_id":"62baf388-b30e-48ac-a007-b6bbf26713aa","uptime":{"ms":33303045}},"memstats":{"gc_next":44016864,"memory_alloc":22763448,"memory_total":492209632}},"filebeat":{"harvester":{"open_files":3,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":7,"events":{"active":4118,"retry":2048}}},"registrar":{"states":{"current":8}},"system":{"load":{"1":0.02,"15":0.05,"5":0.03,"norm":{"1":0.01,"15":0.025,"5":0.015}}}}}}
Apr 23 06:23:57 SVGWMA-XXXX-04 filebeat: 2019-04-23T06:23:57.054+0200#011DEBUG#011[input]#011input/input.go:152#011Run input
Apr 23 06:23:57 SVGWMA-XXXX-04 filebeat: 2019-04-23T06:23:57.054+0200#011DEBUG#011[input]#011log/input.go:174#011Start next scan
Apr 23 06:23:57 SVGWMA-XXXXX-04 filebeat: 2019-04-23T06:23:57.054+0200#011DEBUG#011[input]#011log/input.go:404#011Check file for harvesting: /var/log/audit/audit.log

```

So filebeats is running and collection and also sends to the correct ip. Connection refused... but why?

---

<div class="post-metadata">

**Author:** ![Debashis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debashis/32/45290_2.png) [@Debashis](https://discuss.elastic.co/u/Debashis)\
**Post date:** [April 23, 2019, 6:23am UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/4 "2019-04-23T06:23:46Z")

</div>

@hispeed What about your firewall and selinux configuration in logstash node? Make sure that firewall is not restricting the connection.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 23, 2019, 3:08pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/5 "2019-04-23T15:08:47Z")

</div>

Hi @Debashis

Both is disabled on both machines.

---

<div class="post-metadata">

**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\
**Post date:** [April 23, 2019, 3:11pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/6 "2019-04-23T15:11:24Z")

</div>

Do you have xpack enabled on your ES cluster or Logstash? With the connection refused error you may need to pass your username and password credentials for ES.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 23, 2019, 3:12pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/7 "2019-04-23T15:12:47Z")

</div>

No X-Pack is not enabled also not in logstash.

```
# ------------ X-Pack Settings (not applicable for OSS build)--------------
#
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
#xpack.monitoring.enabled: false
#xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: password
#xpack.monitoring.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.monitoring.elasticsearch.ssl.certificate_authority: ["/path/to/ca.crt"]
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.monitoring.elasticsearch.ssl.keystore.password: password
#xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.elasticsearch.sniffing: false
#xpack.monitoring.collection.interval: 10s
#xpack.monitoring.collection.pipeline.details.enabled: true
#
# X-Pack Management
# https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html
#xpack.management.enabled: false
#xpack.management.pipeline.id: ["main", "apache_logs"]
#xpack.management.elasticsearch.username: logstash_admin_user
#xpack.management.elasticsearch.password: password
#xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.management.elasticsearch.ssl.certificate_authority: ["/path/to/ca.crt"]
#xpack.management.elasticsearch.ssl.truststore.path: /path/to/file
#xpack.management.elasticsearch.ssl.truststore.password: password
#xpack.management.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.management.elasticsearch.ssl.keystore.password: password
#xpack.management.elasticsearch.ssl.verification_mode: certificate
#xpack.management.elasticsearch.sniffing: false
#xpack.management.logstash.poll_interval: 5s
```

---

<div class="post-metadata">

**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\
**Post date:** [April 23, 2019, 3:32pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/8 "2019-04-23T15:32:34Z")

</div>

If you've created a username and password for logstash\_internal try passing those credentials through in your output config. The failed to connect message is whats most important to troubleshot. Also, take a look at what indices your logstash\_writer role has access to. Your probably going to have to add filebeat\* to its permissions.

```auto
output {
  elasticsearch {
    hosts => ["https://abc.gov:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => "logstash_internal"
    password => "team_password"
    #ssl => true
    #ssl_certificate_verification => false
    #cacert => "/etc/logstash/globalcert/ca/ca.crt"
    }

```

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 23, 2019, 3:42pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/9 "2019-04-23T15:42:41Z")

</div>

I have not created anyhting with X-Pack or a role in kibana. So this is probably not the problem. I think that Logstash is not starting the pipeline.

```
[2019-04-23T17:28:21,020][WARN][logstash.runner] SIGTERM received. Shutting down.
[2019-04-23T17:28:21,928][INFO][logstash.javapipeline] Pipeline terminated {"pipeline.id"=>"module-netflow"}
[2019-04-23T17:28:52,557][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-04-23T17:28:52,576][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.0.0"}
[2019-04-23T17:28:54,015][INFO][logstash.config.modulescommon] Starting the netflow module
[2019-04-23T17:29:16,769][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-04-23T17:29:17,100][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-04-23T17:29:17,182][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-04-23T17:29:17,188][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-04-23T17:29:17,294][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-04-23T17:29:18,175][INFO][logstash.filters.geoip] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.0-java/vendor/GeoLite2-ASN.mmdb"}
[2019-04-23T17:29:18,234][INFO][logstash.filters.geoip] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.0-java/vendor/GeoLite2-City.mmdb"}
[2019-04-23T17:29:18,242][INFO][logstash.filters.geoip] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.0-java/vendor/GeoLite2-City.mmdb"}
[2019-04-23T17:29:18,840][INFO][logstash.filters.geoip] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.0-java/vendor/GeoLite2-ASN.mmdb"}
[2019-04-23T17:29:18,917][INFO][logstash.filters.geoip] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.0-java/vendor/GeoLite2-ASN.mmdb"}
[2019-04-23T17:29:18,921][INFO][logstash.filters.geoip] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.0-java/vendor/GeoLite2-City.mmdb"}
[2019-04-23T17:29:19,455][INFO][logstash.filters.geoip] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.0-java/vendor/GeoLite2-City.mmdb"}
[2019-04-23T17:29:19,679][INFO][logstash.filters.geoip] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.0-java/vendor/GeoLite2-ASN.mmdb"}
[2019-04-23T17:29:19,752][INFO][logstash.javapipeline] Starting pipeline {:pipeline_id=>"module-netflow", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, :thread=>"#<Thread:0x6d07a31 run>"}
[2019-04-23T17:29:19,994][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"module-netflow"}
[2019-04-23T17:29:20,246][INFO][logstash.inputs.udp] Starting UDP listener {:address=>"0.0.0.0:2055"}
[2019-04-23T17:29:20,296][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:"module-netflow"], :non_running_pipelines=>[]}
[2019-04-23T17:29:20,599][INFO][logstash.inputs.udp] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"212992", :queue_size=>"2000"}
[2019-04-23T17:29:21,407][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2019-04-23T17:30:02,782][WARN][logstash.codecs.netflow] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute.

```

How Can I check if the input 5044 is working and the pipeline is up and running?

---

<div class="post-metadata">

**Author:** ![Ryan\_Downey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_downey/32/35987_2.png) [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\
**Post date:** [April 23, 2019, 4:04pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/10 "2019-04-23T16:04:07Z")

</div>

> [@hispeed](#):
>
> [2019-04-23T17:29:20,296][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:"module-netflow"], :non\_running\_pipelines=\>}

According to this your pipeline is running.

This is going back a way for me but if I remember correctly I had to copy my config and copy it into the Pipelines tool in Kibana to get things working. I'm not sure why that worked but it did. After that I had to make sure that the logstash\_writer role, assuming you've created one, had permissions to write, delete and create indices for the specific beat.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 23, 2019, 6:25pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/11 "2019-04-23T18:25:02Z")

</div>

I can't connect via telnet to logstash so the problem is probably logstash.  
I hope someone from logstash is reading this?

I recieve: connection refused when I try via telnet.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 23, 2019, 7:18pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/12 "2019-04-23T19:18:57Z")

</div>

Try explicitly setting host =\> 192.168.0.160 in the beats input configuration.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 23, 2019, 8:02pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/13 "2019-04-23T20:02:40Z")

</div>

Hi Badger,

Doesn't seem to work.

What was happening here? [https://github.com/elastic/logstash/issues/8420](https://github.com/elastic/logstash/issues/8420)

I think this is the real problem i have. It's not picking up the config files. The netflow module is working fine and I have in conf.d the configuration which it should pick. At the moment pipelines.yml is commented out.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 24, 2019, 6:10pm UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/14 "2019-04-24T18:10:13Z")

</div>

Update:

This all happened because of the netflow module which i had activated. Does anyone actually know how the netflow module should be started and configured. (It would be nice stept by step on CentOS 7)?  
@theuntergeek and @guyboertje  
I'm sorry for adding you in this but guyboertje this is a information problem look at:

> [@Run "---setup" netflow module when LS is a service (was Install netflow module error)](https://discuss.elastic.co/t/run-setup-netflow-module-when-ls-is-a-service-was-install-netflow-module-error/109166/10):
>
> Yeah, i added var.kibana.scheme to my config but still got same result. I'm wondering why I can not see the log file.

> [@Pipelines.yml ignored on logstash running as a service](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828):
>
> First, I saw and tried the solution mentioned [here](https://discuss.elastic.co/t/logstash-v6-beta-with-multiple-pipelines-configuration/102559) and commented out path.config from /etc/logstash/logstash.yml Looks like when I run logstash from command line all works well, but I need the logstash service to run. ie. "systemctl start logstash" Logstash completely ignores pipelines.yml Any advice or ideas? Best, JD

When I deactivate the netflow module i recieve the filebeat notfications. So the question is, how to start and configure the netflow module.

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 30, 2019, 6:01am UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/15 "2019-04-30T06:01:34Z")

</div>

HI,

Refer this blog

> **[LOG Centralization: Using Filebeat and Logstash](https://medium.com/tensult/log-centralization-using-filebeat-and-logstash-11640f77cf70)**
>
> Logs give information about system behavior. In every service, there will be logs with different content and different format. The…

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [May 3, 2019, 4:37am UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/16 "2019-05-03T04:37:28Z")

</div>

Hi jawad846,

Thanks for your link but it doesn't help me a lot.  
Also if I look on GitHub it looks to me that the Netflow at least won't be supported in the future.

Any statements?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 31, 2019, 4:37am UTC](https://discuss.elastic.co/t/cant-recieve-filebeat-logs-via-logstash/177893/17 "2019-05-31T04:37:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.