# Filebeat not fowarding to Logstash

**URL:** <https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 16, 2017, 3:25pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358 "2017-02-16T15:25:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 3:25pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/1 "2017-02-16T15:25:51Z")

</div>

I can't seem to get filebeat to send the logs to logstash. I'm trying to set up the elk stack on a remote ssh and visualize the logs in kibana. I've read some of the other posts, but still can't figure out the error. I have posted my filebeat.yml

```
> filebeat.prospectors:

- input_type: log

  paths:

    - /var/log/auth.log
   # - /var/log/*.log
    - /var/log/syslog

> document_type: syslog

> output.logstash:

> hosts: ["hostname:5044"]
> bulk_max_size: 1024

> tls:
> certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

```

My logstash.conf:

```
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
    ssl_key => "/etc/pki/tls/private/logstash.key"

    ssl_verify_mode => "force_peer"
    ssl_certificate_authorities => ["/etc/pki/tls/certs/filebeat.crt"],
  }
}

```

...

> output {  
> elasticsearch {  
> hosts =\> ["localhost:9200"]  
> sniffing =\> true  
> manage\_template =\> false  
> index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
> document\_type =\> "%{[@metadata][type]}"  
> }  
> }

I am receiving error when performing a tail on my filebeat:

2017-02-16T10:24:24-05:00 DBG Try to publish 1024 events to logstash with window size 1  
2017-02-16T10:24:24-05:00 DBG handle error: read tcp elk\_privateip:52790-\>elk\_privateip:5044: read: connection reset by peer  
2017-02-16T10:24:24-05:00 DBG 0 events out of 1024 events sent to logstash. Continue sending  
2017-02-16T10:24:24-05:00 DBG close connection  
2017-02-16T10:24:24-05:00 DBG closing  
2017-02-16T10:24:24-05:00 ERR Failed to publish events caused by: read tcp elk\_privateip-\>elk\_privateip: read: connection reset by peer  
2017-02-16T10:24:24-05:00 INFO Error publishing events (retrying): read tcp elk\_privateip-\>elk\_privateip: read: connection reset by peer  
2017-02-16T10:24:24-05:00 DBG close connection

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 16, 2017, 4:37pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/2 "2017-02-16T16:37:51Z")

</div>

What Filebeat version are you using?

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 4:53pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/3 "2017-02-16T16:53:58Z")

</div>

version 5.2

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 16, 2017, 4:56pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/4 "2017-02-16T16:56:02Z")

</div>

For 5.x the configuration options are named `ssl` and not `tls`. This was changed to be consistent across Elastic projects. See [https://www.elastic.co/guide/en/beats/filebeat/current/configuration-output-ssl.html](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-output-ssl.html)

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 5:12pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/5 "2017-02-16T17:12:50Z")

</div>

ah, interesting, now when I try to restart filebeat...it throws an error loading the yaml: line 104: did not find expected key. But, I've commented out this section...

```
output.logstash:

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/tls/certs/filebeat.crt"

  # Client Certificate Key
  #ssl.key: "/etc/pki/tls/private/filebeat.key"
```

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 5:16pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/6 "2017-02-16T17:16:45Z")

</div>

Do these need to match my logstash.conf inputs? I re edited my original post with my current logstash.conf

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 16, 2017, 5:26pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/7 "2017-02-16T17:26:02Z")

</div>

> [@kmrychl](#):
>
> error loading the yaml: line 104: did not find expected key

I don't see where you defined `hosts: [xxx]`. That's probably what it's complaining about with that error.

And you have commented out the `ssl.certificate` and `ssl.key`. Those will be needed since `force_peer` is used in Logstash.

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 5:32pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/8 "2017-02-16T17:32:50Z")

</div>

Would this be correct?

```
output.logstash:
    # The Logstash hosts
        hosts: ["elkserver:5044"]

    bulk_max_size: 1024

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
    ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

  # Certificate for SSL client authentication
  ssl.certificate: "/etc/pki/tls/certs/filebeat.crt"

  # Client Certificate Key
  ssl.key: "/etc/pki/tls/private/filebeat.key"
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 16, 2017, 5:38pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/9 "2017-02-16T17:38:30Z")

</div>

No. It's a YAML configuration file and in YAML [indentation](https://en.wikipedia.org/wiki/YAML#Indented_delimiting) is very important.

```auto
output.logstash:
  hosts: ["elkserver:5044"]
  bulk_max_size: 1024
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
  ssl.certificate: "/etc/pki/tls/certs/filebeat.crt"
  ssl.key: "/etc/pki/tls/private/filebeat.key"

```

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 5:43pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/10 "2017-02-16T17:43:27Z")

</div>

Thanks, but it is still throwing an error. No events are being published

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 16, 2017, 6:02pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/11 "2017-02-16T18:02:34Z")

</div>

Please repost the complete configuration file. The `</>` button to format it.

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 6:53pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/12 "2017-02-16T18:53:01Z")

</div>

logstash.conf

```
input {
  beats {
    port => 5044
    ssl => true # enable TLS/SSL

    # configure logstash server certificate being presented to filebeat
    ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
    ssl_key => "/etc/pki/tls/private/logstash.key"

    # configure client auth + filebeat cert for validation
    ssl_verify_mode => "force_peer",
    ssl_certificate_authorities => ["/etc/pki/tls/certs/filebeat.crt"],
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    }
    syslog_pri { }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

filebeat yaml:

```
filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
   
    - /var/log/auth.log
   # - /var/log/*.log
    - /var/log/syslog
  
  document_type: syslog

#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
    hosts: ["elkserver:5044"]
   
    bulk_max_size: 1024 
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
    ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"]
  
  # Certificate for SSL client authentication
    ssl.certificate: "/etc/pki/tls/certs/filebeat.crt"

  # Client Certificate Key
    ssl.key: "/etc/pki/tls/private/filebeat.key"
#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug
logging.level: debug
logging.to_files: true
logging.to_syslog: false
logging.files:
path: /var/log/mybeat
name: mybeat.log
keepfiles: 7
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 16, 2017, 7:53pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/13 "2017-02-16T19:53:10Z")

</div>

The logging configuration is invalid due to indentation. There is a example provided with the beat in filebeat.full.yml that shows the correct indentation.

```auto
logging.files:
  path: /var/log/mybeat
  name: mybeat.log
  keepfiles: 7

```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 16, 2017, 8:20pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/15 "2017-02-16T20:20:01Z")

</div>

What's in the Filebeat log file?

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 8:22pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/16 "2017-02-16T20:22:38Z")

</div>

tail -f /var/log/mybeat/mybeat.log

```
2017-02-16T15:24:26-05:00 DBG Check file for harvesting: /var/log/auth.log
2017-02-16T15:24:26-05:00 DBG Update existing file for harvesting: /var/log/auth.log, offset: 334917
2017-02-16T15:24:26-05:00 DBG Harvester for file is still running: /var/log/auth.log
2017-02-16T15:24:26-05:00 DBG Check file for harvesting: /var/log/syslog
2017-02-16T15:24:26-05:00 DBG Update existing file for harvesting: /var/log/syslog, offset: 315276
2017-02-16T15:24:26-05:00 DBG Harvester for file is still running: /var/log/syslog
2017-02-16T15:24:26-05:00 DBG Prospector states cleaned up. Before: 2, After: 2
2017-02-16T15:24:29-05:00 DBG connect
2017-02-16T15:24:29-05:00 ERR Connecting error publishing events (retrying): dial tcp server:5044: getsockopt: connection refused
2017-02-16T15:24:29-05:00 DBG send fail
2017-02-16T15:24:36-05:00 DBG Run prospector
2017-02-16T15:24:36-05:00 DBG Start next scan
2017-02-16T15:24:36-05:00 DBG Check file for harvesting: /var/log/auth.log
2017-02-16T15:24:36-05:00 DBG Update existing file for harvesting: /var/log/auth.log, offset: 334917

```

tail -f /var/log/filebeat/filebeat

```
2017-02-16T14:42:52-05:00 INFO Stopping spooler
2017-02-16T14:42:52-05:00 DBG Spooler has stopped
2017-02-16T14:42:52-05:00 DBG Shutting down sync publisher
2017-02-16T14:42:52-05:00 INFO Stopping Registrar
2017-02-16T14:42:52-05:00 INFO Ending Registrar
2017-02-16T14:42:52-05:00 DBG Write registry file: /var/lib/filebeat/registry
2017-02-16T14:42:52-05:00 DBG Registry file updated. 0 states written.
2017-02-16T14:42:52-05:00 INFO Total non-zero values: filebeat.harvester.closed=2 filebeat.harvester.started=2 libbeat.publisher.published_events=2046 registrar.writes=1
2017-02-16T14:42:52-05:00 INFO Uptime: 2.881754748s
2017-02-16T14:42:52-05:00 INFO filebeat stopped.
```

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 8:37pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/17 "2017-02-16T20:37:19Z")

</div>

Also, if I run sudo service logstash configtest, I get:

```
sudo service logstash configtest

/etc/init.d/logstash: 156: /etc/init.d/logstash: /opt/logstash/bin/logstash: not found
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 16, 2017, 8:41pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/18 "2017-02-16T20:41:20Z")

</div>

> [@kmrychl](#):
>
> dial tcp server:5044: getsockopt: connection refused

It seem like there's a problem with the connection to Logstash. And based on the other output you pasted it seems like there's a problem with your Logstash installation.

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 16, 2017, 8:42pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/19 "2017-02-16T20:42:03Z")

</div>

Is there a way to do a clean uninstall. I'd like to start over, but am worried to delete everything. I just don't know how to debug the error at this point.

**Am editing here becuase I have reached the max edits**

It was installed by someone else and now I am accessing the remote server. I believe it was installed by a package apt-get

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 16, 2017, 8:42pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/20 "2017-02-16T20:42:32Z")

</div>

How did you install it?

---

<div class="post-metadata">

**Author:** ![kmrychl](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kmrychl](https://discuss.elastic.co/u/kmrychl)\
**Post date:** [February 17, 2017, 3:46pm UTC](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358/21 "2017-02-17T15:46:54Z")

</div>

I think now the problem is that my /opt/logstash/bin/logstash is not found so I am missing the binaries. My opt/logstash has no bin. The only path is /opt/logstash/vendor/bundle/jruby/1.9, therefore my init.d file is not connecting. Just not sure what needs to be fixed

[Next page](https://discuss.elastic.co/t/filebeat-not-fowarding-to-logstash/75358.md?page=2)