Filebeat -> Logstash "write: broken pipe"

Elastic Stack Beats
1

I'm having trouble getting TLS communication working between Filebeat and Logstash. I have followed @andrewkroh 's advice and got this far:

$ curl -v --cacert ca.crt https://logs.andrewkroh.com:5044
$ curl: (52) Empty reply from server

Starting Filebeat (which works with TLS turned off for both Logstash and Filebeat), however, results in this error:

2016/03/22 07:48:32.846449 publish.go:88: INFO Start sending events to output
2016/03/22 07:48:32.846759 log.go:113: INFO Harvester started for file: /var/log/messages
2016/03/22 07:48:35.360607 single.go:76: INFO Error publishing events (retrying): write tcp 192.168.0.56:51704->192.168.0.56:5043: write: broken pipe
2016/03/22 07:48:35.360650 single.go:152: INFO send fail 
2016/03/22 07:48:35.360666 single.go:159: INFO backoff retry: 1s

Can anyone offer assistance with this? Tomorrow I'll dig out tcpdump again and see if I can understand what's going wrong, but some tips would be appreciated.

2

@gregoryo Generally the “write: broken pipe” problem is found when there is a connection lost between you and logstash

3

can you check logstash logs too?

4

Starting logstash with this...

$ logstash -f /usr/local/etc/logstash/logstash.conf --debug

... nothing seems to happen in the output in response to the startup and attempted connection by filebeat. This line is output roughly every one second, regardless of what filebeat is doing:

Flushing buffer at interval {:instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x2729ffc2 @operations_mutex=#<Mutex:0x29ee1557>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x2dd194b2>, @submit_proc=#<Proc:0xec2a5b@/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:54>, @logger=#<Cabin::Channel:0x59f1d3bc @metrics=#<Cabin::Metrics:0x7824b834 @metrics_lock=#<Mutex:0x8fce023>, @metrics={}, @channel=#<Cabin::Channel:0x59f1d3bc ...>>, @subscriber_lock=#<Mutex:0x69a3bd95>, @level=:debug, @subscribers={12450=>#<Cabin::Outputs::IO:0x632015d1 @io=#<IO:fd 1>, @lock=#<Mutex:0xcb21c6e>>}, @data={}>, @last_flush=2016-03-22 22:35:55 +0800, @flush_interval=1, @stopping=#<Concurrent::AtomicBoolean:0x78894778>, @buffer=[], @flush_thread=#<Thread:0x19136d8 run>>", :interval=>1, :level=>:info, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}
5

(I have replaced @ with @' in the above, since the forum thought I was trying to mention more than 10 users).

If you format the log snippet as code this won't be a problem.

6

can you shre you filebeat and logstash config?

7

logstash.conf:

input {
  beats {
    port            => '5043'
    ssl             => true
    ssl_certificate => '/var/ck-logstash/ssl/certs/HOSTNAME.pem'
    ssl_key         => '/var/ck-logstash/ssl/private/HOSTNAME.key'
  }
}

output {
  stdout {}
}

filebeat.yml:

filebeat:
  prospectors:
    -
      paths:
        - '/var/log/messages'
      fields:
        type: syslog

output:
  logstash:
    hosts: ['HOSTNAME:5043']
  tls:
    certificate_authorities: /var/ck-logstash/ssl/cacert.pem
    certificate: /var/ck-logstash/ssl/certs/HOSTNAME.pem
    certificate_key: /var/ck-logstash/ssl/private/HOSTNAME.key
    timeout: 15

It works fine (although clearly without security) when I comment out 'ssl => true' line and 'tls' section.

Also, here's snippets of the curl details again (from @andrewkroh's reference above):

$ curl -v --cacert /var/ck-logstash/ssl/cacert.pem https://HOSTNAME:5043
*        SSL certificate verify ok.
curl: (52) Empty reply from server

And hey, it looks like I managed to make the 'Preformatted text' button work this time. I thought that was the right one when I was first trying.

8

Try this:

output:
  logstash:
    hosts: ['HOSTNAME:5043']
    tls:
      certificate_authorities: [/var/ck-logstash/ssl/cacert.pem]

All your tls options need right shifted by two spaces. certificate_authorities is a list. And certificate and certificate_key are only needed if you are doing mutual auth (authenticating the Filebeat client).

9

Excellent, thank you. Problem fixed. Another read of https://www.elastic.co/guide/en/beats/filebeat/1.1/configuration-output-tls.html was what I needed!