Help newcomer: Payloads received but no output

[tehowe]

Hello!

I'm trying to get a single filebeat to logstash test connection up and running in my environment and have been for some time. I'd been struggling with what I thought was an encryption problem for a while on 5.6 until it finally occurred to me to remove the encryption lines from the filebeat and logstash configurations, try it in the clear, and discovered I was still getting connection resets in the filebeat logs. No data was being sent. I upped the timeout in logstash and the problem continued.

Next, a recent discussion on IRC led me to update all the things to 6.0.0 in order to make use of the filebeat test output feature and under 6.0.0 it appears the connection is now being made... which is confusing... but I'm not seeing any output regardless of whether I use the stdout, file, or elasticsearch output modules. The way forward is probably to a) take a deep breath, b) get this working unencrypted, then if I still have issues with encryption I'll open another topic. So hopefully someone can help!

Here are the relevant clauses of the configuration files:

/etc/filebeat/filebeat.yml

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.1.20:5044"]

#  ssl.enabled: true
#  ssl.supported_protocols: [TLSv1.2]
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
#  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"]

  # Certificate for SSL client authentication
#  ssl.certificate: "/etc/pki/tls/certs/logstash.crt"

  # Client Certificate Key
#  ssl.key: "/etc/pki/tls/client/logstash.key"

/etc/logstash/conf.d/logstash.conf

input {
   beats {
      port => 5044
      codec => "json_lines"
      client_inactivity_timeout => 1200
    }
}
output {
   stdout {}
}

I've also tried, in place of stdout,

output {
 file {
   path => "/var/log/logstash/output.txt"
   codec => line { format => "custom format: %{message}"}
 }
}

As well as

output {
  elasticsearch {
    hosts => ["192.168.1.30:9200", "192.168.1.31:9200", "192.168.1.32:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

But regardless of whether I run logstash as a daemon or from the command line with

/usr/share/logstash/bin/logstash --debug --path.settings /etc/logstash

I see neither output to the console, file on disk, nor indices in my ES cluster being generated.

I do see this in the logs, though, which suggests the data is getting to logstash

[2017-11-17T16:33:07,572][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x5ea02d32@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290 sleep>"}
[2017-11-17T16:33:12,439][DEBUG][org.logstash.beats.BeatsHandler] [local: 192.168.1.20:5044, remote: 192.168.1.10:49582] Received a new payload
[2017-11-17T16:33:12,439][DEBUG][org.logstash.beats.BeatsHandler] [local: 192.168.1.20:5044, remote: 192.168.1.10:49582] Sending a new message for the listener, sequence: 1
[2017-11-17T16:33:12,573][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x5ea02d32@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290 sleep>"}

What is going on???

[tehowe]

I removed this line in the input clause

codec => "json_lines"

Apparently filebeat doesn't ship in json

On to backfilling the crypto!

[Christian_Dahlqvist]

The beats plugin decides the message, so the codec will be applied to the message field as usual.