Issue with filebeat & logstash "Beats input: unhandled exception"

Elastic Stack Beats
1

OS: Ubuntu 14.094
filebeat: filebeat version 1.0.0-rc1 (amd64)
logstash: 2.0.0-1

Current ELK user in production. Building a test Filebeat => Logstash => Elasticsearch stack to try out filebeat. After installing lostash & filebeat on a new system I keep getting the following errors in logstash.log

{:timestamp=>"2015-11-05T22:20:53.195000+0000", :message=>"Beats input: unhandled exception", :exception=>#<TypeError: The field '@timestamp' must be a (LogStash::Timestamp, not a String (2015-11-05T22:20:50.676Z)>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/event.rb:138:in `[]='", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/logstash/inputs/beats.rb:138:in `create_event'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/logstash/inputs/beats.rb:138:in `create_event'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-plain-2.0.2/lib/logstash/codecs/plain.rb:35:in `decode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/logstash/inputs/beats.rb:136:in `create_event'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/logstash/inputs/beats.rb:150:in `invoke'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:370:in `data'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:349:in `read_socket'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:361:in `ack_if_needed'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:345:in `read_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:246:in `json_data_payload'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:163:in `feed'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:296:in `compressed_payload'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:163:in `feed'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:330:in `read_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats/server.rb:315:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/logstash/inputs/beats.rb:150:in `invoke'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-0.9.1-java/lib/concurrent/executor/executor_service.rb:515:in `run'", "Concurrent$$JavaExecutorService$$Job_1733552081.gen:13:in `run'"], :level=>:error}

No update available for the input so I'm not sure what else to look at. Everything is latest.

/etc/filebeat/filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - /var/log/*.log
      input_type: log
  registry_file: /var/lib/filebeat/registry
  config_dir:

output:
  elasticsearch:
    enabled: false
    hosts: ["localhost:9200"]

  logstash:
    enabled: true
    hosts: ["localhost:5044"]

shipper:

logstash.conf

input {
  beats {
   port => 5044
   type => "logs"
  }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Other than that everything is default. Not sure what else to try so I'm coming here. Thanks!

2

Hi, try to update the logstash beats plugin:

./bin/plugin update logstash-input-beats
1 Like
3

Hi Tudor,

After running that logstash doesn't start and throws the following error:

RuntimeError: Logstash expects concurrent-ruby version 0.9.1 and version 0.9.2 is installed, please verify this patch: /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/patches/silence_concurrent_ruby_warning.rb
 (root) at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/patches/silence_concurrent_ruby_warning.rb:53
require at org/jruby/RubyKernel.java:1040
 (root) at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/patches.rb:1
require at org/jruby/RubyKernel.java:1040
 (root) at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/patches.rb:6
require at org/jruby/RubyKernel.java:1040
 (root) at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/environment.rb:1
require at org/jruby/RubyKernel.java:1040
 (root) at /opt/logstash/lib/bootstrap/environment.rb:48

This happens with 1.5.5 and 2.0 versions of logstash.

4

@tudor

I went down the same path, with the same errors. I'm currently stuck at the version conflict error.

Ubuntu 14.04.3
Logstash 2.0.0-1
Filebeat 1.0.0~rc1

I also note that a fresh install of logstash-input-beats in a clean Logstash directory fetches 0.9.2, and updating it gets 0.9.4.

root@dev-kelp-ingress1:/opt/logstash# bin/plugin list beats ERROR: No plugins found

root@ingress1:/opt/logstash# bin/plugin install logstash-input-beats
Validating logstash-input-beats
Installing logstash-input-beats
Installation successful

root@ingress1:/opt/logstash# find . -name *beats*
./vendor/bundle/jruby/1.9/cache/logstash-input-beats-0.9.2.gem
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/logstash-input-beats.gemspec
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/spec/inputs/beats_spec.rb
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/spec/lumberjack/beats
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/logstash/inputs/beats.rb
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats.rb
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.2/lib/lumberjack/beats
./vendor/bundle/jruby/1.9/specifications/logstash-input-beats-0.9.2.gemspec

root@ingress1:/opt/logstash# bin/plugin update logstash-input-beats
Updating logstash-input-beats
Updated logstash-input-beats 0.9.2 to 0.9.4

root@ingress1:/opt/logstash# find . -name *beats*
./vendor/bundle/jruby/1.9/cache/logstash-input-beats-0.9.4.gem
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.4
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.4/logstash-input-beats.gemspec
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.4/spec/inputs/beats_spec.rb
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.4/spec/lumberjack/beats
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.4/lib/logstash/inputs/beats.rb
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.4/lib/lumberjack/beats.rb
./vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.4/lib/lumberjack/beats
./vendor/bundle/jruby/1.9/specifications/logstash-input-beats-0.9.4.gemspec

1 Like
5

@Matthew_Prinvale it might be an issue with the logstash releases not referencing the correct jruby version. At least I did read something like this, but don't remember where (maybe ask in Logstash forum). Until the issue is resolved you can try to running a recent logstash release (check past releases download page). Both Logstash 2.0.0-rc1 and 1.5.4 should work. In either case you still have to update logstash-input-beats to get most recent version of the plugin.

6

I tried a few different previous versions and got the same errors. Just tried 2.0.0-rc1 again and after updating the plugin got the error. Appreciate the suggestion though.

7

Hmm... weird. Our automated integration tests still run with logstash 1.5.4, but haven't seen this problem with 2.0rc1 yet. Maybe it's a problem with 0.9.4 beats plugin?

Now it gets ugly, but we can try to install 0.9.3 (untested shell commands, some flags might be wrongish):

# download 0.9.3 from rubygems
$ curl -O /tmp/logstash-input-beast-0.9.3gem https://rubygems.org/downloads/logstash-input-beats-0.9.3.gem

# uninstall current plugin (uninstall or remove?)
$ bin/plugin uninstall logstash-input-beats

# install 0.9.3
$ bin/plugin install /tmp/logstash-input-beast-0.9.3gem

All of this should not be necessary :frowning: I'm no export in Logstash and JRuby versioning. Maybe Logstash team has some more background on problems you are seeing.

8

@steffens => https://github.com/elastic/logstash/commit/972e13d52e103dfaf5f8ab5c33ceb852402cde49

Let's hope they release the patch soon!

9

plugin version 0.9.5 just published

10

@steffens Looks good. Thanks :smile:

11

Working for me as well! Thanks! Looking forward to continuing testing so we can hit the ground running when it goes RC.

12

Logging in to reply and say this fixed my issue.

  • Updated logstash-input-beats 0.9.2 to 0.9.6
  • Logstash 2.0.0
  • Filebeat version 1.0.0-rc1 (386)
13

Hi! Other unhandled exception

{:timestamp=>"2015-11-11T12:46:11.492000+0000", :message=>"Beats input: unhandled exception", :exception=>#<RuntimeError: unsupported protoc
ol 22>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/lib/lumberjack/beats/server.rb:225:in `handle_ve
rsion'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/lib/lumberjack/beats/server.rb:210:in `header'", "/opt/logst
ash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/lib/lumberjack/beats/server.rb:163:in `feed'", "/opt/logstash/vendor/bundle/jrub
y/1.9/gems/logstash-input-beats-0.9.6/lib/lumberjack/beats/server.rb:338:in `read_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logs
tash-input-beats-0.9.6/lib/lumberjack/beats/server.rb:315:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/
lib/logstash/inputs/beats.rb:169:in `invoke'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/concurre
nt-ruby-0.9.1-java/lib/concurrent/executor/executor_service.rb:515:in `run'", "Concurrent$$JavaExecutorService$$Job_1535659861.gen:13:in `ru
n'"], :level=>:error}

Logstash 2.0.0-1
Filebeat 1.0.0~rc1
on
Docker image cebp/ELK

I'm just a beginner, it may be it's very simple problem.

14

UPDATE: @Stanislav_Stolbov Ignore my previous message. I just saw that you are using input-beats-0.9.6. Have to check what the problem could be here.

15

@Stanislav_Stolbov can you share your beats and logstash config? Is any other process connecting to logstash port by accident? The exception happens on very first byte processed by protocol, which means some data not matching the protocol have been send to logstash.

16

filebeat.yml on win8x64

   prospectors:
       -
      paths:   
      - "e:/tmp/*.log"         
      input_type: log
      ignore_older: 87600h
                         
     registry_file: "C:/ProgramData/filebeat/registry"
     config_dir:
     
     output:

  logstash:
    enabled: true

    hosts:
    - 192.168.10.48:5044
    tls: 
       certificate: ./logstash-forwarder.crt
       certificate_key: ./logstash-forwarder.key
       certificate_authorities:
       - ./logstash-forwarder.crt
       timeout: 15

/etc/logstash/conf.d/02-filebeats.conf

input {
  beats {
    port => 5044
    type => "logs"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Logstash-forwarder is working on other port.

root@b099924718f8:/# find . -name beats
./opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6
./opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/spec/inputs/beats_spec.rb
./opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/spec/lumberjack/beats
./opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/lib/lumberjack/beats
./opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/lib/lumberjack/beats.rb
./opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/lib/logstash/inputs/beats.rb
./opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-0.9.6/logstash-input-beats.gemspec
./opt/logstash/vendor/bundle/jruby/1.9/cache/logstash-input-beats-0.9.6.gem
./opt/logstash/vendor/bundle/jruby/1.9/specifications/logstash-input-beats-0.9.6.gemspec
./etc/logstash/conf.d/02-filebeats.conf

17

problem solved with adding to filebeat.yml

elasticsearch:
enabled: false
hosts: ["192.168.10.48:9200"]

18

I see you've a configuration error. In logstash you configure the plugin with ssl disabled, but in filebeat you enable ssl. The value '22' is the first byte send by ssl handshake which was quite unexpected by input plugin. I agree the error message should be more descriptive in this case (will create an issue on input plugin).

In order to correctly configure TLS support you need to use the certificate and key on the logstash server side. The client (filebeat) only requires the certificate for validation (CA).

In filebeat.yml:

logstash:
  enabled: true
  hosts:
    - 192.168.10.48:5044
  tls:
    certificate_authorities:
      - ./logstash-forwarder.crt
    timeout: 15

in /etc/logstash/conf.d/02-filebeats.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/path/to/logstash-forwarder.crt"
    ssl_key => "/path/to/logstash-forwarder.key"
  }
}

be aware of 'type' config in logstash being of no effect due to all beats publishing an 'type' field (as type is special in logstash and will set type only if missing).

19

hello! I meet the same problem with you,so I want to know the problem whether is be solved or not and if solved,how to sovle it? thank you.

20

please open another discussion instead of hijacking a very old one.