"Beats input: unhandled exception", :exception=>#RuntimeError: Unknown frame type: `48`

[rlonberg]

Hi All,

I've been running the ELK stack successfully for a few weeks now, my setup is:
filebeat+winlogbeat+topbeat on remote windows servers --to--> logstash on a linux env.

However, recently my logstash instance has been failing to process incoming logs. The logstash log shows the following:

`_{:timestamp=>"2016-09-23T11:32:40.564000-0400", :message=>**"Beats input: unhandled exception",** :exception=>#<RuntimeError: **Unknown frame type: `48`**>, :backtrace=>["/local/apps/software/logstash-1/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.9/lib/lumberjack/beats/server.rb:305:inheader'", "/local/apps/software/logstash-1/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.9/lib/lumberjack/beats/server.rb:251:in feed'", "/local/apps/software/logstash-1/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.9/lib/lumberjack/beats/server.rb:463:in read_socket'", "/local/apps/software/logstash-1/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.9/lib/lumberjack/beats/server.rb:443:in run'", "/local/apps/software/logstash-1/vendor/bundle/jruby/1.9/gems/logstash-input-beats 2.2.9/lib/logstash/inputs/beats_support/connection_handler.rb:34:in accept'", "/local/apps/software/logstash-1/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.9/lib/logstash/inputs/beats.rb:211:in handle_new_connection'", "/local/apps/software/logstash-1/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.9/lib/logstash/inputs/beats_support/circuit_breaker.rb:42:in execute'", "/local/apps/software/logstash-1/vendor/bundle/jruby/1.9/gems/log_`

... my logstash config looks like:
input {

  beats {
        port => 3501
    tags => ["IIS_Log"]
   }

  beats {
        port => 3502
    tags => ["Event_Log"]
   }

  beats {
        port => 3515
    tags => ["Perf_Metrics"]
   }
file {
        path => "/local/apps/lma1/lsnodes/logstash-shipper/logs/LMA1_NODE_LSSHIPPER01/LMA1_NODE_LSSHIPPER01.log"
        type => "Logstash_Log"
   }
}

And like I said, I'm having no connection issues on the forwarding side. The beats logs are all getting shipped to logstash, but logstash is failing to process.

Beats input: unhandled exception ... RuntimeError: Unknown frame type: '48'. What does this mean?

[steffens (Steffen Siering)]

I'm not sure, but there might be a race in LS potentially mixing read buffers for multiple connections. Have you tried upgrading LS to version 2.4?

[rlonberg]

Ok, so you were right about upgrading to 2.4, I was running 2.3.4 and after upgrading, my input errors disappeared.

My setup is logstash->redis->logstash->elasticsearch. I noticed that even though upgrading logstash fixed my input errors, I still wasn't seeing any new data on elasticsearch. Turns out all that corrupted data from earlier had overloaded redis, so after clearing out redis everything is OK.

Thank!