# Beats input: unhandled exception **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 13, 2016, 12:52pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101 "2016-01-13T12:52:55Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![axil\_76](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Post date:** [January 13, 2016, 12:52pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/1 "2016-01-13T12:52:55Z")
Hello I just migrate to filebeat logstash-forwarder, and I end up with the following error message `{:timestamp=>"2016-01-13T13:46:10.478000+0100", :message=>"Beats input: unhandled exception", :exception=>#, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:225:in`handle\_version'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:210:in `header'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:163:in`feed'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:342:in `read_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/lumberjack/beats/server.rb:319:in`run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.0.3/lib/logstash/inputs/beats.rb:184:in `invoke'", "org/jruby/RubyProc.java:281:in`call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-0.9.2-java/lib/concurrent/executor/executor\_service.rb:515:in `run'", "Concurrent$$JavaExecutorService$$Job_1598968662.gen:13:in`run'"], :level=\>:error}` What's the problem ? Thank you ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 13, 2016, 12:58pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/2 "2016-01-13T12:58:28Z")
Are you using Filebeat or logstash-forwarder to send data to Logstash's beats input? ---
**Author:** ![axil\_76](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Post date:** [January 13, 2016, 1:08pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/3 "2016-01-13T13:08:17Z")
> [@magnusbaeck](#): > > Are you using Filebeat or logstash-forwarder to send data to Logstash's beats input? I use filebeat to send logs to logstash ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 13, 2016, 1:09pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/4 "2016-01-13T13:09:55Z")
Have you enabled SSL on the Logstash side but not on the Filebeat side or vice versa? Please show your Filebeat and Logstash configurations. ---
**Author:** ![axil\_76](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Post date:** [January 13, 2016, 1:26pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/5 "2016-01-13T13:26:03Z")
> [@magnusbaeck](#): > > Have you enabled SSL on the Logstash side but not on the Filebeat side or vice versa? Please show your Filebeat and Logstash configurations. configuration filebeat #logstash: # The Logstash hosts hosts: ["10.0.0.45:5044","10.0.0.30:5044"] # Number of workers per Logstash host. worker: 2 # Optional load balance the events between the Logstash hosts loadbalance: true ``` # Optional index name. The default index name depends on the each beat. # For Packetbeat, the default is set to packetbeat, for Topbeat # top topbeat and for Filebeat to filebeat. #index: filebeat # Optional TLS. By default is off. #tls: # List of root certificates for HTTPS server verifications #certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for TLS client authentication #certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #certificate_key: "/etc/pki/client/cert.key" # Controls whether the client verifies server certificates and host name. # If insecure is set to true, all server host names and certificates will be # accepted. In this mode TLS based connections are susceptible to # man-in-the-middle attacks. Use only for testing. #insecure: true # Configure cipher suites to be used for TLS connections #cipher_suites: [] # Configure curve types for ECDHE based cipher suites #curve_types: [] ``` ### File as output Configuration logstash #Input logstash input { file { type =\> "syslog" path =\> ["/var/log/messages", "/var/log/\*.log"] tags =\> ["syslog"] } tcp { port =\> 514 type =\> syslog tags =\> ["syslog"] } udp { port =\> 514 type =\> syslog tags =\> ["syslog"] } tcp { port =\> 12530 type =\> syslog tags =\> ["netcat"] } udp { port =\> 12530 type =\> syslog tags =\> ["netcat"] } snmptrap { port =\> 162 type =\> 'snmptrap' host =\> "0.0.0.0" yamlmibdir =\> "/opt/logstash/vendor/bundle/jruby/1.9/gems/snmp-1.2.0/data/ruby/snmp/mibs/" tags =\> ["snmptrap"] } redis { key =\> phplogs data\_type =\> ['list'] } beats { port =\> 5044 tags =\> ["filebeat"] } } ---
**Author:** ![axil\_76](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Post date:** [January 13, 2016, 1:42pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/6 "2016-01-13T13:42:24Z")
Now I have much following error message ... `{:timestamp=>"2016-01-13T14:40:31.047000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 13, 2016, 2:30pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/7 "2016-01-13T14:30:07Z")
Your Logstash pipeline is clogged, probably because the outputs are failing. What kind of outputs do you have? ---
**Author:** ![axil\_76](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Post date:** [January 13, 2016, 2:51pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/8 "2016-01-13T14:51:50Z")
I add the following line in the configuration file logstash stdout { codec =\> rubydebug } but I have a lot of information for example { "@timestamp" =\> "2016-01-13T14:47:30.000Z", "tags" =\> [ [0] "nagios", [1] "\_grokparsefailure" ], "nagios\_epoch" =\> "1452696450", "nagios\_hostname" =\> "RPJ00201", "nagios\_service" =\> "ORA-ERR:ORA16001", "nagios\_state" =\> "2", "nagios\_type" =\> "EXTERNAL COMMAND", "nagios\_check\_result" =\> "ORA-12012: error on auto execute of job "V5CADM"."JOB\_APPLY\_GRANT\_CARTO"", "nagios\_command" =\> "PROCESS\_SERVICE\_CHECK\_RESULT", "hour" =\> "2016-01-13T14.000Z", "syslog\_severity\_code" =\> 5, "syslog\_facility\_code" =\> 1, "syslog\_facility" =\> "user-level", "syslog\_severity" =\> "notice", "syslog\_source-IP" =\> "[REY00060.spia.log.intra.laposte.fr](http://REY00060.spia.log.intra.laposte.fr)", "message-raw" =\> "[1452696450] EXTERNAL COMMAND: PROCESS\_SERVICE\_CHECK\_RESULT;RPJ00201;ORA-ERR:ORA16001;2;ORA-12012: error on auto execute of job "V5CADM"."JOB\_APPLY\_GRANT\_CARTO"" } how can you see the line that is the problem I recently add the following lines filter { if "filebeat" in [tags] { # hraccess log } grok { match =\> ["message", "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME} %{WORD:level} Authentication succeeded - Login ID \<%{USERNAME:user}\> - Hint - Remote IP address \<%{IP:client}/%{IP:client2}"] match =\> ["message", "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME} %{WORD:level} Authentication request received - Login ID \<%{USERNAME:useruknown}\>"] #match =\> ["message", "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME} %{WORD:level} %{GREEDYDATA:codeerror}%{IP:client}%{DATA:message}"] #match =\> ["message", "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME} %{WORD:level} %{GREEDYDATA:message}"] } ``` # } ``` ---
**Author:** ![axil\_76](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Post date:** [January 13, 2016, 3:28pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/9 "2016-01-13T15:28:26Z")
what does it mean ? `{:timestamp=>"2016-01-13T16:27:04.380000+0100", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 13, 2016, 3:31pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/10 "2016-01-13T15:31:52Z")
Is the stdout output really the only output? ---
**Author:** ![axil\_76](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Post date:** [January 13, 2016, 3:35pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/11 "2016-01-13T15:35:54Z")
> [@magnusbaeck](#): > > Is the stdout output really the only output? it is one of the outputs, I have many more, it was an example after a few minutes of operation I have the following message {:timestamp=\>"2016-01-13T16:35:53.392000+0100", :message=\>"CircuitBreaker::rescuing exceptions", :name=\>"Beats input", :exception=\>LogStash::SizedQueueTimeout::TimeoutError, :level=\>:warn} {:timestamp=\>"2016-01-13T16:35:53.394000+0100", :message=\>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=\>LogStash::CircuitBreaker::HalfOpenBreaker, :level=\>:warn} ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 13, 2016, 7:06pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/12 "2016-01-13T19:06:51Z")
Well, as I said the error message indicates that at least one output is having trouble which slows down the whole Logstash pipeline. With the information at hand I have nothing more to add. ---
**Author:** ![axil\_76](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Post date:** [January 14, 2016, 8:43am UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/13 "2016-01-14T08:43:33Z")
can you tell me how to find the part that slowed the processing of data ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [January 14, 2016, 1:39pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/14 "2016-01-14T13:39:10Z")
Have you checked one of your output target logs? With all outputs being affected by only one small output I'm not sure if you can easily debug this from logstash. Maybe people in logstash sub-forum have a better solution. You basically have to single out the connections slowing down logstash. some debugging in logstash: 1. remove(comment out) all outputs and add`null` output. If problem persists, filters are too slow (consider adding more filter workers?) 2. remove `null` output and enable first output. If problem persists, remove output and try another one 3. add outputs one after another until LS gets to slow. If LS gets to slow disable last one added and try another one. Always check resource usage CPU/memory when adding more outputs. Maybe one output is too slow, or you've got too many outputs slowing down logstash in general. If you've got too many outputs, try to disable/enable in groups first. Once you singled-out one or two outputs check the target (e.g. mysql server) for possible optimizations. Maybe you've got too many inputs too and try to process to many events overall in one logstash instance. Maybe your machine is not properly sized for your workload. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:56pm UTC](https://discuss.elastic.co/t/beats-input-unhandled-exception/39101/15 "2017-07-05T21:56:43Z")