Logstash crashing


(Karl Trasschaert) #1

Hi,

I have regulary my logstash server crashing.

Sometimes after a few restart it works again but alwais crash after a fex hours.

As i'm not a king in linux/logstash can you help me to find out what's wrong ?

Here is the logstas.log:

{:timestamp=>"2016-12-19T09:54:28.139000+0100", :message=>"Could not start TCP server: Address in use", :host=>"0.0.0.0", :port=>1514, :level=>:error}
{:timestamp=>"2016-12-19T09:54:28.169000+0100", :message=>"An unexpected error occurred!", :error=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :class=>"Errno::EADDRINUSE", :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in initialize'", "org/jruby/RubyIO.java:853:innew'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.4/lib/logstash/inputs/tcp.rb:244:in new_server_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.4/lib/logstash/inputs/tcp.rb:79:inregister'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:322:in start_inputs'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:321:in start_inputs'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:172:instart_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:126:in run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/agent.rb:210:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:90:in run'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:95:in run'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in `initialize'"], :level=>:warn}
~


(Magnus Bäck) #2

Is another process listening on port 1514? Do you have multiple tcp inputs listing the same port somewhere in /etc/logstash/conf.d?


(Karl Trasschaert) #3

in /etc/logstash/conf.d i have only one listening on 1514.

How can i check if another process is listening that port ?


(Magnus Bäck) #4

How can i check if another process is listening that port ?

Use netstat or lsof.


(Karl Trasschaert) #5

I've chec and i can't see any other app using port 1514


(Karl Trasschaert) #6

Actually the service is up again, but i can't see data in kibana.

I don't know if it can't receive it or if it didn't parse it.


(Karl Trasschaert) #7

Actually the service is up again, but i can't see data in kibana.

I don't know if it can't receive it or if it didn't parse it.

I just start it in debug:

Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory
3071
Try using the -Djava.io.tmpdir= option to select an alternate temp location.


(Karl Trasschaert) #8

Something looks to be connected but i have no idea what it is ...

tcp6 218 0 10.9.8.218:1514 184.39-65-87.adsl:59659 ESTABLISHED
tcp6 113 0 10.9.8.218:1514 184.39-65-87.adsl:65035 ESTABLISHED
tcp6 0 0 10.9.8.218:1514 184.39-65-87.adsl:65248 ESTABLISHED
tcp6 452 0 10.9.8.218:1514 184.39-65-87.adsl:55448 ESTABLISHED


(Karl Trasschaert) #9

Maybe a space issue ??

E: You don't have enough free space in /var/cache/apt/archives/.

Can i delete logs ? where are they stored ?


(Magnus Bäck) #10

Can i delete logs ? where are they stored ?

Use the Elasticsearch REST APIs to delete indexes.


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.