Logstash crashing

Elastic Stack Logstash
1

Hi,

I have regulary my logstash server crashing.

Sometimes after a few restart it works again but alwais crash after a fex hours.

As i'm not a king in linux/logstash can you help me to find out what's wrong ?

Here is the logstas.log:

{:timestamp=>"2016-12-19T09:54:28.139000+0100", :message=>"Could not start TCP server: Address in use", :host=>"0.0.0.0", :port=>1514, :level=>:error}
{:timestamp=>"2016-12-19T09:54:28.169000+0100", :message=>"An unexpected error occurred!", :error=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :class=>"Errno::EADDRINUSE", :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in `initialize'", "org/jruby/RubyIO.java:853:in `new'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.4/lib/logstash/inputs/tcp.rb:244:in `new_server_socket'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.4/lib/logstash/inputs/tcp.rb:79:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:322:in `start_inputs'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:321:in `start_inputs'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:172:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:126:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/agent.rb:210:in `execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:90:in `run'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:95:in `run'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in `initialize'"], :level=>:warn}
~

2

Is another process listening on port 1514? Do you have multiple tcp inputs listing the same port somewhere in /etc/logstash/conf.d?

3

in /etc/logstash/conf.d i have only one listening on 1514.

How can i check if another process is listening that port ?

4

How can i check if another process is listening that port ?

Use netstat or lsof.

5

I've chec and i can't see any other app using port 1514

6

Actually the service is up again, but i can't see data in kibana.

I don't know if it can't receive it or if it didn't parse it.

7

Actually the service is up again, but i can't see data in kibana.

I don't know if it can't receive it or if it didn't parse it.

I just start it in debug:

Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory
3071
Try using the -Djava.io.tmpdir= option to select an alternate temp location.

8

Something looks to be connected but i have no idea what it is ...

tcp6 218 0 10.9.8.218:1514 184.39-65-87.adsl:59659 ESTABLISHED
tcp6 113 0 10.9.8.218:1514 184.39-65-87.adsl:65035 ESTABLISHED
tcp6 0 0 10.9.8.218:1514 184.39-65-87.adsl:65248 ESTABLISHED
tcp6 452 0 10.9.8.218:1514 184.39-65-87.adsl:55448 ESTABLISHED

9

Maybe a space issue ??

E: You don't have enough free space in /var/cache/apt/archives/.

Can i delete logs ? where are they stored ?

10

Can i delete logs ? where are they stored ?

Use the Elasticsearch REST APIs to delete indexes.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.