Logstash not working!

Elastic Stack Logstash
1

Hello,

I'm running Ubuntu 16.04. When I try to start Logstash with systemctl start logstash, and then systemctl status logstash, it shows these:

logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Sun 2021-02-21 15:19:09 +03; 8min ago
Process: 19820 ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash (code=exited, status=1/FAILURE)
Main PID: 19820 (code=exited, status=1/FAILURE)

Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Unit entered failed state.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Failed with result 'exit-code'.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Feb 21 15:19:09 lthlibprod systemd[1]: Stopped logstash.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Start request repeated too quickly.
Feb 21 15:19:09 lthlibprod systemd[1]: Failed to start logstash.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Unit entered failed state.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Failed with result 'start-limit-hit'.

I'm trying to solve this issue for three days. Please help.

Thank you.

2

What do the logstash logs show? What is your configuration?

1 Like
3

Thank you Badger for your reply.

The logs shows:
[2021-02-21T14:38:35,105][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.241-b07 on 1.8.0_241-b07 +jit [linux-x86_64]"}
[2021-02-21T14:38:36,080][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-02-21T14:38:37,126][INFO ][org.reflections.Reflections] Reflections took 94 ms to scan 1 urls, producing 23 keys and 47 values
[2021-02-21T14:38:37,619][WARN ][logstash.javapipeline ][main] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2021-02-21T14:38:37,663][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/etc/logstash/conf.d/02-beats-people_counter-input.conf"], :thread=>"#<Thread:0x7d17809c@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[2021-02-21T14:38:39,014][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.34}
[2021-02-21T14:38:39,024][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
[2021-02-21T14:38:39,065][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-02-21T14:38:39,123][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2021-02-21T14:38:39,340][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-02-21T14:38:39,342][INFO ][org.logstash.beats.Server][main][34c9226aa720892fcffaf85222a301e5e875c08f4df5a512eda79a67b10f3218] Starting server on port: 5043
[2021-02-21T14:39:02,432][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2021-02-21T14:39:07,474][WARN ][logstash.runner ] Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss.
[2021-02-21T14:39:09,641][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2021-02-21T14:39:10,538][INFO ][logstash.runner ] Logstash shut down.
[2021-02-21T14:57:39,702][WARN ][logstash.runner ] --config.debug was specified, but log.level was not set to 'debug'! No config info will be logged.
[2021-02-21T14:57:39,793][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.241-b07 on 1.8.0_241-b07 +jit [linux-x86_64]"}
[2021-02-21T14:57:39,984][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-02-21T14:57:40,396][INFO ][org.reflections.Reflections] Reflections took 102 ms to scan 1 urls, producing 23 keys and 47 values
[2021-02-21T14:57:40,767][WARN ][logstash.javapipeline ][main] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2021-02-21T14:57:40,792][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/etc/logstash/conf.d/02-beats-people_counter-input.conf"], :thread=>"#<Thread:0x28a22f0@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[2021-02-21T14:57:41,670][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.88}
[2021-02-21T14:57:41,681][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
[2021-02-21T14:57:41,701][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-02-21T14:57:41,749][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2021-02-21T14:57:41,880][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-02-21T14:57:41,903][INFO ][org.logstash.beats.Server][main][34c9226aa720892fcffaf85222a301e5e875c08f4df5a512eda79a67b10f3218] Starting server on port: 5043
[2021-02-21T15:05:36,539][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2021-02-21T15:05:41,558][WARN ][logstash.runner ] Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss.
[2021-02-21T15:05:42,863][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2021-02-21T15:05:43,610][INFO ][logstash.runner ] Logstash shut down.

This is my configuration:
node.name: logstashProd
path.data: /data/logstash
pipeline.workers: 1
path.config: /etc/logstash/conf.d
config.debug: true
path.logs: /var/log/logstash

And, here is the "path.config: /etc/logstash/conf.d":
input {
beats {
port => "5043"
ssl => false
}
}
file {
path => "/people-count-data.txt"
codec => line { format => "%{message}"}
}
}

4

Something external to logstash is telling it to exit.

5

Is there any way to solve it out?

6

It happens sometimes to me when i try to restart the service or kill it using kill -9

You might want to restart the host or try to "pkill logstash" command.

Like badger is saying something is sending SIGINT signals to logstashs service.

7

Thank you grumo35.

I tried pkill logstash and it doesn't work, even I tried to use kill -9 {process ID} and killall logstash and it throws no process found

The status still as it is:
sudo systemctl status logstash
logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Sun 2021-02-21 15:19:09 +03; 1 day 22h ago
Process: 19820 ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash (code=exited, status=1/FAILURE)
Main PID: 19820 (code=exited, status=1/FAILURE)

Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Unit entered failed state.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Failed with result 'exit-code'.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Feb 21 15:19:09 lthlibprod systemd[1]: Stopped logstash.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Start request repeated too quickly.
Feb 21 15:19:09 lthlibprod systemd[1]: Failed to start logstash.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Unit entered failed state.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Failed with result 'start-limit-hit'.
Feb 23 13:53:04 lthlibprod systemd[1]: Stopped logstash.
8

What distribution you use ? Do you have any cron or scripts that runs on that machine ?

Please be aware that your logstash process is stopping and starting and might just be misconfigured.

The command you type does in fact kill the process but it just restart by itself using systemd.

What's the configuration of the service in systemd ? And please show the output or journalctl -ru logstash

9

Hello,
You might wanna check the logstash pipelines (input, filter, output). Also check the ports to be open for the connection :thinking: Sharing logs would be helpful.

10

Thank you all for your help and support. Our central IT team solve it for us.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.