Logstash not working!

Mohyden · February 21, 2021, 12:29pm

Hello,

I'm running Ubuntu 16.04. When I try to start Logstash with systemctl start logstash, and then systemctl status logstash, it shows these:

logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Sun 2021-02-21 15:19:09 +03; 8min ago
Process: 19820 ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash (code=exited, status=1/FAILURE)
Main PID: 19820 (code=exited, status=1/FAILURE)

Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Unit entered failed state.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Failed with result 'exit-code'.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Feb 21 15:19:09 lthlibprod systemd[1]: Stopped logstash.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Start request repeated too quickly.
Feb 21 15:19:09 lthlibprod systemd[1]: Failed to start logstash.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Unit entered failed state.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Failed with result 'start-limit-hit'.

I'm trying to solve this issue for three days. Please help.

Thank you.

Badger · February 21, 2021, 5:54pm

What do the logstash logs show? What is your configuration?

Mohyden · February 22, 2021, 5:23am

Thank you Badger for your reply.

The logs shows:
[2021-02-21T14:38:35,105][INFO ][logstash.runner [2021-02-21T14:38:36,080][WARN ][logstash.config [2021-02-21T14:38:37,126][INFO ][org.reflections.Reflections] [2021-02-21T14:38:37,619][WARN ][logstash.javapipeline [2021-02-21T14:38:37,663][INFO ][logstash.javapipeline [2021-02-21T14:38:39,014][INFO ][logstash.javapipeline [2021-02-21T14:38:39,024][INFO ][logstash.inputs.beats [2021-02-21T14:38:39,065][INFO ][logstash.javapipeline [2021-02-21T14:38:39,123][INFO ][logstash.agent [2021-02-21T14:38:39,340][INFO ][logstash.agent [2021-02-21T14:38:39,342][INFO ][org.logstash.be [2021-02-21T14:39:02,432][WARN ][logstash.runner [2021-02-21T14:39:07,474][WARN ][logstash.runner to be processed. Sending another ^C will data loss.
[2021-02-21T14:39:09,641][INFO ][logstash.javapipeline [2021-02-21T14:39:10,538][INFO ][logstash.runner [2021-02-21T14:57:39,702][WARN ][logstash.runner [2021-02-21T14:57:39,793][INFO ][logstash.runner [2021-02-21T14:57:39,984][WARN ][logstash.config [2021-02-21T14:57:40,396][INFO ][org.reflections.Reflections] [2021-02-21T14:57:40,767][WARN ][logstash.javapipeline [2021-02-21T14:57:40,792][INFO ][logstash.javapipeline [2021-02-21T14:57:41,670][INFO ][logstash.javapipeline [2021-02-21T14:57:41,681][INFO ][logstash.inputs.beats [2021-02-21T14:57:41,701][INFO ][logstash.javapipeline [2021-02-21T14:57:41,749][INFO ][logstash.agent [2021-02-21T14:57:41,880][INFO ][logstash.agent [2021-02-21T14:57:41,903][INFO ][org.logstash.be [2021-02-21T15:05:36,539][WARN ][logstash.runner [2021-02-21T15:05:41,558][WARN ][logstash.runner to be processed. Sending another ^C will data loss.
[2021-02-21T15:05:42,863][INFO ][logstash.javapipeline [2021-02-21T15:05:43,610][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.241-b07 on 1.8.0_241-b07 +jit [linux-x86_64]"}
.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Reflections took 94 ms to scan 1 urls, producing 23 keys and 47 values
][main] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/etc/logstash/conf.d/02-beats-people_counter-input.conf"], :thread=>"#<Thread:0x7d17809c@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
][main] Pipeline Java execution initialization time {"seconds"=>1.34}
][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
][main] Pipeline started {"pipeline.id"=>"main"}
] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
] Successfully started Logstash API endpoint {:port=>9600}
ats.Server][main][34c9226aa720892fcffaf85222a301e5e875c08f4df5a512eda79a67b10f3218] Starting server on port: 5043
] SIGINT received. Shutting down.
] Received shutdown signal, but pipeline is still waiting for in-flight events
force quit Logstash, but this may cause
][main] Pipeline terminated {"pipeline.id"=>"main"}
] Logstash shut down.
] --config.debug was specified, but log.level was not set to 'debug'! No config info will be logged.
] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.241-b07 on 1.8.0_241-b07 +jit [linux-x86_64]"}
.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Reflections took 102 ms to scan 1 urls, producing 23 keys and 47 values
][main] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/etc/logstash/conf.d/02-beats-people_counter-input.conf"], :thread=>"#<Thread:0x28a22f0@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
][main] Pipeline Java execution initialization time {"seconds"=>0.88}
][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
][main] Pipeline started {"pipeline.id"=>"main"}
] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
] Successfully started Logstash API endpoint {:port=>9600}
ats.Server][main][34c9226aa720892fcffaf85222a301e5e875c08f4df5a512eda79a67b10f3218] Starting server on port: 5043
] SIGINT received. Shutting down.
] Received shutdown signal, but pipeline is still waiting for in-flight events
force quit Logstash, but this may cause
][main] Pipeline terminated {"pipeline.id"=>"main"}
] Logstash shut down.

This is my configuration:
node.name: logstashProd
path.data: /data/logstash
pipeline.workers: 1
path.config: /etc/logstash/conf.d
config.debug: true
path.logs: /var/log/logstash

And, here is the "path.config: /etc/logstash/conf.d":
input {
beats {
port => "5043"
ssl => false
}
}
file {
path => "/people-count-data.txt"
codec => line { format => "%{message}"}
}
}

Badger · February 22, 2021, 2:40pm

Something external to logstash is telling it to exit.

Mohyden · February 23, 2021, 6:11am

Is there any way to solve it out?

grumo35 · February 23, 2021, 10:28am

It happens sometimes to me when i try to restart the service or kill it using kill -9

You might want to restart the host or try to "pkill logstash" command.

Like badger is saying something is sending SIGINT signals to logstashs service.

Mohyden · February 23, 2021, 11:00am

Thank you grumo35.

I tried pkill logstash and it doesn't work, even I tried to use kill -9 {process ID} and killall logstash and it throws no process found

The status still as it is:
sudo systemctl status logstash
logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Sun 2021-02-21 15:19:09 +03; 1 day 22h ago
Process: 19820 ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash (code=exited, status=1/FAILURE)
Main PID: 19820 (code=exited, status=1/FAILURE)

Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Unit entered failed state.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Failed with result 'exit-code'.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Feb 21 15:19:09 lthlibprod systemd[1]: Stopped logstash.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Start request repeated too quickly.
Feb 21 15:19:09 lthlibprod systemd[1]: Failed to start logstash.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Unit entered failed state.
Feb 21 15:19:09 lthlibprod systemd[1]: logstash.service: Failed with result 'start-limit-hit'.
Feb 23 13:53:04 lthlibprod systemd[1]: Stopped logstash.

grumo35 · March 1, 2021, 10:30am

What distribution you use ? Do you have any cron or scripts that runs on that machine ?

Please be aware that your logstash process is stopping and starting and might just be misconfigured.

The command you type does in fact kill the process but it just restart by itself using systemd.

What's the configuration of the service in systemd ? And please show the output or journalctl -ru logstash

rojin · March 5, 2021, 5:39pm

Hello,
You might wanna check the logstash pipelines (input, filter, output). Also check the ports to be open for the connection Sharing logs would be helpful.

Mohyden · March 7, 2021, 5:17am

Thank you all for your help and support. Our central IT team solve it for us.

system · April 4, 2021, 5:18am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash not working when started via systemctl, however it work when manually starting Logstash	1	986	January 14, 2021
Systemctl status logstash Logstash	2	6539	March 27, 2018
Start up logstash Logstash	3	3109	June 4, 2018
Logstash status: Failed to start logstash Logstash	10	11520	February 25, 2020
When logstash is stopped it goes to failed state Logstash	1	1211	July 21, 2017

Logstash not working!

Related topics