# Lostash becoming unresponsive (?)

**URL:** <https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242>\
**Category:** Logstash\
**Created:** [September 29, 2017, 10:26am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242 "2017-09-29T10:26:01Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\
**Post date:** [September 29, 2017, 10:26am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/1 "2017-09-29T10:26:01Z")

</div>

I am trying to start `logstash` via

```
sudo systemctl start logstash

```

It seems to succeed (service status now is `active`), however nothing gets written in `logstash`'s own logs, under `/var/log/logstash/logstash-plain.log`

Until a few minutes ago, starting / stopping the `logstash` service did produce output in the above log. (e.g. with the respective informative messages on the service having been started or stopped, or the error messages when I messed up come config files).

The system (`debian 8`) is indeed poor on resources (4GB / 1 Xeon core) however the top command shows log cpu consumption and 1G still available, after starting `logstash`.

Any suggestions about how can I troubleshoot this?

---

<div class="post-metadata">

**Author:** ![Vishal\_Sharma1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_sharma1/32/20207_2.png) [@Vishal\_Sharma1](https://discuss.elastic.co/u/Vishal_Sharma1)\
**Post date:** [September 29, 2017, 10:31am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/2 "2017-09-29T10:31:42Z")

</div>

well ... do you have enough events generating for logstash to generate the logs ?

---

<div class="post-metadata">

**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\
**Post date:** [September 29, 2017, 10:35am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/3 "2017-09-29T10:35:58Z")

</div>

Well (at least up until a while ago), I did not need any events to see things happening in `/var/log/logstash/logstash-plain.log`.

Starting and stopping the service was enough to make the file change.

What is more, the client services that are supposed to be sending logs to my `logstash` instance and on the specific ports I am using, fail to do so (connection refused).

---

<div class="post-metadata">

**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\
**Post date:** [September 29, 2017, 10:57am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/4 "2017-09-29T10:57:03Z")

</div>

dunno if this helps:

```
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled)
   Active: active (running) since Fri 2017-09-29 13:55:02 EEST; 51s ago
 Main PID: 112945 (java)
   CGroup: /system.slice/logstash.service
           └─112945 /usr/bin/java -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby....

Sep 29 13:55:02 myhostname systemd[1]: logstash.service holdoff time over, scheduling restart.
Sep 29 13:55:02 myhostname systemd[1]: Stopping logstash...
Sep 29 13:55:02 myhostname systemd[1]: Starting logstash...
Sep 29 13:55:02 myhostname systemd[1]: Started logstash.

```

It seems to be keeping rescheduling restarts due to some `holdoff` timer expiring...

---

<div class="post-metadata">

**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\
**Post date:** [September 29, 2017, 11:19am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/5 "2017-09-29T11:19:01Z")

</div>

```
$ sudo journalctl -f -u logstash
-- Logs begin at Mon 2017-09-18 17:11:57 EEST. --
Sep 29 14:02:21 docker-elk01 systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
Sep 29 14:02:21 myhostname systemd[1]: Unit logstash.service entered failed state.
Sep 29 14:02:21 myhostname systemd[1]: logstash.service holdoff time over, scheduling restart.
Sep 29 14:02:21 myhostname systemd[1]: Stopping logstash...
Sep 29 14:02:21 myhostname systemd[1]: Starting logstash...
Sep 29 14:02:21 myhostname systemd[1]: Started logstash.
Sep 29 14:03:19 myhostname systemd[1]: Stopping logstash...
Sep 29 14:03:20 myhostname systemd[1]: logstash.service: main process exited, code=exited, status=143/n/a
Sep 29 14:03:20 myhostname systemd[1]: Stopped logstash.
Sep 29 14:03:20 myhostname systemd[1]: Unit logstash.service entered failed state.
Sep 29 14:07:55 myhostname systemd[1]: Starting logstash...
Sep 29 14:07:55 myhostname systemd[1]: Started logstash.
Sep 29 14:09:26 myhostname logstash[115716]: WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Sep 29 14:09:27 myhostname logstash[115716]: Could not find log4j2 configuration at path //etc/logstash/log4j2.properties. Using default config which logs to console
Sep 29 14:09:27 myhostname logstash[115716]: 14:09:27.397 [main] FATAL logstash.runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
Sep 29 14:09:27 myhostname systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
Sep 29 14:09:27 myhostname systemd[1]: Unit logstash.service entered failed state.
Sep 29 14:09:27 myhostname systemd[1]: logstash.service holdoff time over, scheduling restart.
```

---

<div class="post-metadata">

**Author:** ![Vishal\_Sharma1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_sharma1/32/20207_2.png) [@Vishal\_Sharma1](https://discuss.elastic.co/u/Vishal_Sharma1)\
**Post date:** [September 29, 2017, 11:22am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/6 "2017-09-29T11:22:13Z")

</div>

[main] FATAL logstash.runner - An unexpected error occurred! {:error=\>#\<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.\>

As per your logs make above writable for logstash and restart the service. Let me know if this solved the issue.

---

<div class="post-metadata">

**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\
**Post date:** [September 29, 2017, 11:23am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/7 "2017-09-29T11:23:54Z")

</div>

I did, but now I get this:

```
Sep 29 14:22:35 myhostname logstash[118552]: WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Sep 29 14:22:35 myhostname logstash[118552]: Could not find log4j2 configuration at path //etc/logstash/log4j2.properties. Using default config which logs to console
Sep 29 14:22:35 myhostname logstash[118552]: ERROR: No configuration file was specified. Perhaps you forgot to provide the '-f yourlogstash.conf' flag?
Sep 29 14:22:35 myhostname logstash[118552]: usage:
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash --help
Sep 29 14:22:36 myhostname systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
```

---

<div class="post-metadata">

**Author:** ![Vishal\_Sharma1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_sharma1/32/20207_2.png) [@Vishal\_Sharma1](https://discuss.elastic.co/u/Vishal_Sharma1)\
**Post date:** [September 29, 2017, 11:25am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/8 "2017-09-29T11:25:56Z")

</div>

do you have a conf file in /etc/logstash/conf.d ?  
see below from your logs  
ERROR: No configuration file was specified. Perhaps you forgot to provide the '-f yourlogstash.conf' flag?

---

<div class="post-metadata">

**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\
**Post date:** [September 29, 2017, 11:52am UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/9 "2017-09-29T11:52:40Z")

</div>

Ιt turns out I changed ownership of `/etc/logstash` by mistake ... giving it to `root:root`.

This messed up everything.

Apologies for the hassle...

---

<div class="post-metadata">

**Author:** ![Vishal\_Sharma1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_sharma1/32/20207_2.png) [@Vishal\_Sharma1](https://discuss.elastic.co/u/Vishal_Sharma1)\
**Post date:** [September 29, 2017, 12:05pm UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/10 "2017-09-29T12:05:13Z")

</div>

ah actually i was just checking the log and I missed it as well ☹

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 27, 2017, 12:05pm UTC](https://discuss.elastic.co/t/lostash-becoming-unresponsive/102242/11 "2017-10-27T12:05:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
