# Logstash giving error which is not clear

**URL:** <https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126>\
**Category:** Logstash\
**Created:** [June 3, 2023, 1:41am UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126 "2023-06-03T01:41:18Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Patr123](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@Patr123](https://discuss.elastic.co/u/Patr123)\
**Post date:** [June 3, 2023, 1:41am UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126/1 "2023-06-03T01:41:18Z")

</div>

I am getting the following error in logstash-plain.log:

```auto
[2023-06-03T01:33:34,256][INFO][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-06-03T01:33:34,272][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.17.4", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [linux-x86_64]"}
[2023-06-03T01:33:34,276][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2023-06-03T01:33:34,285][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2023-06-03T01:33:34,286][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x63d478a7 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2023-06-03T01:33:34,288][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2023-06-03T01:33:34,288][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x47e0e23f @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2023-06-03T01:33:34,329][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

```

My logstash.yml looks like:

```auto
path.data: /var/lib/logstash
pipeline.id: main
log.level: debug
path.logs: /var/log/logstash

```

The logstash service status looks ok and running.  
If I run my test.conf file located in /etc/logstash/conf.d/ with the command:

```auto
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf 

```

I get the output sent to elastic nodes and I can see the documents in kibana.  
But I still see the above errors in my logstash-plain.log file.  
Please help.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 3, 2023, 2:24am UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126/2 "2023-06-03T02:24:33Z")

</div>

> [@Patr123](#):
>
> `/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf `

Try this command and see if you get the same errors

`/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash`

Also did you try

`systemctl daemon-reload logstash`

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 3, 2023, 2:36am UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126/3 "2023-06-03T02:36:34Z")

</div>

> [@Patr123](#):
>
> ```auto
> [2023-06-03T01:33:34,329][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
> org.jruby.exceptions.SystemExit: (SystemExit) exit
> at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
> at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
> at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
> 
> ```

This is an error on the start of logstash, you need to check in your systems logs, `/var/log/messages` or `/var/log/syslog` depending on your distro.

Check the system logs and look for any hint about any issue with logstash.

> [@Patr123](#):
>
> The logstash service status looks ok and running

With the error you shared the service is not ok, it may be show as running/up when you run `systemctl status logstash`, but it is probably on a restart loop.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [June 3, 2023, 5:25am UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126/4 "2023-06-03T05:25:26Z")

</div>

If doesn't help what Leandro suggest, try with [Oracle JDK](https://www.oracle.com/my/java/technologies/downloads/) installation, not [OpenJDK](https://openjdk.org/) once I had similar issue and only Oracle JDK solve this. It might be related to java libs or paths which Oracle JDK set.

---

<div class="post-metadata">

**Author:** ![Patr123](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@Patr123](https://discuss.elastic.co/u/Patr123)\
**Post date:** [June 3, 2023, 6:14pm UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126/5 "2023-06-03T18:14:53Z")

</div>

I tried the below command and got the same errors in my logstash-plain.log.

```auto
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash

```

I tried daemon-reload logstash but got invalid number of arguments error message:

```auto
[root@server1 logstash]# systemctl daemon-reload logstash
Invalid number of arguments

```

---

<div class="post-metadata">

**Author:** ![Patr123](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@Patr123](https://discuss.elastic.co/u/Patr123)\
**Post date:** [June 3, 2023, 6:24pm UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126/6 "2023-06-03T18:24:58Z")

</div>

You were right.

1. I looked under `/var/log/messages` and I found the message `Your settings are invalid. Reason: Path "/var/lib/logstash/dead_letter_queue" must be a writable directory. It is not writable.`  
So I ran `chown logstash:logstash /var/lib/logstash/dead_letter_queue` and restarted the logstash service.
2. I checked the logstash-plain.log and got this error now:  
`An unexpected error occurred! {:error=>java.nio.file.AccessDeniedException: /var/lib/logstash/.lock, `  
To fix this I deleted the `.lock` file from `/var/lib/logstash` and restarted the logstash service.

After these I don't see any errors in my logstash-plain.log and now it's a good weekend 🙂  
Thank you for your help.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 3, 2023, 6:36pm UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126/7 "2023-06-03T18:36:30Z")

</div>

> [@stephenb](#):
>
> systemctl daemon-reload

Apologies command is, glad you got it working

---

<div class="post-metadata">

**Author:** ![Patr123](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@Patr123](https://discuss.elastic.co/u/Patr123)\
**Post date:** [June 3, 2023, 7:06pm UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126/8 "2023-06-03T19:06:59Z")

</div>

I tried that too, and then restarted the logstash service but it didn't fix the issue.  
Thank you for helping me on this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 1, 2023, 7:07pm UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126/9 "2023-07-01T19:07:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
