# logstash-output-elasticsearch fails with Permission denied

**URL:** <https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787>\
**Category:** Logstash\
**Created:** [November 22, 2023, 6:37pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787 "2023-11-22T18:37:24Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 22, 2023, 6:37pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/1 "2023-11-22T18:37:24Z")

</div>

**Logstash information** :

1. Logstash version: 7.17.9
2. Logstash installation source: deb
3. How is Logstash being run: systemd
4. How was the Logstash Plugin installed: `sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch`
5. logstash-output-elasticsearch version: 11.19.0

**JVM** :  
Bundled JDK  
openjdk version "17.0.9" 2023-10-17  
OpenJDK Runtime Environment Temurin-17.0.9+9 (build 17.0.9+9)  
OpenJDK 64-Bit Server VM Temurin-17.0.9+9 (build 17.0.9+9, mixed mode, sharing)

**OS version** :  
Linux rpi4srv2 6.1.0-rpi6-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.58-1+rpt2 (2023-10-27) aarch64 GNU/Linux

**Description of the problem including expected versus actual behavior** :  
Plug-in fails with the error as show in the log excerpt and keeps retrying.  
Permissions look good.

/etc/logstash/conf.d/wazuh-elasticsearch.conf:

```auto
input {
  file {
    id => "wazuh_alerts"
    codec => "json"
    start_position => "beginning"
    stat_interval => "1 second"
    path => "/var/ossec/logs/alerts/alerts.json"
    mode => "tail"
    ecs_compatibility => "disabled"
  }
}

output {
    elasticsearch {
         hosts => "elasticsearch"
         index => "wazuh-alerts-4.x-%{+YYYY.MM.dd}"
         user => '${ELASTICSEARCH_USERNAME}'
         password => '${ELASTICSEARCH_PASSWORD}'
         ssl => true
         cacert => "/etc/logstash/certs/ca.cer"
         template => "/etc/logstash/templates/wazuh.json"
         template_name => "wazuh"
         template_overwrite => true
    }
}

```

Alerts file permissions:  
`-rw-rw---- 2 wazuh wazuh 447351 Nov 22 18:06 /var/ossec/logs/alerts/alerts.json`

logstash user is in wazuh group

**Provide logs (if relevant)**:

```auto
[2023-11-22T17:53:56,162][INFO][filewatch.observingtail][main][wazuh_alerts] START, creating Discoverer, Watch with file and sincedb collections
[2023-11-22T17:53:56,165][ERROR][logstash.javapipeline][main][wazuh_alerts] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File start_position=>"beginning", mode=>"tail", codec=><LogStash::Codecs::JSON id=>"json_8f7b39d1-927e-4299-ab3c-9d83575efb86", enable_metric=>true, charset=>"UTF-8">, path=>["/var/ossec/logs/alerts/alerts.json"], id=>"wazuh_alerts", stat_interval=>1.0, ecs_compatibility=>:disabled, enable_metric=>true, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc", exit_after_read=>false, check_archive_validity=>false>
  Error: Permission denied - Permission denied
  Exception: Errno::EACCES
  Stack: org/jruby/RubyFile.java:1323:in `utime'
/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/fileutils.rb:1132:in `block in touch'
org/jruby/RubyArray.java:1987:in `each'
/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/fileutils.rb:1129:in `touch'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.5/lib/filewatch/sincedb_collection.rb:22:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.5/lib/filewatch/observing_base.rb:62:in `build_watch_and_dependencies'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.5/lib/filewatch/observing_base.rb:56:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.5/lib/logstash/inputs/file.rb:352:in `start_processing'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.5/lib/logstash/inputs/file.rb:368:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:414:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 22, 2023, 6:39pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/2 "2023-11-22T18:39:09Z")

</div>

> [@mirceastoian](#):
>
> `sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch`

Just to be clear you do not need to install that plugin it comes pre-installed.

> [@mirceastoian](#):
>
> ```auto
> Error: Permission denied - Permission denied
> Exception: Errno::EACCES
> 
> ```

Think your errors is a permission on the file you are trying to read

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 22, 2023, 7:15pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/3 "2023-11-22T19:15:41Z")

</div>

Understood, thanks. I followed a Wazuh server integration guide using Logstash and it included this step (first time installing and configuring the stack).

The culprit file seems to be /var/ossec/logs/alerts/alerts.json and the problem is that I just went ahead and gave all permissions possible for the file and I still get this error.  
The Logstash process user should have all permissions but is unable to tail the file.

Other processes do:

`sudo lsof /var/ossec/logs/alerts/alerts.json`

```auto
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
wazuh-ana 1570657 wazuh 12w REG 179,2 487946 646565 /var/ossec/logs/alerts/alerts.json
filebeat 1604418 root 10r REG 179,2 487946 646565 /var/ossec/logs/alerts/alerts.json

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 22, 2023, 7:52pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/4 "2023-11-22T19:52:18Z")

</div>

> [@mirceastoian](#):
>
> The Logstash process user should have all permissions but is unable to tail the file.

What is the result of `sudo id logstash` ?

Also, did you restart the system after adding Logstash into the `wazuh` group?

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 22, 2023, 8:10pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/5 "2023-11-22T20:10:23Z")

</div>

> [@leandrojmp](#):
>
> What is the result of `sudo id logstash` ?

`uid=999(logstash) gid=992(logstash) groups=992(logstash),113(wazuh)`

> [@leandrojmp](#):
>
> Also, did you restart the system after adding Logstash into the `wazuh` group?

Rebooting did not help.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 22, 2023, 8:43pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/6 "2023-11-22T20:43:06Z")

</div>

Yeah, I'm not sure this issue is a Logstash issue or a systemd issue.

Can you stop your Logstash service and try to run it manually with the following command:

```auto
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings="/etc/logstash"

```

Share the logs it generates.

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 22, 2023, 8:54pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/7 "2023-11-22T20:54:13Z")

</div>

```auto
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2023-11-22T22:52:47,043][INFO][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-11-22T22:52:47,066][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.11.1", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.9+9 on 17.0.9+9 +indy +jit [aarch64-linux]"}
[2023-11-22T22:52:47,077][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-11-22T22:52:50,008][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::OrgLogstashSecretStore::SecretStoreException::LoadException", :message=>"Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore.", :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.load(JavaKeyStore.java:294)", "org.logstash.secret.store.backend.JavaKeyStore.load(JavaKeyStore.java:77)", "org.logstash.secret.store.SecretStoreFactory.doIt(SecretStoreFactory.java:129)", "org.logstash.secret.store.SecretStoreFactory.load(SecretStoreFactory.java:115)", "org.logstash.secret.store.SecretStoreExt.getIfExists(SecretStoreExt.java:60)", "org.logstash.execution.AbstractPipelineExt.getSecretStore(AbstractPipelineExt.java:790)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:238)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:173)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:516)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
[2023-11-22T22:52:50,074][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-11-22T22:52:50,120][INFO][logstash.runner] Logstash shut down.
[2023-11-22T22:52:50,142][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:795) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:758) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 22, 2023, 9:20pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/8 "2023-11-22T21:20:16Z")

</div>

> [@mirceastoian](#):
>
> `[2023-11-22T22:52:50,008][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::OrgLogstashSecretStore::SecretStoreException::LoadException", :message=>"Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore."`

I'm assuming you followed this [Wazuh post](https://wazuh.com/blog/detection-with-elastic-stack-integration/), right?

This will then make it more complicated to run manually.

I don't think this is a Logstash issue, the error is pretty clear, it cannot read the files, but since the logstash user also is on the `wazuh` group, I'm suspecting that this can be a systemd issue.

The `logstash.service` will run logstash as the `logstash` user and the `logstash` group, I do not have deep knowledge on systemd, but I'm supecting that this can be the issue.

The `logstash` user has access to the files, but the process is run strictly with the `logstash` group, so it cannot access the files.

Can you try the following:

```auto
sudo -u logstash tail -n2 /var/ossec/logs/alerts/alerts.json

```

And see if you it will show 2 lines from your `alerts.json`, if this works it means that indeed the `logstash` user can read the files owned by the `wazuh` group.

You may need to edit the logstash.service then, but I wait for confirmation that this works.

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 22, 2023, 9:39pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/9 "2023-11-22T21:39:36Z")

</div>

> [@leandrojmp](#):
>
> I'm assuming you followed this [Wazuh post](https://wazuh.com/blog/detection-with-elastic-stack-integration/), right?

Exactly.

> [@leandrojmp](#):
>
> Can you try the following:
> 
> ```auto
> sudo -u logstash tail -n2 /var/ossec/logs/alerts/alerts.json
> 
> ```
> 
> And see if you it will show 2 lines from your `alerts.json`, if this works it means that indeed the `logstash` user can read the files owned by the `wazuh` group.

It does indeed show 2 lines from the json file.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 22, 2023, 9:45pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/10 "2023-11-22T21:45:41Z")

</div>

So, try the following.

Edit the `logstash.service` and add the following line under the line that starts with `Group=logstash`

```auto
SupplementaryGroups=wazuh

```

To edit the service you can use:

```auto
vim /usr/lib/systemd/system/logstash.service

```

After that run `systemctl daemon-reload` and restart the logstash service.

If it still doesn't work I suggest that you check with the Wazuh community how they made it work, because I do not see any reason with Logstash, it is a permissions issue.

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 22, 2023, 9:53pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/11 "2023-11-22T21:53:18Z")

</div>

Unfortunately it's still the same.  
Thanks for your help, I will reach out to the Wazuh community.

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 24, 2023, 11:08am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/12 "2023-11-24T11:08:11Z")

</div>

Is there a reason why Logstash is running fine as a systemd service but when running in command line it fails complaining about the keystore?  
I rebuilt the keystore following guides and didn't help.

Also, trying to list the keystore results in this:

```auto
mircea@rpi4srv2:~ $ sudo -u logstash /usr/share/logstash/bin/logstash-keystore --path.settings="/etc/logstash" list
Using bundled JDK: /usr/share/logstash/jdk
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

Usage:
--------
bin/logstash-keystore [option] command [argument]

Commands:
--------
create - Creates a new Logstash keystore (e.g. bin/logstash-keystore create)
list - List entries in the keystore (e.g. bin/logstash-keystore list)
add - Add a value to the keystore (e.g. bin/logstash-keystore add my-secret)
remove - Remove a value from the keystore (e.g. bin/logstash-keystore remove my-secret)

Argument:
--------
--help - Display command specific help (e.g. bin/logstash-keystore add --help)

Options:
--------
--path.settings - Set the directory for the keystore. This is should be the same directory as the logstash.yml settings file. The default is the config directory under Logstash home. (e.g. bin/logstash-keystore --path.settings /tmp/foo create)

```

Somehow it does not use paths correctly. All needed files are there in /etc/logstash.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 24, 2023, 12:04pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/13 "2023-11-24T12:04:10Z")

</div>

> [@mirceastoian](#):
>
> Somehow it does not use paths correctly. All needed files are there in /etc/logstash.

You are running the wrong command line.

You are running this:

```auto
sudo -u logstash /usr/share/logstash/bin/logstash-keystore --path.settings="/etc/logstash" list

```

When it should be this:

```auto
sudo -u logstash /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list

```

The last line in the error message tells you the correct weay to run:

> Options:  
> --path.settings - Set the directory for the keystore. This is should be the same directory as the logstash.yml settings file. The default is the config directory under Logstash home. (e.g. **bin/logstash-keystore --path.settings /tmp/foo create** )

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 24, 2023, 12:30pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/14 "2023-11-24T12:30:13Z")

</div>

Oh, my bad.  
I just assumed it works like in this case:

`sudo -u logstash /usr/share/logstash/bin/logstash --path.settings="/etc/logstash"`

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 24, 2023, 12:50pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/15 "2023-11-24T12:50:40Z")

</div>

I recreated the keystore just to get the same error:

```auto
mircea@rpi4srv2:~ $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ELASTICSEARCH_USERNAME --allow-root
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

Enter value for ELASTICSEARCH_USERNAME:
Added 'elasticsearch_username' to the Logstash keystore.
mircea@rpi4srv2:~ $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ELASTICSEARCH_PASSWORD --allow-root
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

Enter value for ELASTICSEARCH_PASSWORD:
Added 'elasticsearch_password' to the Logstash keystore.

```

When listing the keystore:

```auto
mircea@rpi4srv2:~ $ sudo -u logstash /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

[2023-11-24T14:42:44,352][ERROR][logstash.secretstorecli] Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore. {:cause=>#<Java::JavaIo::IOException: Integrity check failed: java.io.IOException: getSecretKey failed: Password is not ASCII>, :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.load(org/logstash/secret/store/backend/JavaKeyStore.java:294)", "org.logstash.secret.store.backend.JavaKeyStore.load(org/logstash/secret/store/backend/JavaKeyStore.java:77)", "org.logstash.secret.store.SecretStoreFactory.doIt(org/logstash/secret/store/SecretStoreFactory.java:129)", "org.logstash.secret.store.SecretStoreFactory.load(org/logstash/secret/store/SecretStoreFactory.java:115)", "org.logstash.secret.cli.SecretStoreCli.command(org/logstash/secret/cli/SecretStoreCli.java:97)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:314)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:181)", "usr.share.logstash.lib.secretstore.cli.<module:SecretStoreCli>(/usr/share/logstash/lib/secretstore/cli.rb:52)", "usr.share.logstash.lib.secretstore.cli.<main>(/usr/share/logstash/lib/secretstore/cli.rb:33)", "usr.share.logstash.lib.secretstore.cli.run(usr/share/logstash/lib/secretstore//usr/share/logstash/lib/secretstore/cli.rb)", "java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:732)", "org.jruby.Ruby.runScript(org/jruby/Ruby.java:1278)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1195)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1177)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1213)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:991)", "org.jruby.Main.doRunFromMain(org/jruby/Main.java:398)", "org.jruby.Main.internalRun(org/jruby/Main.java:282)", "org.jruby.Main.run(org/jruby/Main.java:227)", "org.jruby.Main.main(org/jruby/Main.java:199)"]}

```

Any clue what could be wrong?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 24, 2023, 1:44pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/16 "2023-11-24T13:44:36Z")

</div>

Did you recreate a Keystore with a password or without a password?

The guide you are following uses a password for the Keystore, if you created it again following the same steps, you need to export the variable with the Keystore password for it to work.

it works as a service because the variable is in `/etc/sysconfig/logstash`, which is loaded by the systemd service.

If your keystore has a password try to run this way:

```auto
sudo -u logstash LOGSTASH_KEYSTORE_PASS=<MY_KEYSTORE_PASSWORD> /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list

```

To run logstash manually you would also need to use it:

```auto
sudo -u logstash LOGSTASH_KEYSTORE_PASS=<MY_KEYSTORE_PASSWORD> 
 /usr/share/logstash/bin/logstash --path.settings="/etc/logstash"

```

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 24, 2023, 2:03pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/17 "2023-11-24T14:03:13Z")

</div>

> [@leandrojmp](#):
>
> Did you recreate a Keystore with a password or without a password?
> 
> The guide you are following uses a password for the Keystore, if you created it again following the same steps, you need to export the variable with the Keystore password for it to work.

Yes, I always echo the variable to make sure it's there. And I used `sudo -E`.

> [@leandrojmp](#):
>
> To run logstash manually you would also need to use it:
> 
> ```auto
> sudo -u logstash LOGSTASH_KEYSTORE_PASS=<MY_KEYSTORE_PASSWORD> 
> /usr/share/logstash/bin/logstash --path.settings="/etc/logstash"
> 
> ```

Great, that did the trick. Thanks.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 24, 2023, 2:21pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/18 "2023-11-24T14:21:18Z")

</div>

> [@mirceastoian](#):
>
> Great, that did the trick. Thanks.

But did it work or you still get the permissiosn error?

---

<div class="post-metadata">

**Author:** ![mirceastoian](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mirceastoian](https://discuss.elastic.co/u/mirceastoian)\
**Post date:** [November 24, 2023, 2:46pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/19 "2023-11-24T14:46:15Z")

</div>

No, same permission error when running as service.  
But at least now I know the keystore is fine.

Strange thing, when I run Logstash like below I don't get an error anymore.  
There are some compatibility warnings I need to go through and understand but not sure they are related to this issue.

`sudo -E /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/wazuh-elasticsearch.conf --path.settings /etc/logstash/`

```auto
[2023-11-24T16:40:24,937][INFO][logstash.outputs.elasticsearch][main] Using mapping template from {:path=>"/etc/logstash/templates/wazuh.json"}
[2023-11-24T16:40:25,007][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/wazuh-elasticsearch.conf"], :thread=>"#<Thread:0x7b5264c1 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-11-24T16:40:25,133][INFO][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"wazuh"}
[2023-11-24T16:40:27,745][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=>2.73}
[2023-11-24T16:40:27,789][INFO][logstash.inputs.file][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_b6991da130c0919d87fbe36c3e98e363", :path=>["/var/ossec/logs/alerts/alerts.json"]}
[2023-11-24T16:40:27,800][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"}
[2023-11-24T16:40:27,838][INFO][filewatch.observingtail][main][wazuh_alerts] START, creating Discoverer, Watch with file and sincedb collections
[2023-11-24T16:40:27,854][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-11-24T16:40:28,054][INFO][logstash.codecs.json][main][wazuh_alerts] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 22, 2023, 2:46pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-fails-with-permission-denied/347787/20 "2023-12-22T14:46:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.