# Logstash security best practices: keystore, monitoring, ssl\_verification\_mode
**URL:** https://discuss.elastic.co/t/logstash-security-best-practices-keystore-monitoring-ssl-verification-mode/365166
**Category:** Logstash
**Tags:** elastic-stack-monitoring, elastic-stack-security
**Created:** [August 19, 2024, 7:49pm UTC](https://discuss.elastic.co/t/logstash-security-best-practices-keystore-monitoring-ssl-verification-mode/365166 "2024-08-19T19:49:00Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@Abdarrahmane](https://discuss.elastic.co/u/Abdarrahmane)
#### Post date: [August 19, 2024, 7:49pm UTC](https://discuss.elastic.co/t/logstash-security-best-practices-keystore-monitoring-ssl-verification-mode/365166/1 "2024-08-19T19:49:00Z")
hello !
can you guys please guide me to the best security practices to secure the communication between Logstash and elasticsearch (logstash configuration (`logstash.yml`) and its pipelines in the `conf.d`)
here is my current `Logstash.yml` configuration:
```auto
node.name: Logstash01
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: 4+=thEApVTmKZ4d4
xpack.monitoring.elasticsearch.hosts: ["https://192.168.1.14:9200", "https://192.168.1.15:9200", "https://192.168.1.16:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/certs/elasticsearch-ca.pem
xpack.monitoring.elasticsearch.ssl.verification_mode: none
```
and here is an example of a pipeline I'm using `pfsense-logs.conf`
```auto
input {
udp {
port => 514
type => "syslog"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGLINE}" }
}
date {
match => ["timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
}
}
}
output {
elasticsearch {
hosts => ["https://192.168.1.14:9200", "https://192.168.1.15:9200", "https://192.168.1.16:9200"]
index => "Test-pfsense-syslog-%{+YYYY.MM.dd}"
user => "elastic"
password => "4+=thEApVTmKZ4d42NU7"
ssl => true
cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
}
stdout { codec => rubydebug }
}
```
like how can I generate a certificate for my logstash node and maybe how can I secretly store my elasticsearch password and call it as a variable
please suggest !
---
### Author:  [@Abdarrahmane](https://discuss.elastic.co/u/Abdarrahmane)
#### Post date: [August 22, 2024, 1:39am UTC](https://discuss.elastic.co/t/logstash-security-best-practices-keystore-monitoring-ssl-verification-mode/365166/2 "2024-08-22T01:39:48Z")
I've done some research, and i hope this help people who might have the same question as me 😅
if u guys have some improvement or remarks please suggest !
for your logstash node to work properly after installing it and be able to monitor it u have to write a `pipeline.conf` in the `conf.d` directory `(/etc/logstash/conf.d)` give it the appropriate permissions/user,group (I wrote a pipeline for my logs from my pfsense firewall to logstash and then output them to elasticsearch)
this wasn't in the documentation, I would be so grateful if u guys can add it to it
optionally set the JVM heap size in `jvm.options` to half of your server's ram I have 8 so 4 `-Xms4g` and `-Xmx4g`
the best way to monitor logstash in my opinion is using metricbeat as I felt like elastic agent+logstash integration were not quite mature yet
here is how to set I up [Collect Logstash monitoring data with Metricbeat | Logstash Reference [8.15] | Elastic](https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html)
for the cluster\_uuid:
```auto
monitoring.cluster_uuid: PRODUCTION_ES_CLUSTER_UUID
```
if you wrote a pipeline and it worked properly logstash should be able to identify the cluster id without any issue but sometimes you will have to precise it in case you are using a dedicated monitoring cluster especially if u have a non basic licence
and you should be able to monitor it under stack monitoring in kibana
for `logstash.yml`
the verification modes that worked for me are `full` and `none` for `certificate` I guess it's not supported yet take a look at this [Elasticsearch output plugin | Logstash Reference [8.15] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ssl_verification_mode)
so to avoid using `none` I have to generate an SSL certificate for my logstash node using its hostname and IP address. for that I used the following commands :
```auto
./bin/elasticsearch-certutil cert \
--name logstash \
--ca-cert /path/to/ca/cert.crt \
--ca-key /path/to/ca/private.key \
--dns your.host.name.here \
--ip 192.168.1.10 \
--pem
```
if your certificate is in pem `elasticsearch-ca.pem` you can convert your CA `elastic-stack-ca.p12` to get `cert.crt` and `private.key`
here is the commands I used:
```auto
openssl pkcs12 -in elastic-stack-ca.p12 -out cert.crt -clcerts -nokeys
openssl pkcs12 -in elastic-stack-ca.p12 -out private.key -nocerts -nodes
```
so my `logstash.yml` file looks like this:
```auto
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: XLo75i-zJblqwd50X9ZF
xpack.monitoring.elasticsearch.hosts: ["https://192.168.1.14:9200", "https://192.168.1.15:9200", "https://192.168.1.16:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/certs/elasticsearch-ca.pem
xpack.monitoring.elasticsearch.ssl.verification_mode: full
xpack.monitoring.elasticsearch.ssl.certificate: /etc/logstash/certs/logstash.crt
xpack.monitoring.elasticsearch.ssl.key: /etc/logstash/certs/logstash.key
```
for the username and password it's better to use the dedicated `logstash_system` for `logstash.yml` so I went and reset its default password using:
```auto
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u -i logstash_sytem
```
the `-i` for a password u input it otherwise u can use delete it to get a random password
u can ofc do this from kibana UI or from its dev tools too
same thing for metricbeat I used the default `remote_monitoring_user` that I reseted its password using the same command
```auto
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u -i remote_monitoring_user
```
finally for the pipeline in instead of using the elastic super user I created a role called `logstash_writer_role`
and then a user called `logstash_writer` and gave it that role more on that here (how to create it ...) [Secure your connection to Elasticsearch | Logstash Reference [8.15] | Elastic](https://www.elastic.co/guide/en/logstash/current/ls-security.html#ls-http-auth-basic)
the only this to pay attention to it is in :
```auto
"indices": [
{
"names": ["logstash-*"],
```
you pipeline's index should start with the same `logstash-...`
in my case :
```auto
output {
elasticsearch {
hosts => ["https://192.168.1.14:9200", "https://192.168.1.15:9200", "https://192.168.1.16:9200"]
index => "logstash-pfsense-syslog-%{+YYYY.MM.dd}"
user => "logstash_writer"
password => "password"
ssl => true
cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
}
stdout { codec => rubydebug }
}
```
a better way is to use API keys instead of usernames and password where you can limit permissions of the API keys... as mentioned here [Secure your connection to Elasticsearch | Logstash Reference [8.15] | Elastic](https://www.elastic.co/guide/en/logstash/current/ls-security.html#ls-create-api-key)
or encrypt the username and password and call them as variables in the `pipeline.conf` or `logstash.yml` but I wasn't able to get it to work properly
[Secrets keystore for secure settings | Logstash Reference [8.15] | Elastic](https://www.elastic.co/guide/en/logstash/current/keystore.html)
here is the command I used
to create my keystore:
```auto
set +o history
export LOGSTASH_KEYSTORE_PASS=mypassword
set -o history
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
```
and then to add my variables:
```auto
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_USER
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_PWD
```
and then list them:
```auto
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list
```
give the `logstash.keystore` file the appropriate permissions/user.group
and in the pipeline.conf and yml files use
```auto
output {
elasticsearch {
...
user => "${ES_USER}"
password => "${ES_PWD}"
...
}
}
```
here is the the output of the keystone creation :
```auto
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:29:55,998 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:29:55,999 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:29:55,999 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:29:56,000 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2024-08-22T01:29:56,574][INFO][org.logstash.secret.store.backend.JavaKeyStore] Created Logstash keystore at /etc/logstash/logstash.keystore
Created Logstash keystore at /etc/logstash/logstash.keystore
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_USER
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:30:14,473 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:30:14,474 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:30:14,475 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:30:14,475 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Enter value for ES_USER:
Added 'es_user' to the Logstash keystore.
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_PWD
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:31:01,325 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:31:01,326 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:31:01,327 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:31:01,327 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Enter value for ES_PWD:
Added 'es_pwd' to the Logstash keystore.
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:31:51,318 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:31:51,319 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:31:51,320 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:31:51,320 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
es_pwd
es_user
root@logstash:/etc/logstash# ls
'${sys:ls.logs}' certs conf.d jvm.options log4j2.properties logstash.keystore logstash-sample.conf logstash.yml pipelines.yml startup.options
root@logstash:/etc/logstash# chown root:logstash logstash.keystore
root@logstash:/etc/logstash# chmod 770 logstash.keystore
```
and here is the error I got in logstash log journal :
```auto
root@logstash:/etc/logstash/conf.d# systemctl restart logstash
root@logstash:/etc/logstash/conf.d# journalctl -u logstash -f
-- Logs begin at Sun 2024-08-18 20:34:53 UTC. --
Aug 22 01:38:00 logstash logstash[886]: }
Aug 22 01:38:05 logstash systemd[1]: Stopping logstash...
Aug 22 01:38:05 logstash logstash[886]: [2024-08-22T01:38:05,217][WARN][logstash.runner] SIGTERM received. Shutting down.
Aug 22 01:38:06 logstash logstash[886]: [2024-08-22T01:38:06,606][INFO][logstash.javapipeline][main] Pipeline terminated {"pipeline.id"=>"main"}
Aug 22 01:38:07 logstash logstash[886]: [2024-08-22T01:38:07,318][INFO][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
Aug 22 01:38:07 logstash logstash[886]: [2024-08-22T01:38:07,352][INFO][logstash.runner] Logstash shut down.
Aug 22 01:38:07 logstash systemd[1]: logstash.service: Succeeded.
Aug 22 01:38:07 logstash systemd[1]: Stopped logstash.
Aug 22 01:38:07 logstash systemd[1]: Started logstash.
Aug 22 01:38:07 logstash logstash[41720]: Using bundled JDK: /usr/share/logstash/jdk
Aug 22 01:38:25 logstash logstash[41720]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,120][INFO][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,123][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.15.0", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,126][INFO][logstash.runner] JVM bootstrap flags: [-Xms4g, -Xmx4g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,130][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,130][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,040][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::OrgLogstashSecretStore::SecretStoreException::LoadException", :message=>"Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore.", :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.load(JavaKeyStore.java:294)", "org.logstash.secret.store.backend.JavaKeyStore.load(JavaKeyStore.java:77)", "org.logstash.secret.store.SecretStoreFactory.doIt(SecretStoreFactory.java:129)", "org.logstash.secret.store.SecretStoreFactory.load(SecretStoreFactory.java:115)", "org.logstash.secret.store.SecretStoreExt.getIfExists(SecretStoreExt.java:60)", "org.logstash.execution.AbstractPipelineExt.getSecretStore(AbstractPipelineExt.java:797)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:238)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:173)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1379)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:949)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"]}
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,076][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,085][INFO][logstash.runner] Logstash shut down.
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,092][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
Aug 22 01:38:27 logstash logstash[41720]: org.jruby.exceptions.SystemExit: (SystemExit) exit
Aug 22 01:38:27 logstash logstash[41720]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
Aug 22 01:38:27 logstash logstash[41720]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
Aug 22 01:38:27 logstash logstash[41720]: at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
Aug 22 01:38:27 logstash systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Aug 22 01:38:27 logstash systemd[1]: logstash.service: Failed with result 'exit-code'.
Aug 22 01:38:27 logstash systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
Aug 22 01:38:27 logstash systemd[1]: Stopped logstash.
Aug 22 01:38:27 logstash systemd[1]: Started logstash.
```
thank you for stopping by
if u have a suggestion feel free !
---
### Author:  [@Abdarrahmane](https://discuss.elastic.co/u/Abdarrahmane)
#### Post date: [August 22, 2024, 3:58pm UTC](https://discuss.elastic.co/t/logstash-security-best-practices-keystore-monitoring-ssl-verification-mode/365166/3 "2024-08-22T15:58:30Z")
[update]
for the logstash.keystore part i've been able to make it work thanks to @Badger
here is the solution :
> [@Not a valid Logstash keystore java exception problem](https://discuss.elastic.co/t/not-a-valid-logstash-keystore-java-exception-problem/365362/3):
>
> Thank you somuch badger It worked ! here is what i did before creating the keystore and adding the secret username and password i went and creat the directory /etc/sysconfig/ and a logstash file in it with the value of LOGSTASH\_KEYSTORE\_PASS here are the commands : sudo systemctl stop logstash.service sudo mkdir /etc/sysconfig sudo nano /etc/sysconfig/logstash add the following (replace with actual password): LOGSTASH\_KEYSTORE\_PASS=mypassword sudo chmod 600 /etc/sysconfig/logstash Then …