Hi all,
playing further with elasticsearch and logstash security (view this post), I received other errors regarding cluster permissions.
The configuration is the same as per the previous post.
I configured the logstash user's cluster permissions as per security configuration guide:
GET _security/role/logstash_writer
{
"logstash_writer" : {
"cluster" : [
"manage_index_templates",
"monitor",
"manage_ilm"
],
"indices" : [
{
"names" : [
"logstash-*"
],
"privileges" : [
"write",
"create",
"delete",
"create_index",
"manage",
"manage_ilm"
],
"allow_restricted_indices" : false
},
{
"names" : [
"logstash"
],
"privileges" : [
"manage",
"write"
],
"field_security" : {
"grant" : [
"*"
]
},
"allow_restricted_indices" : false
}
],
"applications" : [ ],
"run_as" : [ ],
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}
I started the logstash_simple.conf:
sudo /usr/share/logstash/bin/logstash -f /home/rciaravolo/elk-test/logstash/conf.d.test/logstash_simple.conf --path.settings /home/rciaravolo/elk-test/logstash/
And received the following output with errors:
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to //home/user_account/elk-test/logstash_log/log which is now configured via log4j2.properties
[2022-07-22T12:53:44,100][INFO ][logstash.runner ] Log4j configuration path used is: /home/user_account/elk-test/logstash/log4j2.properties
[2022-07-22T12:53:44,117][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.17.5", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-07-22T12:53:44,122][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:MaxGCPauseMillis=300, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2022-07-22T12:53:44,652][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-07-22T12:53:48,472][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2022-07-22T12:53:48,474][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2022-07-22T12:53:48,700][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-07-22T12:53:50,137][INFO ][org.reflections.Reflections] Reflections took 145 ms to scan 1 urls, producing 119 keys and 419 values
[2022-07-22T12:53:51,060][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://localhost:9200"]}
[2022-07-22T12:53:51,100][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://service_account:xxxxxx@localhost:9200/]}}
[2022-07-22T12:53:51,163][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://service_account:xxxxxx@localhost:9200/"}
[2022-07-22T12:53:51,174][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (7.17.5) {:es_version=>7}
[2022-07-22T12:53:51,174][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-07-22T12:53:51,246][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2022-07-22T12:53:51,291][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-07-22T12:53:51,304][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2022-07-22T12:53:51,310][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://service_account:xxxxxx@localhost:9200/]}}
[2022-07-22T12:53:51,336][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://service_account:xxxxxx@localhost:9200/"}
[2022-07-22T12:53:51,347][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.17.5) {:es_version=>7}
[2022-07-22T12:53:51,347][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-07-22T12:53:51,385][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x592b0457 run>"}
[2022-07-22T12:53:51,433][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-07-22T12:53:51,435][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-07-22T12:53:51,456][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-07-22T12:53:51,482][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/home/user_account/elk-test/logstash/conf.d.test/logstash_simple.conf"], :thread=>"#<Thread:0x125d5f04 run>"}
[2022-07-22T12:53:52,634][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>1.24}
[2022-07-22T12:53:52,742][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.26}
[2022-07-22T12:53:52,751][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in
WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2022-07-22T12:53:52,843][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2022-07-22T12:53:52,904][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
[2022-07-22T12:53:53,191][ERROR][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash][86a8d8b43cce4867d8c6cf2303852f52195f108f267a29e871601fadb703e1b9] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"http://localhost:9200/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", :content_length=>1894}
[2022-07-22T12:53:55,220][ERROR][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash][86a8d8b43cce4867d8c6cf2303852f52195f108f267a29e871601fadb703e1b9] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"http://localhost:9200/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", :content_length=>1894}
[2022-07-22T12:53:59,234][ERROR][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash][86a8d8b43cce4867d8c6cf2303852f52195f108f267a29e871601fadb703e1b9] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"http://localhost:9200/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", :content_length=>1894}
^C[2022-07-22T12:54:01,515][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2022-07-22T12:54:02,455][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-07-22T12:54:02,624][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2022-07-22T12:54:06,521][WARN ][logstash.runner ] Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss.
[2022-07-22T12:54:06,644][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>48, "name"=>"[.monitoring-logstash]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}]}}
[2022-07-22T12:54:06,647][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[2022-07-22T12:54:07,249][ERROR][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash][86a8d8b43cce4867d8c6cf2303852f52195f108f267a29e871601fadb703e1b9] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"http://localhost:9200/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", :content_length=>1894}
[2022-07-22T12:54:11,675][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>48, "name"=>"[.monitoring-logstash]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2022-07-22T12:54:16,703][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>48, "name"=>"[.monitoring-logstash]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
^C[2022-07-22T12:54:17,905][FATAL][logstash.runner ] SIGINT received. Terminating immediately..
[2022-07-22T12:54:17,982][FATAL][org.logstash.Logstash ]
org.jruby.exceptions.ThreadKill: null
Please note that when I send SIGTERM (with CTRL+C) the thread is hang and I must forcefully kill the process.
If I add superuser role all runs fine.
I tried adding all permissions to the logstash_writer role and removing them one at a time.
The result is that I need "manage" permissions which I think are exceeding the requirements (a security hole).
GET _security/role/logstash_writer
{
"logstash_writer" : {
"cluster" : [
"manage"
],
"indices" : [
{
"names" : [
"logstash-*"
],
"privileges" : [
"write",
"create",
"delete",
"create_index",
"manage",
"manage_ilm"
],
"allow_restricted_indices" : false
},
{
"names" : [
"logstash"
],
"privileges" : [
"manage",
"write"
],
"field_security" : {
"grant" : [
"*"
]
},
"allow_restricted_indices" : false
}
],
"applications" : [ ],
"run_as" : [ ],
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to //home/user_account/elk-test/logstash_log/log which is now configured via log4j2.properties
[2022-07-22T13:13:13,903][INFO ][logstash.runner ] Log4j configuration path used is: /home/user_account/elk-test/logstash/log4j2.properties
[2022-07-22T13:13:13,916][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.17.5", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-07-22T13:13:13,920][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:MaxGCPauseMillis=300, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2022-07-22T13:13:14,349][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-07-22T13:13:17,475][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2022-07-22T13:13:17,476][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2022-07-22T13:13:17,721][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-07-22T13:13:19,037][INFO ][org.reflections.Reflections] Reflections took 117 ms to scan 1 urls, producing 119 keys and 419 values
[2022-07-22T13:13:19,787][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://localhost:9200"]}
[2022-07-22T13:13:19,832][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://service_account:xxxxxx@localhost:9200/]}}
[2022-07-22T13:13:19,910][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://service_account:xxxxxx@localhost:9200/"}
[2022-07-22T13:13:19,934][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (7.17.5) {:es_version=>7}
[2022-07-22T13:13:19,935][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-07-22T13:13:20,019][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2022-07-22T13:13:20,068][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-07-22T13:13:20,067][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://service_account:xxxxxx@localhost:9200/]}}
[2022-07-22T13:13:20,084][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2022-07-22T13:13:20,098][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://service_account:xxxxxx@localhost:9200/"}
[2022-07-22T13:13:20,107][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.17.5) {:es_version=>7}
[2022-07-22T13:13:20,107][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-07-22T13:13:20,183][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-07-22T13:13:20,183][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-07-22T13:13:20,196][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-07-22T13:13:20,257][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x59201b6 run>"}
[2022-07-22T13:13:20,257][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/home/user_account/elk-test/logstash/conf.d.test/logstash_simple.conf"], :thread=>"#<Thread:0x3c6d40 run>"}
[2022-07-22T13:13:21,541][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>1.28}
[2022-07-22T13:13:21,653][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.39}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in
WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2022-07-22T13:13:21,682][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-07-22T13:13:21,766][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2022-07-22T13:13:21,824][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
good test
{
"@timestamp" => 2022-07-22T11:13:35.680Z,
"message" => "good test",
"@version" => "1",
"host" => "hostname.domain"
}
^C[2022-07-22T13:13:37,716][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2022-07-22T13:13:38,216][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-07-22T13:13:38,842][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2022-07-22T13:13:39,046][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2022-07-22T13:13:39,934][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:".monitoring-logstash"}
[2022-07-22T13:13:40,003][INFO ][logstash.runner ] Logstash shut down.
My goal is to carve out the thinnest possible authorization profile for the logstash user.
Thanks in advance, I hope someone can help me.
Rocco