Hi everyone
I'm trying to set up logstash in a kubernetes cluster, but whenever the container starts the logs say it terminates the pipeline I have: [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
My configuration files are as follows:
pipeline config
input {
syslog {
port => 5144
host => "0.0.0.0"
}
}
filter {
grok {
match => {
"message" => [
",attack_type=\"%{DATA:attack_type}\"",
",blocking_exception_reason=\"%{DATA:blocking_exception_reason}\"",
",bot_signature_name=\"%{DATA:bot_signature_name}\"",
",bot_category=\"%{DATA:bot_category}\"",
",bot_anomalies=\"%{DATA:bot_anomalies}\"",
",client_class=\"%{DATA:client_class}\"",
",date_time=\"%{DATA:date_time}\"",
",dest_port=\"%{DATA:dest_port}\"",
",ip_client=\"%{DATA:ip_client}\"",
",is_truncated=\"%{DATA:is_truncated}\"",
",method=\"%{DATA:method}\"",
",policy_name=\"%{DATA:policy_name}\"",
",protocol=\"%{DATA:protocol}\"",
",request_status=\"%{DATA:request_status}\"",
",response_code=\"%{DATA:response_code}\"",
",severity=\"%{DATA:severity}\"",
",sig_cves=\"%{DATA:sig_cves}\"",
",sig_ids=\"%{DATA:sig_ids}\"",
",sig_names=\"%{DATA:sig_names}\"",
",sig_set_names=\"%{DATA:sig_set_names}\"",
",src_port=\"%{DATA:src_port}\"",
",sub_violations=\"%{DATA:sub_violations}\"",
",support_id=\"%{DATA:support_id}\"",
",unit_hostname=\"%{DATA:unit_hostname}\"",
",uri=\"%{DATA:uri}\"",
",violation_rating=\"%{DATA:violation_rating}\"",
",vs_name=\"%{DATA:vs_name}\"",
",x_forwarded_for_header_value=\"%{DATA:x_forwarded_for_header_value}\"",
",outcome=\"%{DATA:outcome}\"",
",outcome_reason=\"%{DATA:outcome_reason}\"",
",violations=\"%{DATA:violations}\"",
",violation_details=\"%{DATA:violation_details}\"",
",request=\"%{DATA:request}\""
]
}
break_on_match => false
}
mutate {
split => { "attack_type" => "," }
split => { "sig_ids" => "," }
split => { "sig_names" => "," }
split => { "sig_cves" => "," }
split => { "staged_sig_ids" => "," }
split => { "staged_sig_names" => "," }
split => { "staged_sig_cves" => "," }
split => { "sig_set_names" => "," }
split => { "threat_campaign_names" => "," }
split => { "staged_threat_campaign_names" => "," }
split => { "violations" => "," }
split => { "sub_violations" => "," }
}
if [x_forwarded_for_header_value] != "N/A" {
mutate { add_field => { "source_host" => "%{x_forwarded_for_header_value}"}}
} else {
mutate { add_field => { "source_host" => "%{ip_client}"}}
}
geoip {
source => "source_host"
}
}
output {
elasticsearch {
hosts => "quickstart-es-http:9200"
index => "waf-logs-%{+YYY.MM.dd}"
user => "waf-logstash"
password => "blahblah"
}
}
config file for logstash
http.host: "0.0.0.0"
http.port: "5144"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: "quickstart-es-http:9200"
xpack.monitoring.elasticsearch.username: "waf-logstash"
xpack.monitoring.elasticsearch.password: "blahblah"
I pass the following arguments to the logstash command on start up
-f /usr/share/logstash/pipeline
I'm using 7.9.3 tagged image of logstash
I would greatly appreciate any help with this as I am a complete noob on ELK!
The full log output is as follows:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby15032451142012517880jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2020-11-12 11:35:50.403 [main] runner - Starting Logstash {"logstash.version"=>"7.9.3", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-LTS on 11.0.8+10-LTS +jit [linux-x86_64]"}
[INFO ] 2020-11-12 11:35:50.416 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2020-11-12 11:35:50.418 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2020-11-12 11:35:50.608 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-11-12 11:35:50.617 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"e2bc3af3-102b-4d42-9e1f-b8ecf260fe59", :path=>"/usr/share/logstash/data/uuid"}
[WARN ] 2020-11-12 11:35:50.691 [LogStash::Runner] pipelineregisterhook - Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
Please configure Metricbeat to monitor Logstash. Documentation can be found at:
https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
[INFO ] 2020-11-12 11:35:51.072 [LogStash::Runner] licensereader - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://waf-logstash:xxxxxx@quickstart-es-http:9200/]}}
[WARN ] 2020-11-12 11:35:51.204 [LogStash::Runner] licensereader - Restored connection to ES instance {:url=>"http://waf-logstash:xxxxxx@quickstart-es-http:9200/"}
[INFO ] 2020-11-12 11:35:51.364 [LogStash::Runner] licensereader - ES Output version determined {:es_version=>7}
[WARN ] 2020-11-12 11:35:51.364 [LogStash::Runner] licensereader - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2020-11-12 11:35:51.429 [LogStash::Runner] internalpipelinesource - Monitoring License OK
[INFO ] 2020-11-12 11:35:51.429 [LogStash::Runner] internalpipelinesource - Validated license for monitoring. Enabling monitoring pipeline.
[INFO ] 2020-11-12 11:35:51.442 [Agent thread] configpathloader - No config files found in path {:path=>"/usr/share/logstash/ /usr/share/logstash/pipeline"}
[INFO ] 2020-11-12 11:35:52.070 [Converge PipelineAction::Create<.monitoring-logstash>] Reflections - Reflections took 37 ms to scan 1 urls, producing 22 keys and 45 values
[INFO ] 2020-11-12 11:35:52.141 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://waf-logstash:xxxxxx@quickstart-es-http:9200/]}}
[WARN ] 2020-11-12 11:35:52.151 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Restored connection to ES instance {:url=>"http://waf-logstash:xxxxxx@quickstart-es-http:9200/"}
[INFO ] 2020-11-12 11:35:52.158 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - ES Output version determined {:es_version=>7}
[WARN ] 2020-11-12 11:35:52.158 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2020-11-12 11:35:52.186 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["//quickstart-es-http:9200"]}
[WARN ] 2020-11-12 11:35:52.187 [[.monitoring-logstash]-pipeline-manager] javapipeline - 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[INFO ] 2020-11-12 11:35:52.210 [[.monitoring-logstash]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x393ee466@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[INFO ] 2020-11-12 11:35:52.961 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.75}
[INFO ] 2020-11-12 11:35:52.969 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[INFO ] 2020-11-12 11:35:52.991 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
[INFO ] 2020-11-12 11:35:53.040 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>5144}
[INFO ] 2020-11-12 11:35:54.188 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[INFO ] 2020-11-12 11:35:55.028 [LogStash::Runner] runner - Logstash shut down.