Running an 8.1.3 cluster with all components updated. Filebeat agent is also latest.
This particular host has multiple log files that I was shipping, then creating a different index on each logfile based on the "type" field that I specify in the filebeat.yml. I was under the impression that this option was going away, but I am still unclear. Here is the current setup:
- type: log
enabled: true
paths:
- /u01/wso2is-5.7.0/repository/logs/audit.log*
fields:
type: ssoaudit
fields_under_root: true
- type: log
enabled: true
paths:
- /u01/wso2is-5.7.0/repository/logs/wso2carbon.log*
fields:
type: carbonlog
fields_under_root: true
Here are some of the errors from the logstash nodes, but I am not sure all which are relevant:
[2022-04-27T15:32:07,882][WARN ][logstash.filters.grok ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2022-04-27T15:32:09,112][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: GeoIP Filter in ECS-Compatiblity mode requires a `target` when `source` is not an `ip` sub-field, eg. [client][ip]>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.12-java/lib/logstash/filters/geoip.rb:143:in `auto_target_from_source!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.12-java/lib/logstash/filters/geoip.rb:133:in `setup_target_field'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.12-java/lib/logstash/filters/geoip.rb:108:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:594:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/001-input.conf", "/etc/logstash/conf.d/200-ldap-filter.conf", "/etc/logstash/conf.d/201-haproxy-filter.conf", "/etc/logstash/conf.d/202-windns-filter.conf", "/etc/logstash/conf.d/203-wso2-filter.conf", "/etc/logstash/conf.d/204-ssoaudit-filter.conf", "/etc/logstash/conf.d/205-carbonlog-filter.conf", "/etc/logstash/conf.d/206-windhcp-filter.conf", "/etc/logstash/conf.d/207-asa-filter.conf", "/etc/logstash/conf.d/900-output.conf"], :thread=>"#<Thread:0x592df691 run>"}
[2022-04-27T15:32:09,134][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2022-04-27T15:32:09,288][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
Finally here are my filters if it is even related:
filter {
if [type] == "wso2" {
grok {
match => {"message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
}
filter {
if [type] == "ssoaudit" {
grok {
match => {
"message" => [
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator=%{NOTSPACE:user}\sAction=%{NOTSPACE:action}\sTarget=%{WORD:framework}\sData=%{NOTSPACE:data}\sOutcome=%{WORD:result}",
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{EMAILADDRESS:user} \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:
context},\"AuthenticatedUser\" : %{QUOTEDSTRING:fulluser},\"AuthenticatedUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %
{QUOTEDSTRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdPs\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}",
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{EMAILADDRESS:user} \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:
context},\"LoggedOutUser\" : %{QUOTEDSTRING:loggedoutuser},\"LoggedOutUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %{QU
OTEDSTRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdPs\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}",
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user} \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:context}
,\"AuthenticatedUser\" : %{QUOTEDSTRING:fulluser},\"AuthenticatedUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %{QUOTEDS
TRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdP\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}",
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user} \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:context}
,\"AuthenticatedUser\" : %{QUOTEDSTRING:fulluser},\"AuthenticatedUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %{QUOTEDS
TRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdPs\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}"
]
}
}
grok { match => { "user" => '%{USERNAME:uid}@' } }
mutate {
remove_field => ["framework","tenant","relyparty","authenticatedips" ]
}
}
}
filter {
if [type] == "carbonlog" {
grok {
match => ["message" , "TID: \[%{INT:tid}\] \[\] \[%{TIMESTAMP_ISO8601:date}\]\s+%{WORD:result} \{%{USERNAME:resultsource}\} - %{GREEDYDATA:value}"]
}
}
}