After update to 8.1.3 Logstash in cluster no longer functioning

Running an 8.1.3 cluster with all components updated. Filebeat agent is also latest.

This particular host has multiple log files that I was shipping, then creating a different index on each logfile based on the "type" field that I specify in the filebeat.yml. I was under the impression that this option was going away, but I am still unclear. Here is the current setup:

- type: log
  enabled: true
  paths:
    - /u01/wso2is-5.7.0/repository/logs/audit.log*
  fields:
    type: ssoaudit
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /u01/wso2is-5.7.0/repository/logs/wso2carbon.log*
  fields:
    type: carbonlog
  fields_under_root: true

Here are some of the errors from the logstash nodes, but I am not sure all which are relevant:

[2022-04-27T15:32:07,882][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated

[2022-04-27T15:32:09,112][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: GeoIP Filter in ECS-Compatiblity mode requires a `target` when `source` is not an `ip` sub-field, eg. [client][ip]>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.12-java/lib/logstash/filters/geoip.rb:143:in `auto_target_from_source!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.12-java/lib/logstash/filters/geoip.rb:133:in `setup_target_field'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.12-java/lib/logstash/filters/geoip.rb:108:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:594:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/001-input.conf", "/etc/logstash/conf.d/200-ldap-filter.conf", "/etc/logstash/conf.d/201-haproxy-filter.conf", "/etc/logstash/conf.d/202-windns-filter.conf", "/etc/logstash/conf.d/203-wso2-filter.conf", "/etc/logstash/conf.d/204-ssoaudit-filter.conf", "/etc/logstash/conf.d/205-carbonlog-filter.conf", "/etc/logstash/conf.d/206-windhcp-filter.conf", "/etc/logstash/conf.d/207-asa-filter.conf", "/etc/logstash/conf.d/900-output.conf"], :thread=>"#<Thread:0x592df691 run>"}

[2022-04-27T15:32:09,134][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

[2022-04-27T15:32:09,288][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

Finally here are my filters if it is even related:

filter {
 if [type] == "wso2" {
    grok {
      match => {"message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
      source => "clientip"
    }
  }
}

filter {
  if [type] == "ssoaudit" {
    grok {
        match => {
         "message" => [
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator=%{NOTSPACE:user}\sAction=%{NOTSPACE:action}\sTarget=%{WORD:framework}\sData=%{NOTSPACE:data}\sOutcome=%{WORD:result}",
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{EMAILADDRESS:user} \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:
context},\"AuthenticatedUser\" : %{QUOTEDSTRING:fulluser},\"AuthenticatedUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %
{QUOTEDSTRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdPs\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}",
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{EMAILADDRESS:user} \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:
context},\"LoggedOutUser\" : %{QUOTEDSTRING:loggedoutuser},\"LoggedOutUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %{QU
OTEDSTRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdPs\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}",
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user} \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:context}
,\"AuthenticatedUser\" : %{QUOTEDSTRING:fulluser},\"AuthenticatedUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %{QUOTEDS
TRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdP\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}",
"%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user} \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:context}
,\"AuthenticatedUser\" : %{QUOTEDSTRING:fulluser},\"AuthenticatedUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %{QUOTEDS
TRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdPs\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}"
                       ]
    }
    }
    grok { match => { "user" => '%{USERNAME:uid}@' } }

    mutate {
         remove_field => ["framework","tenant","relyparty","authenticatedips" ]
       }
  }
}

filter {
  if [type] == "carbonlog" {
    grok {
        match => ["message" , "TID: \[%{INT:tid}\] \[\] \[%{TIMESTAMP_ISO8601:date}\]\s+%{WORD:result} \{%{USERNAME:resultsource}\} - %{GREEDYDATA:value}"]
    }
  }
}

In 8.x ecs_compatability is enabled by default, and when it is enabled the geoip filter no longer has geoip as the default target. You can disable ecs_compatability at the filter, pipeline, or process level to get the old behaviour.

Thanks Badge, I didn't think this would prevent logstash from running, but apparently I was wrong.

I disabled the ecs_compatability and switched from using type to [tags] when differentiating the different log files. Everything is working now.

Thank you again for assistance.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.