Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:test, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #,

[Mehak_Bhargava]

I am not sure why is the if[fields][log_type] throwing error here

Here is the error my pipeline.conf file is giving below

[2020-01-09T11:51:31,810][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:test, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 12, column 10 (byte 257) after filter {\n if[fields][log_type] ==\"access\"{\n    grok {\n\tmatch => {\"message\" => \"%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA} %{NONNEGINT:threadId}\"}\n  } else ", :backtrace=>["/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}

Pipeline.conf-

input {
  
  beats {
    port => 5044
  }
}

filter {
 if[fields][log_type] =="access"{
    grok {
	match => {"message" => "%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA} %{NONNEGINT:threadId}"}
  } else if [fields][log_type] == "errors" {
        grok {
            match => { "message" => "%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA:message}" }
        }
  }else [fields][log_type] == "dispatch" {
        grok {
            match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:threadId}]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{JAVACLASS:javaClass}%{SPACE}-%{SPACE}?(\[%{NONNEGINT:incidentId}])%{GREEDYDATA:message}" }
        }
    }
}
 
output {
    elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    ilm_enabled => false
    index    => "%{log_type}-%{+YYYY.MM.dd}"  
  }
  stdout {
    codec => rubydebug
  }
}

[Badger]

@Mehak_Bhargava

else [fields][log_type] == "dispatch"

That should be "else if", not just "else"

[Mehak_Bhargava]

@Badger,

I made the above change and still have this error

[2020-01-09T12:34:36,429][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:test, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 12, column 10 (byte 257) after filter {\n if[fields][log_type] ==\"access\"{\n    grok {\n\tmatch => {\"message\" => \"%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA} %{NONNEGINT:threadId}\"}\n  } else ", :backtrace=>["/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/home/mehak/Documents/logstash-7.4.0/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}

Here it states that line 12, which is the else if line has the error thrown at.

filter {
 if[fields][log_type] =="access"{
    grok {
	match => {"message" => "%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA} %{NONNEGINT:threadId}"}
  } else if [fields][log_type] == "errors" {
        grok {
            match => { "message" => "%{DATESTAMP:timestamp} %{NONNEGINT:code} %{GREEDYDATA} %{LOGLEVEL} %{NONNEGINT:anum} %{GREEDYDATA:message}" }
        }
  }else if [fields][log_type] == "dispatch" {
        grok {
            match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:threadId}]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{JAVACLASS:javaClass}%{SPACE}-%{SPACE}?(\[%{NONNEGINT:incidentId}])%{GREEDYDATA:message}" }
        }
    }
}

Above is the updated block

[Badger]

You are missing a } to close your grok filter.

[Mehak_Bhargava]

@Badger,

there is no error thrown in filter block but still in kibana the message as it should be outputted after extracting timestamp, and all isn't shown. So why is the grok pattern not working?

Below is the message I am getting instead of

08/10/2019 12:32:18 608   (null)                  INFO   60   Leftside Filter Expression : SubCategory="ATA VTA Reported" AND SourceProblemName="Touch Screen" for User ZK0DUBO Item Count : 7

Instead of this for instance

{
  "code": "774",
  "anum": "61",
  "message": "Leftside Filter Expression : IncidentType=1 AND StatusCode=\"01\" for User ZKH481F",
  "timestamp": "08/10/2019 12:38:09"
}