Grok filter not working as expected

Elastic Stack Logstash
1

I am attempting to parse in index log files where the "source" field sometimes contains strings and other times objects. The goal is to identify if the source is JSON and if so rename the source field to something else so that the logs can route to the same index.

Something is wrong with my grok filter as it terminates my LS pipeline. With the error message indicating that I am missing a close parenthesis, but I cannot see where.

Thank you for reading!

Here is my pipeline...

input {
  beats {
    port => 5044
    ssl_certificate => "/etc/pki/tls/certs/elk.cert.pem"
    ssl_key => "/etc/pki/tls/certs/private/elk.key.pem"
    #ssl_certificate_authorities => ["/etc/pki/tls/certs/dps-ca-chain.cert.pem"]
    ssl => true
  }
}

filter {
  grok {
    match => { "message" => "%{SYSLOGBASE2}%{GREEDYDATA}" }
  }
    json {
      source => "source"
      tag_on_failure => ["error_json_parse"]
      skip_on_invalid_json => false
    }
    if [source] !~ "/.*/" {
      mutate {
        rename => { "source" => "json" }
      }
   }
}

output {
  stdout {
    codec => rubydebug
  }

  if ([logsource] != "INFO") and ([logsource] != "WARN") {
    elasticsearch {
      hosts => "elasticsearch1:9200"
      user => "elastic"
      password => "changeme"
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}-new"
      document_type => "%{[@metadata][type]}"
     }
  }
  else {
    file {
      path => "/var/log/noMatch.log"
    }
  }
}

Here is the LS error...

[2018-10-02T17:58:12,472][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x6d7b083d run>"}
[2018-10-02T17:58:15,127][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"SyntaxError", :message=>"(eval):127: (RegexpError) unmatched close pare
nthesis: /))))) # if ([source] !~ \"/\n              if (((((event.get(\"[source]\") !~ //.*//))))) # if ([source] !~ \"/.*/\")\n", :backtrace=>["org/jruby/RubyKernel.java:994:in `eval'", "/usr/share/logstash/logstash-core/lib/log
stash/pipeline.rb:84:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logst
ash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block i
n converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_s
tate_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core
/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initia
lize'"]}
2

It is an error with your if query. Have you tried without the quotes (")

if ([source] !~ /.*/) {
  mutate {
    rename => { "source" => "json" }
  }

}

3

Thank you for the reply bardie, I decided to go with a different approach.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.