How to use IF ELSE conduction in grok pattern in logstash

[Vinit_Kumar (Vinit Jordan)]

I have web and API log combined and I want to save it separately in elasticsearch. So I want to write one pattern if the request is for API then if past should execute, the request is web then else part of the log should be executed.

Below are few web and API logs.

    00:06:27,778 INFO  [stdout] (ajp--0.0.0.0-8009-38) 00:06:27.777 [ajp--0.0.0.0-8009-38] INFO  c.r.s.web.rest.WidgetController - Method getWidgetDetails() started to get widget details.
    00:06:27,783 INFO  [stdout] (ajp--0.0.0.0-8009-38) ---> HTTP GET http://api.survey.me/v1/getwidgetdetails?profileName=jeremy-steffens&profileLevel=INDIVIDUAL&companyProfileName=premier-nationwide-lending&hideHistory=true
    00:06:27,817 INFO  [stdout] (ajp--0.0.0.0-8009-38) <--- HTTP 200 http://api.survey.me/v1/getwidgetdetails?profileName=jeremy-steffens&profileLevel=INDIVIDUAL&companyProfileName=premier-nationwide-lending&hideHistory=true (29ms)
    00:06:27,822 INFO  [stdout] (ajp--0.0.0.0-8009-38) 00:06:27.822 [ajp--0.0.0.0-8009-38] INFO  c.r.s.web.rest.WidgetController - Method getWidgetDetails() finished.
    00:06:27,899 INFO  [stdout] (ajp--0.0.0.0-8009-40) 00:06:27.899 [ajp--0.0.0.0-8009-40] INFO  c.r.s.web.controller.LoginController - Inside initLoginPage() of LoginController

I tried to write conduction but it's not working. It's working only up to thread name. After thread, I have multiple type log so not able to write without if conduction.

(?:%{TIME:CREATED_ON})(?:%{SPACE})%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)

Can anybody give me a suggestion?

[Christian_Dahlqvist]

Create two separate grok patterns for the different lines and specify both in the match parameter so the grok filter automatically can pick the first one that matches.

[Vinit_Kumar (Vinit Jordan)]

Hi @Christian_Dahlqvist,

Thank you for your suggestion, I'll try this.

One quick question @Christian_Dahlqvist, Can we use if else condition in one grok pattern?

[Vinit_Kumar (Vinit Jordan)]

I'm not getting any error but still indexes aren't creating in elasticsearch. Below is my configuration files.

input {
  beats {
    port => 5044
  }
}
filter {
    if [log_type] == "apache-apis" {
        grok {
            match => { "message" => "^%{IP:CLIENT_IP} (?:-|%{USER:IDEN}) (?:-|%{USER:AUTH}) \[%{HTTPDATE:CREATED_ON}\] \"(?:%{WORD:REQUEST_METHOD} (?:/|%{NOTSPACE:REQUEST})(?: HTTP/%{NUMBER:HTTP_VERSION})?|-)\" %{NUMBER:RESPONSE_CODE} (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})"}
            add_field => {
                "LOG_TYPES" => "api-log"
            }
        overwrite => [ "message" ]
        }
    }
    if [log_type] == "apache-webs" {
        grok {
            match => { "message" => "%{HTTPDATE:CREATED_ON}%{NOTSPACE}%{SPACE} (?:-|%{IP:CLIENT_IP})%{SPACE} %{NOTSPACE}(?:-|%{WORD:REQUEST_METHOD}%{SPACE}) (?:-|%{NOTSPACE:REQUEST})(?: HTTP/%{NUMBER:HTTP_VERSION})%{NOTSPACE}(?:-|%{GREEDYDATA:OTHER_INFO}) (?:-|%{NUMBER:RESPONSE_CODE}) (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})"}
            add_field => {
                "LOG_TYPES" => "web-log"
            }
        overwrite => [ "message" ]
        }
    }
    if [log_type] == "jboss-apis" {
        grok {
            match => { "message" => "%{TIME:CREATED_ON}%{SPACE}%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE:URI_CLASS}\]%{SPACE}\(%{NOTSPACE:THREAD}\)(?<MESSAGE_LOG>[^\r\n]+)((\r?\n)(?<MORE-INFO>(.|\r?\n)+))?"}
            add_field => {
                "LOG_TYPES" => "jboss-api"
            }
        overwrite => [ "message" ]
        }
    }
    if [log_type] == "jboss-webs" {
        grok {
            match => { "message" => "(?:%{TIME:CREATED_ON})(?:%{SPACE})(?:%{WORD:LEVEL})%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)%{SPACE}(?:%{TIME})(?:%{SPACE})%{SPACE}\[%{NOTSPACE}\]%{SPACE}(?:%{SPACE})%{WORD:LEVEL}%{SPACE}%{JAVACLASS:CLASS} - (?<MESSAGE_LOG>[^\r\n]+)((\r?\n)(?<extra>(.|\r?\n)+))?"}
            add_field => {
                "LOG_TYPES" => "jboss-web"
            }
        }
        grok {
            match => { "message" => "(?:%{TIME:CREATED_ON})(?:%{SPACE})(?:%{WORD:LEVEL})%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:PROTOTYPE}%{SPACE}(?:%{NOTSPACE:STATUS})(?:%{SPACE})(?:%{URI:URI_CLASS})"}
            add_field => {
                "LOG_TYPES" => "jboss-web"
            }
        }
        grok {
            match => { "message" => "%{TIME:CREATED_ON}%{SPACE}%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)\ (?<MESSAGE_LOG>[^\r\n]+)\n%{SPACE}%{NOTSPACE}%{SPACE}%{JAVACLASS:URI-CLASS}"}
            add_field => {
                "LOG_TYPES" => "jboss-web"
            }
        }
    }
}
output {
    if [log_type] == "apache-apis" or [log_type] == "apache-webs" {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "apache-server-logs"
        }
    }
    if [log_type] == "jboss-apis" or [log_type] == "jboss-webs" {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "jboss-server-logs"
        }
    }
  stdout { codec => rubydebug } 
}

filebeat.yml conf file

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/user/path/originallogs/apache/api/a-api.log
  fields:
    log_type: apache-apis
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /home/user/path/originallogs/apache/web/a-web.log
  fields:
    log_type: apache-webs
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /home/user/path/originallogs/jboss/api/jboss-api-log.log
  fields:
    log_type: jboss-apis
  fields_under_root: true


- type: log
  enabled: true
  paths:
    - /home/user/path/originallogs/jboss/web/jboss-web-log.log
  fields:
    log_type: jboss-webs
  fields_under_root: true

  #exclude_lines: ['^DBG']

[Badger]

You never parse a field called message, so setting overwrite has no effect.

grok {
    match => { "message" => [
"^%{IP:CLIENT_IP} (?:-|%{USER:IDEN}) (?:-|%{USER:AUTH}) \[%{HTTPDATE:CREATED_ON}\] \"(?:%{WORD:REQUEST_METHOD} (?:/|%{NOTSPACE:REQUEST})(?:HTTP/%{NUMBER:HTTP_VERSION})?|-)\" %{NUMBER:RESPONSE_CODE} (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})",
"%{HTTPDATE:CREATED_ON}%{NOTSPACE}%{SPACE} (?:-|%{IP:CLIENT_IP})%{SPACE} %{NOTSPACE}(?:-|%{WORD:REQUEST_METHOD}%{SPACE})(?:-|%{NOTSPACE:REQUEST})(?:HTTP/%{NUMBER:HTTP_VERSION})%{NOTSPACE}(?:-|%{GREEDYDATA:OTHER_INFO}) (?:-|%{NUMBER:RESPONSE_CODE}) (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})",
"%{TIME:CREATED_ON}%{SPACE}%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE:URI_CLASS}\]%{SPACE}\(%{NOTSPACE:THREAD}\)(?<MESSAGE_LOG>[^\r\n]+)((\r?\n)(?<MORE-INFO>(.|\r?\n)+))?",
"(?:%{TIME:CREATED_ON})(?:%{SPACE})(?:%{WORD:LEVEL})%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)%{SPACE}(?:%{TIME})(?:%{SPACE})%{SPACE}\[%{NOTSPACE}\]%{SPACE}(?:%{SPACE})%{WORD:LEVEL}%{SPACE}%{JAVACLASS:CLASS} - (?<MESSAGE_LOG>[^\r\n]+)((\r?\n)(?<extra>(.|\r?\n)+))?",
"(?:%{TIME:CREATED_ON})(?:%{SPACE})(?:%{WORD:LEVEL})%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:PROTOTYPE}%{SPACE}(?:%{NOTSPACE:STATUS})(?:%{SPACE})(?:%{URI:URI_CLASS})",
"%{TIME:CREATED_ON}%{SPACE}%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)\ (?<MESSAGE_LOG>[^\r\n]+)\n%{SPACE}%{NOTSPACE}%{SPACE}%{JAVACLASS:URI-CLASS}"
        ]
    }
}

It will stop when it matches one.