Logstash unable to identify tags from filebeat ( grokparsefailure)


(Nikhil Pawar) #1

Hi, i am sending two different logs files with filebeat agent with different tags on filebeat side to logstash but logstash only able to parse one file and failing for other . Below is my config.

Can someone help .

input {
            beats {
            client_inactivity_timeout => 86400
            port => 5044
            type => "log"
                  }
              }
        filter {
            if "vus" in [tags] {
            mutate {
            gsub => [
              "message", "\t", " ",
              "message", "\n", " "
                    ]
                }
            grok {
        match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp_match}\]%{SPACE}%{WORD:level}%{SPACE}%{JAVACLASS:coidkey}%{SPACE}%
        {USER:ident}%{SPACE}%{GREEDYDATA:Url}"}
          }
        }
         else if "download" in [tags] {
            mutate {
            gsub => [
              "message", "\t", " ",
              "message", "\n", " "
                    ]
                }
            grok {
            match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp_match}\]%{SPACE}\:\|\:%{SPACE}%{WORD:level}%{SPACE}\:\|\:%{SPA
        CE}%{USERNAME:host_name}%{SPACE}\:\|\:%{SPACE}%{DATA:coidkey}%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:clientinfo}%{SPACE}\:
        \|\:%{SPACE}(%{IP:clientip})?%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:Url}%{SPACE}\:\|\:%{SPACE}%{JAVACLASS:class}%{SPACE}\:\|\:%
        {SPACE}%{USER:ident}%{SPACE}%{GREEDYDATA:msg}"}
           remove_field => [ "ident","offset","name","version","host" ]
                 }
                         }
          }
         output {
            stdout { codec => rubydebug }

          if "_grokparsefailure" in [tags] {
            # write events that didn't match to a file
            file { "path" => "/tmp/grok_failures.txt" }
          }
        # if "vus" in [tags] {
        #   elasticsearch {
        #       hosts => "dfdevelasticp1.df.jabodo.com:9200"
        #       user => "elastic"
        #       password => "fd5dd89c"
        #       index => "download-%{+YYYY.MM.dd}"
        #       document_type => "log"
        #     }
        #   }
        #if [type] == "download"{
        else{
        elasticsearch {
               hosts => "dfsyselastic.df.jabodo.com:9200"
               user => "UN"
               password => "PW"
               index => "vicinio-%{+YYYY.MM.dd}"
               document_type => "log"
             }
           }
        }

I have tested both my grok they work fine alone.


(Magnus Bäck) #2

Show what an incorrectly filtered event looks like. Use a stdout { codec => rubydebug } output or copy/paste from Kibana's JSON tab.


(Nikhil Pawar) #3

Hi mangus,

Below is the output from stdout { codec => rubydebug }

{
    "@timestamp" => 2017-08-08T14:07:38.276Z,
        "offset" => 138524460,
      "@version" => "1",
    "input_type" => "log",
          "beat" => {
        "hostname" => "dfprdsndlbfe2",
            "name" => "dfprdsndlbfe2",
         "version" => "5.5.1"
    },
          "host" => "dfprdsndlbfe2",
        "source" => "/archives/logs/tomcat7-8080/download.log",
       "message" => "[2017-08-08 10:07:38,116]  :|:  INFO   :|:  dfprdsndlbfe2.df.jabodo.com  :|:  a5fa8005dbe049a7b8d9bafa95400b18  :|:  [BT:CHROME, BV:59, BL:en, CC:PH]  :|:\n\t\t\t\t130.105.229.204  :|:  http://www.radiorage.com/index.jhtml?partner=ZXxpt535&s2=-5503188656016333761&s1=604364  :|:  c.m.w.d.m.UnifiedLoggerWrapper                                :|:   - [ET: DLPInfo, IP: 130.105.229.204]",
          "type" => "log",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ]
}
{
 

"@timestamp" => 2017-08-08T14:12:38.379Z,
        "offset" => 139773176,
      "@version" => "1",
          "beat" => {
        "hostname" => "dfprdsndlbfe2",
            "name" => "dfprdsndlbfe2",
         "version" => "5.5.1"
    },
    "input_type" => "log",
          "host" => "dfprdsndlbfe2",
        "source" => "/archives/logs/tomcat7-8080/download.log",
       "message" => "[2017-08-08 10:12:38,321]  :|:  INFO   :|:  dfprdsndlbfe2.df.jabodo.com  :|:    :|:  [BT:FF, BV:54, BL:en, CC:US]  :|:\n\t\t\t\t65.35.140.148  :|:  http://internetspeedtracker.dl.myway.com/blank.jhtml  :|:  c.m.w.d.m.UnifiedLoggerWrapper                                :|:   - [ET: BlankPageServed, IP: 65.35.140.148]",
          "type" => "log",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ]
}
{

And on kibana (json)

{
  "_index": "vicinio-2017.08.08",
  "_type": "log",
  "_id": "AV3CNGLArR1KM0reUPvn",
  "_score": null,
  "_source": {
    "@timestamp": "2017-08-08T14:16:38.465Z",
    "offset": 140810549,
    "@version": "1",
    "input_type": "log",
    "beat": {
      "hostname": "dfprdsndlbfe2",
      "name": "dfprdsndlbfe2",
      "version": "5.5.1"
    },
    "host": "dfprdsndlbfe2",
    "source": "/archives/logs/tomcat7-8080/download.log",
    "message": "[2017-08-08 10:16:38,403]  :|:  INFO   :|:  dfprdsndlbfe2.df.jabodo.com  :|:    :|:  [BT:CHROME, BV:51, BL:en, CC:US]  :|:\n\t\t\t\t66.249.91.151  :|:    :|:  c.m.w.d.m.UnifiedLoggerWrapper                                :|:   - [ET: PageView, IP: 66.249.91.151]",
    "type": "log",
    "tags": [
      "beats_input_codec_plain_applied"
    ]
  },
  "fields": {
    "@timestamp": [
      1502201798465
    ]
  },
  "highlight": {
    "beat.name": [
      "@kibana-highlighted-field@dfprdsndlbfe2@/kibana-highlighted-field@"
    ],
    "beat.hostname": [
      "@kibana-highlighted-field@dfprdsndlbfe2@/kibana-highlighted-field@"
    ],
    "host": [
      "@kibana-highlighted-field@dfprdsndlbfe2@/kibana-highlighted-field@"
    ],
    "message": [
      "[2017-08-08 10:16:38,403]  :|:  INFO   :|:  @kibana-highlighted-field@dfprdsndlbfe2@/kibana-highlighted-field@.df.jabodo.com  :|:    :|:  [BT:CHROME, BV:51, BL:en, CC:US]  :|:\n\t\t\t\t66.249.91.151  :|:    :|:  c.m.w.d.m.UnifiedLoggerWrapper                                :|:   - [ET: PageView, IP: 66.249.91.151]"
    ]
  },
  "sort": [
    1502201798465
  ]
}

(Magnus Bäck) #4

You only apply filters to events tagged "download" or "vus" and these events have no such tags.


(Nikhil Pawar) #5

Hi mangus ,

I checked the filebeat connfig - download tag was missing on one server but the vus tag was present and i see those getting parsed correctly .

I corrected config and restarted filebeat now seeing following

{
    "@timestamp" => 2017-08-08T14:31:11.724Z,
        "offset" => 144374649,
      "@version" => "1",
    "input_type" => "log",
          "beat" => {
        "hostname" => "dfprdsndlbfe2",
            "name" => "dfprdsndlbfe2",
         "version" => "5.5.1"
    },
          "host" => "dfprdsndlbfe2",
        "source" => "/archives/logs/tomcat7-8080/download.log",
       "message" => "[2017-08-08 10:31:11,372]  :|:  INFO   :|:  dfprdsndlbfe2.df.jabodo.com  :|:    :|:  [BT:CHROME, BV:59, BL:en, CC:IN]  :|:     47.247.11.150  :|:    :|:  c.m.w.d.m.UnifiedLoggerWrapper                                :|:   - [ET: PageView, IP: 47.247.11.150]",
          "type" => "log",
          "tags" => [
        [0] "download",
        [1] "beats_input_codec_plain_applied",
        [2] "_grokparsefailure"
    ]
}
{
    "@timestamp" => 2017-08-08T14:31:11.724Z,
        "offset" => 144374907,
      "@version" => "1",
          "beat" => {
        "hostname" => "dfprdsndlbfe2",
            "name" => "dfprdsndlbfe2",
         "version" => "5.5.1"
    },
    "input_type" => "log",
          "host" => "dfprdsndlbfe2",
        "source" => "/archives/logs/tomcat7-8080/download.log",
       "message" => "[2017-08-08 10:31:11,389]  :|:  INFO   :|:  dfprdsndlbfe2.df.jabodo.com  :|:    :|:  [BT:CHROME, BV:59, BL:en, CC:IN]  :|:     171.79.71.196  :|:    :|:  c.m.w.d.m.UnifiedLoggerWrapper                                :|:   - [ET: PageView, IP: 171.79.71.196]",
          "type" => "log",
          "tags" => [
        [0] "download",
        [1] "beats_input_codec_plain_applied",
        [2] "_grokparsefailure"
    ]
}

Also i am able to see that tag is coming with input line

root@dfsyslogstash2:~# more /tmp/grok_failures.txt
{"@timestamp":"2017-08-08T14:30:42.710Z","offset":144172281,"@version":"1","input_type":"log","beat":{"hostname":"dfprdsndl
bfe2","name":"dfprdsndlbfe2","version":"5.5.1"},"host":"dfprdsndlbfe2","source":"/archives/logs/tomcat7-8080/download.log",
"message":"[2017-08-08 10:30:23,758]  :|:  INFO   :|:  dfprdsndlbfe2.df.jabodo.com  :|:  c30f631536ef4862933590aa4fc58678
:|:  [BT:IE, BV:11, BL:en, CC:PT]  :|:     195.23.12.27  :|:  http://free.fromdoctopdf.com/index.jhtml?partner=Y6xpu065&s1=
1579971&s2=15022026003273067547260080817699725  :|:  c.m.w.d.m.UnifiedLoggerWrapper                                :|:   -
[ET: DLPInfo, IP: 195.23.12.27]","type":"log","tags":["download","beats_input_codec_plain_applied","_grokparsefailure"]}
{"@timestamp":"2017-08-08T14:30:42.710Z","offset":144172539,"@version":"1","input_type":"log","beat":{"hostname":"dfprdsndl
bfe2","name":"dfprdsndlbfe2","version":"5.5.1"},"host":"dfprdsndlbfe2","source":"/archives/logs/tomcat7-8080/download.log",
"message":"[2017-08-08 10:30:23,881]  :|:  INFO   :|:  dfprdsndlbfe2.df.jabodo.com  :|:    :|:  [BT:FF, BV:49, BL:en, CC:ID
]  :|:     180.241.166.108  :|:    :|:  c.m.w.d.m.UnifiedLoggerWrapper                                :|:   - [ET: backFill
, IP: 180.241.166.108]","type":"log","tags":["download","beats_input_codec_plain_applied","_grokparsefailure"]}

(Nikhil Pawar) #6

Found it. I guess it was a space issue. Not seeing any grokfailures


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.