Missing event while report to elastic search

Elastic Stack Logstash
1

Hi All,
i could see some events are missing while reporting logs to elastic search. Take an example i am sending 5 logs event only 4 or 3 are reporting.

Basically i am using logstash 7.4 to read my log messages and store the information on elastic search 7.4. below is my logstash configuration

input {
      file {
            type => "web"
            path => ["/Users/znrind-a0053/Downloads/logs/**/*-web.log"]
            start_position => "beginning"
            sincedb_path => "/tmp/sincedb_file"
            codec => multiline {
            pattern => "^(%{MONTHDAY}-%{MONTHNUM}-%{YEAR} %{TIME}) "
            negate => true
            what => previous
            }
       }
 }
 filter {
         if [type] == "web" {
             grok {
               match => [ "message","(?<frontendDateTime>%{MONTHDAY}-%{MONTHNUM}-%{YEAR} %{TIME})%{SPACE}(\[%{DATA:thread}\])?( )?%{LOGLEVEL:level}%{SPACE}%{USERNAME:zhost}%{SPACE}%{JAVAFILE:javaClass} %{USERNAME:orgId} (?<loginId>[\w.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:[.](?:[0-9A-Za-z][0-9A-Za-z‌​-]{0,62}))*) %{GREEDYDATA:jsonstring}"]
             }

             json {
               source => "jsonstring"
               target => "parsedJson"
               remove_field=>["jsonstring"]
               }
           mutate {
               add_field => {
              "actionType" => "%{[parsedJson][actionType]}"
              "errorMessage" => "%{[parsedJson][errorMessage]}"
              "actionName" => "%{[parsedJson][actionName]}"
              "Payload" => "%{[parsedJson][Payload]}"
              "pageInfo" => "%{[parsedJson][pageInfo]}"
              "browserInfo" => "%{[parsedJson][browserInfo]}"
              "dateTime" => "%{[parsedJson][dateTime]}"
              }
            }
         }
   }

   output{
       if "_grokparsefailure" in [tags]
       {
         elasticsearch
         {
           hosts => "localhost:9200"
           index => "grokparsefailure-%{+YYYY.MM.dd}"
         }
       }
       else {
         elasticsearch
         {
           hosts => "localhost:9200"
           index => "zindex"
         }
       }
       stdout{codec => rubydebug}
   }

below is my sample logs

    05-04-2020 13:38:18 [z-weblog-writer] INFO  z-weblog-writer WebLogQueueListner z2store abc.nayak@abc.com {"dateTime":"02-04-2019 20:17:18","actionType":"UI Render","errorMessage":"TypeError: Cannot read property 'name' of undefined","pageInfo":"appId ZOrganization moduleId ZEntityConfig pageId entityConfigGrid","Payload":"payload","browserInfo":"Chrome 8000","actionName":"Unexpected Error"}
    05-04-2020 13:38:18 [z-weblog-writer] INFO  z-weblog-writer WebLogQueueListner z2store abc.nayak@abc.com {"dateTime":"01-04-2019 20:17:18","actionType":"UI Render","errorMessage":"TypeError: Cannot read property 'name' of undefined","pageInfo":"appId : ZOrganization, moduleId: ZEntityConfig, pageId:entityConfigGrid","Payload":"payload","browserInfo":"Chrome 8000","actionName":"Unexpected Error"}
    05-04-2020 13:43:32 [z-weblog-writer] INFO  z-weblog-writer WebLogQueueListner z2store abc.nayak@abc.com {"dateTime":"02-04-2019 20:17:18","actionType":"UI Render","errorMessage":"TypeError: Cannot read property 'name' of undefined","pageInfo":"appId ZOrganization moduleId ZEntityConfig pageId entityConfigGrid","Payload":"payload","browserInfo":"Chrome 8000","actionName":"Unexpected Error"}
    05-04-2020 13:43:32 [z-weblog-writer] INFO  z-weblog-writer WebLogQueueListner z2store abc.nayak@abc.com {"dateTime":"01-04-2019 20:17:18","actionType":"UI Render","errorMessage":"TypeError: Cannot read property 'name' of undefined","pageInfo":"appId : ZOrganization, moduleId: ZEntityConfig, pageId:entityConfigGrid","Payload":"payload","browserInfo":"Chrome 8000","actionName":"Unexpected Error"}

i couldn't figure out why this issue is happening. Any suggestion would be helpful.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.