Logstash working on Linux but not on Windows

[Mehak_Bhargava]

I have Logstash config file with grok filters that work when I am passing logs from linux. But when logs are passed from windows, the filters arent being applied to it.

below is my linux filebeat

filebeat.inputs:
- 
  paths:
     - /home/mehak/Documents/filebeat-7.5.2-linux-x86_64/logs/xxxx.log
  enabled: true
  input_type: log
  fields:  
    tags: ngta-common-test
output.logstash:
  hosts: ["localhost:5044"]

Below is my logstash config

#listening on this port
input {
  beats {
    port => 5044
  }
}

filter {
   if [fields][tags] == "ngta-common-test" {
    #mutate {
      #add_field => { "host" => "%{[event_data][IpAddress]}" }
    #}
      grok{
        match => { 
	"message" => [
	   "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{GREEDYDATA}\]%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{JAVACLASS:javaClass}%{GREEDYDATA:logmessage} " 
 	]
       }
    }
  }
  
  
}
output {
    elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    ilm_enabled => false
    index    => "%{[fields][tags]}"  
  }
  stdout {
    codec => rubydebug
  }
}

But below is the filebeat from windows

filebeat.inputs:
- 
  paths:
     - C:\Program Files(x86)\filebeat-7.5.2-windows-x86_64\filebeat.yml
  enabled: true
  input_type: log
  fields:  
    tags: ngta-common-test

  
output.logstash:
  hosts: ["192.168.xx.xx:5044"]

[Badger]

Your linux configuration adds [fields][tags] equal to ngta-common-test, your Windows configuration adds it equal to ngta-common-test2, and your logstash configuration tests for common-ngta-test. I would not expect either to work.

[Mehak_Bhargava]

@badger I miss wrote here but I had the correct tags and names mentioned when I ran the test. I copied the wrong file with different index name. So even with having same tag name in fikebeat and logsstash, I had same issue.

[xeraa]

Do you have a "tags": ["_grokparsefailure"] on the messages that didn't get converted? That would indicate an issue with the parsing. Otherwise the filter isn't applied and we'd probably need some logs to get a better idea what is going wrong.

[Mehak_Bhargava]

@xeraa In the Kibana console, I saw this

tags: beats_input_codec_plain_applied, _grokparsefailure

Also, here is a sample of the log I am sending in the file-

2020-03-05 09:50:48,977 [qtp1121620284-30561]   INFO   c.e.n.c.r.LogEvent - Deleting older logs
2020-03-05 09:50:49,443 [qtp1121620284-30447]   INFO   c.e.n.c.r.LogEvent - Deleting older logs
2020-03-05 09:50:49,641 [qtp1121620284-30498]   INFO   c.e.n.c.r.LogEvent - Deleting older logs

When passed in grok debugger, I get

{
  "javaClass": "c.e.n.c.r.LogEvent",
  "loglevel": "INFO",
  "logmessage": " - Deleting older logs",
  "timestamp": "2020-03-05 09:50:48,977"
}

[xeraa]

"tags": ["_grokparsefailure"] would be on the documents in Elasticsearch that didn't get parsed correctly. Can you show one of those that didn't get parsed as intended?

And I meant Logstash logs — sorry for being too vague there.

[Mehak_Bhargava]

Below is the result of GET /ob-dispatcher-apps/_search where there is no tags: _grokparsefailure. And in my last post I edited, and added this

@Mehak_Bhargava

tags: beats_input_codec_plain_applied, _grokparsefailure

{
        "_index" : "ob-dispatcher-apps",
        "_type" : "_doc",
        "_id" : "yVLrwXABCTD8YXuhkI4v",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "hostname" : "GlobalDemo",
            "ephemeral_id" : "d220af8a-8f75-4c9e-97ce-ebf1da99a1fb",
            "version" : "7.5.2",
            "type" : "filebeat",
            "id" : "980ab6dc-bb04-4b43-abf1-32348b8090a6"
          },
          "log" : {
            "file" : {
              "path" : """C:\Program Files (x86)\ESQ SST\DispatcherApp\logs\dispatcher.log"""
            },
            "offset" : 0
          },
          "input" : {
            "type" : "log"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "GlobalDemo"
          },
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "message" : "2020-03-09 03:36:21,222 [DispatcherScheduler_Worker-6] INFO   o.a.c.h.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect",
          "@version" : "1",
          "@timestamp" : "2020-03-10T00:50:03.806Z",
          "fields" : {
            "tags" : [
              "ob-dispatcher-apps"
            ]
          }
        }
      }

The file coming from server has following log type and grok pattern. I changed the tag to ob-dispatcher-apps

2020-03-04 02:36:21,595 [DispatcherScheduler_Worker-1] INFO   o.a.c.h.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect  
%{DATESTAMP:timestamp}%{SPACE}%{GREEDYDATA}%{LOGLEVEL:loglevel}%{SPACE}%{JAVACLASS:javaClass}%{GREEDYDATA:logmessage}

[Mehak_Bhargava]

Here is the logstash-plain.log of the model where filters didnt get applied as expected from windows server.

[2020-03-09T17:49:53,023][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.4.0"}
[2020-03-09T17:49:54,748][INFO ][org.reflections.Reflections] Reflections took 34 ms to scan 1 urls, producing 20 keys and 40 values 
[2020-03-09T17:49:55,590][INFO ][logstash.outputs.elasticsearch][test] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-03-09T17:49:55,734][WARN ][logstash.outputs.elasticsearch][test] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-03-09T17:49:55,771][INFO ][logstash.outputs.elasticsearch][test] ES Output version determined {:es_version=>7}
[2020-03-09T17:49:55,773][WARN ][logstash.outputs.elasticsearch][test] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-03-09T17:49:55,802][INFO ][logstash.outputs.elasticsearch][test] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2020-03-09T17:49:55,968][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][test] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-03-09T17:49:55,971][INFO ][logstash.javapipeline    ][test] Starting pipeline {:pipeline_id=>"test", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1, :thread=>"#<Thread:0x76694b52 run>"}
[2020-03-09T17:49:56,415][INFO ][logstash.inputs.beats    ][test] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-03-09T17:49:56,444][INFO ][logstash.javapipeline    ][test] Pipeline started {"pipeline.id"=>"test"}
[2020-03-09T17:49:56,510][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:test], :non_running_pipelines=>[]}
[2020-03-09T17:49:56,552][INFO ][org.logstash.beats.Server][test] Starting server on port: 5044
[2020-03-09T17:49:56,816][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-03-09T17:50:00,819][INFO ][logstash.outputs.elasticsearch][test] Elasticsearch pool URLs updated {:changes=>{:removed=>[http://localhost:9200/], :added=>[http://127.0.0.1:9200/]}}
[2020-03-09T17:50:00,824][WARN ][logstash.outputs.elasticsearch][test] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}

And below is the std output in logstash terminal

"message" => "2020-03-09 05:15:03,016 [DispatcherScheduler_Worker-9] INFO   o.a.c.h.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect",
      "@version" => "1",
    "@timestamp" => 2020-03-10T00:50:03.814Z,
        "fields" => {
        "tags" => [
            [0] "ob-dispatcher-apps"
        ]
    }
}
{
           "log" => {
          "file" => {
            "path" => "C:\\Program Files (x86)\\ESQ SST\\DispatcherApp\\logs\\dispatcher.log"
        },
        "offset" => 3551
    },
         "input" => {
        "type" => "log"
    },
         "agent" => {
            "hostname" => "GlobalDemo",
        "ephemeral_id" => "d220af8a-8f75-4c9e-97ce-ebf1da99a1fb",
                "type" => "filebeat",
             "version" => "7.5.2",
                  "id" => "980ab6dc-bb04-4b43-abf1-32348b8090a6"
    },
           "ecs" => {
        "version" => "1.1.0"
    },
          "host" => {
        "name" => "GlobalDemo"
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],

[xeraa]

I'm sorry, but this is pretty hard to follow with edits to earlier posts, the change to ob-dispatcher-apps (not really sure why this is needed),...

If you have any "tags": ["_grokparsefailure"] we don't need to look at the logs but can focus on those messages that failed to parse. What are those documents that have the _grokparsefailure?

If I had to guess I'd say it's a stacktrace. The I/O exception is probably the first line of a stacktrace that gets parsed correctly but the rest isn't.

[Mehak_Bhargava]

Sorry about that. Will just reply as new post now on.

Below is the sample of a log file which when passed, had the _grokparsefailure tag. This log file was passed from linux and showed the extracted fields applied in grok pattern. is this the document you asked for above?

09/10/2019 12:38:09 741   (null)                 DEBUG   61   Filter : 
08/10/2019 12:38:09 742   (null)                 DEBUG   10   Add activty in cache (152782646)
08/10/2019 12:38:09 758   (null)                 DEBUG   10   Add incident activity to cache

Could you explain this a bit more please?

@xeraa

If I had to guess I'd say it's a stacktrace. The I/O exception is probably the first line of a stacktrace that gets parsed correctly but the rest isn't.

[Christian_Dahlqvist]

Not sure if it is relevant, but given that [fields][tags] is an array, could you try changing

if [fields][tags] == "ngta-common-test" {

to

if  "ngta-common-test" in [fields][tags] {
```

[xeraa]

@Mehak_Bhargava

Below is the sample of a log file which when passed, had the _grokparsefailure tag.

The _grokparsefailure tag is set on specific documents in Elasticsearch. I'm interested in a 2 or 3 such documents — the full document please

[Mehak_Bhargava]

Sure, do you mean something like this? If so, I will send 2-3 more samples!

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.1814985,
    "hits" : [
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "zU4srHAB6A9xeN1mnOv3",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.291Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "agent" : {
            "type" : "filebeat",
            "hostname" : "mehak-VirtualBox",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "version" : "7.4.0",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "message" : "  <additionalinfo />",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679173
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "zk4srHAB6A9xeN1mnOv9",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "agent" : {
            "type" : "filebeat",
            "hostname" : "mehak-VirtualBox",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "version" : "7.4.0",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "message" : "  <clientReferenceKey>EZMxzk4quI</clientReferenceKey>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679195
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "z04srHAB6A9xeN1mnesE",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "message" : "  <Category>Dispatch</Category>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679250
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "0E4srHAB6A9xeN1mnesJ",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "message" : "  <externalcategory></externalcategory>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679283
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "0U4srHAB6A9xeN1mnesP",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "message" : "  <targetpartyid>100</targetpartyid>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679324
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "0k4srHAB6A9xeN1mnesV",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "message" : "  <operationtype>ACTIVITY</operationtype>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679362
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "004srHAB6A9xeN1mnesd",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "message" : "  <userid>-101</userid>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679405
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "1E4srHAB6A9xeN1mnesl",
        "_score" : 1.1814985,
        "_source" : {
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "message" : "  <IncidentId>24750142</IncidentId>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679430
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "1U4srHAB6A9xeN1mness",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "message" : "  <externalticketstatus></externalticketstatus>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679467
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "1k4srHAB6A9xeN1mnesx",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "agent" : {
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "hostname" : "mehak-VirtualBox",
            "version" : "7.4.0",
            "type" : "filebeat",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "message" : "  <result>Success</result>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679516
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      }
    ]
  }
}

[Mehak_Bhargava]

There is something that i just noticed regarding _grokparsefailure. @xeraa

1. On linux, for log file log2.log, I see _grokparsefailure in tags as you mentioned above like this- "tags: beats_input_codec_plain_applied, _grokparsefailure ". But the grok filters are not applied.
2. On linux, for log file dispatcher_ob, I dont see _grokparsefailure in tags. But the grok filters are applied with logmessage, javaClass ,loglevel, anum, code all extracted.
3. On Windows server, for ob_dispatcher_app, I dont see _grokparsefailure tag or the filters applied.

There are only three log files I had downloaded a while back which have _grokparsefailure tag.

[Mehak_Bhargava]

@xeraa Below is the document of dispatcher-app-logs which has the filters working as well as _grokparsefailure in tags.

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.1814985,
    "hits" : [
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "zU4srHAB6A9xeN1mnOv3",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.291Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "agent" : {
            "type" : "filebeat",
            "hostname" : "mehak-VirtualBox",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "version" : "7.4.0",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "message" : "  <additionalinfo />",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679173
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "zk4srHAB6A9xeN1mnOv9",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "agent" : {
            "type" : "filebeat",
            "hostname" : "mehak-VirtualBox",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "version" : "7.4.0",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "message" : "  <clientReferenceKey>EZMxzk4quI</clientReferenceKey>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679195
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "z04srHAB6A9xeN1mnesE",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "message" : "  <Category>Dispatch</Category>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679250
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "0E4srHAB6A9xeN1mnesJ",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "message" : "  <externalcategory></externalcategory>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679283
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "0U4srHAB6A9xeN1mnesP",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "message" : "  <targetpartyid>100</targetpartyid>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679324
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "0k4srHAB6A9xeN1mnesV",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "message" : "  <operationtype>ACTIVITY</operationtype>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679362
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "004srHAB6A9xeN1mnesd",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "message" : "  <userid>-101</userid>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679405
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "1E4srHAB6A9xeN1mnesl",
        "_score" : 1.1814985,
        "_source" : {
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "message" : "  <IncidentId>24750142</IncidentId>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679430
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "1U4srHAB6A9xeN1mness",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "agent" : {
            "version" : "7.4.0",
            "hostname" : "mehak-VirtualBox",
            "type" : "filebeat",
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "message" : "  <externalticketstatus></externalticketstatus>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679467
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      },
      {
        "_index" : "dispatcher-app-logs",
        "_type" : "_doc",
        "_id" : "1k4srHAB6A9xeN1mnesx",
        "_score" : 1.1814985,
        "_source" : {
          "@version" : "1",
          "@timestamp" : "2020-03-05T19:28:43.292Z",
          "agent" : {
            "ephemeral_id" : "9b0a94c4-4cf8-4347-97d7-19594af4d99e",
            "hostname" : "mehak-VirtualBox",
            "version" : "7.4.0",
            "type" : "filebeat",
            "id" : "bad135c8-d359-4936-b515-79eb4bb24630"
          },
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "mehak-VirtualBox"
          },
          "message" : "  <result>Success</result>",
          "log" : {
            "file" : {
              "path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log"
            },
            "offset" : 8679516
          },
          "tags" : [
            "beats_input_codec_plain_applied",
            "_grokparsefailure"
          ],
          "fields" : {
            "log_type" : "dispatcher-app-logs"
          }
        }
      }
    ]
  }
}

[Mehak_Bhargava]

@xeraa do you have any insight on what is happening or what to look into? Please let me know! Thanks.

[Mehak_Bhargava]

@xeraa Below is the log document where filters are applied and no _grokparsefailure in tags

{
          "host" => {
        "name" => "mehak-VirtualBox"
    },
           "ecs" => {
        "version" => "1.1.0"
    },
      "loglevel" => "INFO",
      "@version" => "1",
    "logmessage" => " - Deleting older",
       "message" => "2020-03-05 09:51:48,323 [qtp1121620284-30567]   INFO   c.e.n.c.r.LogEvent - Deleting older logs",
     "javaClass" => "c.e.n.c.r.LogEvent",
         "agent" => {
        "ephemeral_id" => "4a55cc11-9299-4131-b6d3-d78426ed50dc",
            "hostname" => "mehak-VirtualBox",
             "version" => "7.4.0",
                "type" => "filebeat",
                  "id" => "bad135c8-d359-4936-b515-79eb4bb24630"
    },
        "fields" => {
        "tags" => "ngta-common-test2"
    },
     "timestamp" => "2020-03-05 09:51:48,323",
    "@timestamp" => 2020-03-11T18:25:34.116Z,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
           "log" => {
        "offset" => 993571,
          "file" => {
            "path" => "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/ngta-common.log"
        }
    }
}
{
          "host" => {
        "name" => "mehak-VirtualBox"
    },
           "ecs" => {
        "version" => "1.1.0"
    },
      "loglevel" => "INFO",
      "@version" => "1",
    "logmessage" => " - Deleting older",
       "message" => "2020-03-05 09:51:48,355 [qtp1121620284-30567]   INFO   c.e.n.c.r.LogEvent - Deleting older logs",
     "javaClass" => "c.e.n.c.r.LogEvent",
         "agent" => {
        "ephemeral_id" => "4a55cc11-9299-4131-b6d3-d78426ed50dc",
            "hostname" => "mehak-VirtualBox",
             "version" => "7.4.0",
                "type" => "filebeat",
                  "id" => "bad135c8-d359-4936-b515-79eb4bb24630"
    },
        "fields" => {
        "tags" => "ngta-common-test2"
    },
     "timestamp" => "2020-03-05 09:51:48,355",
    "@timestamp" => 2020-03-11T18:25:34.116Z,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
           "log" => {
        "offset" => 993668,
          "file" => {
            "path" => "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/ngta-common.log"
        }
    }
}
{
          "host" => {
        "name" => "mehak-VirtualBox"
    },
           "ecs" => {
        "version" => "1.1.0"
    },
      "loglevel" => "INFO",
      "@version" => "1",
    "logmessage" => " - Deleting older",
       "message" => "2020-03-05 09:51:48,386 [qtp1121620284-30576]   INFO   c.e.n.c.r.LogEvent - Deleting older logs",
     "javaClass" => "c.e.n.c.r.LogEvent",
         "agent" => {
        "ephemeral_id" => "4a55cc11-9299-4131-b6d3-d78426ed50dc",
            "hostname" => "mehak-VirtualBox",
             "version" => "7.4.0",
                "type" => "filebeat",
                  "id" => "bad135c8-d359-4936-b515-79eb4bb24630"
    },
        "fields" => {
        "tags" => "ngta-common-test2"
    },
     "timestamp" => "2020-03-05 09:51:48,386",
    "@timestamp" => 2020-03-11T18:25:34.121Z,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
           "log" => {
        "offset" => 993765,
          "file" => {
            "path" => "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/ngta-common.log"
        }
    }
}
{
          "host" => {
        "name" => "mehak-VirtualBox"
    },
           "ecs" => {
        "version" => "1.1.0"
    },
      "loglevel" => "INFO",
      "@version" => "1",
    "logmessage" => " - Deleting older",
       "message" => "2020-03-05 09:51:48,418 [qtp1121620284-30498]   INFO   c.e.n.c.r.LogEvent - Deleting older logs",
     "javaClass" => "c.e.n.c.r.LogEvent",
         "agent" => {
            "hostname" => "mehak-VirtualBox",
                  "id" => "bad135c8-d359-4936-b515-79eb4bb24630",
                "type" => "filebeat",
             "version" => "7.4.0",
        "ephemeral_id" => "4a55cc11-9299-4131-b6d3-d78426ed50dc"
    },
        "fields" => {
        "tags" => "ngta-common-test2"
    },
     "timestamp" => "2020-03-05 09:51:48,418",
    "@timestamp" => 2020-03-11T18:25:34.121Z,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
           "log" => {
        "offset" => 993862,
          "file" => {
            "path" => "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/ngta-common.log"
        }
    }
}
{
          "host" => {
        "name" => "mehak-VirtualBox"
    },
           "ecs" => {
        "version" => "1.1.0"
    },
      "loglevel" => "INFO",
      "@version" => "1",
    "logmessage" => " - Deleting older",
       "message" => "2020-03-05 09:51:48,450 [qtp1121620284-30543]   INFO   c.e.n.c.r.LogEvent - Deleting older logs",
     "javaClass" => "c.e.n.c.r.LogEvent",
         "agent" => {
            "hostname" => "mehak-VirtualBox",
                  "id" => "bad135c8-d359-4936-b515-79eb4bb24630",
             "version" => "7.4.0",
                "type" => "filebeat",
        "ephemeral_id" => "4a55cc11-9299-4131-b6d3-d78426ed50dc"
    },
        "fields" => {
        "tags" => "ngta-common-test2"
    },
     "timestamp" => "2020-03-05 09:51:48,450",
    "@timestamp" => 2020-03-11T18:25:34.121Z,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
           "log" => {
        "offset" => 993959,
          "file" => {
            "path" => "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/ngta-common.log"
        }
    }
}
{
          "host" => {
        "name" => "mehak-VirtualBox"
    },
           "ecs" => {
        "version" => "1.1.0"
    },
      "loglevel" => "INFO",
      "@version" => "1",
    "logmessage" => " - Deleting older",
       "message" => "2020-03-05 09:51:48,481 [qtp1121620284-30567]   INFO   c.e.n.c.r.LogEvent - Deleting older logs",
     "javaClass" => "c.e.n.c.r.LogEvent",
         "agent" => {
            "hostname" => "mehak-VirtualBox",
                  "id" => "bad135c8-d359-4936-b515-79eb4bb24630",
             "version" => "7.4.0",
                "type" => "filebeat",
        "ephemeral_id" => "4a55cc11-9299-4131-b6d3-d78426ed50dc"
    },
        "fields" => {
        "tags" => "ngta-common-test2"
    },
     "timestamp" => "2020-03-05 09:51:48,481",
    "@timestamp" => 2020-03-11T18:25:34.121Z,
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
           "log" => {
        "offset" => 994056,
          "file" => {
            "path" => "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/ngta-common.log"
        }
    }
}

[xeraa]

1. Discuss is best effort support without an SLA. Pinging me multiple times within a day is not how this is supposed to work.
2. The documents with _grokparsefailure help: Those cannot be parsed by your Grok pattern, because if doesn't match your message; for example "message" : " <additionalinfo />". These messages look like additional metadata from your log appender. The trick is not to split them up, but define the right multi line pattern in Filebeat. You can probably build a pattern around not having leading spaces for a new log event; see the Java example in the docs.
3. For the Windows messages that do not get parsed: Please post a full Elasticsearch document of one of those. Maybe you can find such messages looking for documents that do not have javaClass or any other field that you are extracting with your Grok pattern.

PS: Could you log to JSON directly (for example with https://github.com/elastic/ecs-logging-java)? Then you wouldn't need any parsing at all. That would be simpler and more efficient.

[Mehak_Bhargava]

@xeraa Please refer to this topic I made with more discreet information and after adding result of testing to show that same files on logstash and Filebeat run on linux but not on windows.

Logstash working on Linux but not on Windows with same config files

I have one logstash configuration file which I am using for logs coming from both Windows and linux environment. I have the same filebeat configuration for both environment. The log file being shipped via filebeats is also same. Still, the grok filters are not filtering fields on logs coming from windows, whereas they are being filtered for the same log file from linux. I added an exact same log line in my log file, on both windows and linux environment to compare the JSON output to see why g…

I tried the multiline pattern and that didnt work either.
Which Elasticsearch document are you referring to? I posted few above in 13/18 comment.

[Mehak_Bhargava]

@Badger Before my post was missing some information. This link has better understanding of the error. Please refer to it. https://www.claudiokuenzler.com/blog/733/solving-grokparsefailure-syslog-messages-entries-elk-logstash