Grok fields not indexed in elasticsearch

[Midiman]

Hi,
I am trying to get grok field values into elasticsearch, but they are just not showing up.
Here's the scenario:

### First, an example syslog message:
<30>Sep 24 19:25:12 win-poc-05t.soca.local MSWinEventLog\t6\tMicrosoft-Windows-ServerManager-MultiMachine/Operational\t751465\tSun Sep 24 19:25:08 2017\t172\tMicrosoft-Windows-ServerManager-MultiMachine\tS-1-5-21-3987990060-1383554934-422310714-1106\tN/A\tInformation\twin-poc-05t.soca.local\t10\tThe description for Event ID 172 from source Microsoft-Windows-ServerManager-MultiMachine cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.If the event originated on another computer, the display information had to be saved with the event.The following information was included with the event: {C97B1F12-4D97-49CE-840D-8EAB05502CCC}. FormatMessage failed with error 15100, The resource loader failed to find MUI file.

### My logstash-syslog-welm.conf file:
input {
  udp {
    port => 514
    type => windows
  }
}

filter {
	grok {
		match => { "message" => "<%{NUMBER:pri}>%{SYSLOGTIMESTAMP:_timestamp}%{SPACE}%{NOTSPACE:source}%{SPACE}MSWinEventLog	%{DATA:level}	%{NOTSPACE:log}	%{NUMBER}	%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NUMBER}%{SPACE}%{NOTSPACE}%{SPACE}%{YEAR}	%{DATA:eventid}	%{GREEDYDATA:windata}" }
	}
	mutate {
		replace => { "type" => "eventlog" }
	}
}

output {
  elasticsearch { hosts => "127.0.0.1:9200" index => "windows" }
  stdout { codec => rubydebug }
  file {
   path => "/var/log/logstash/testlog.log"
   create_if_deleted => true
  }
}

### My command line:
 /usr/share/logstash/bin/logstash -w 4 -f /etc/logstash/conf.d/logstash-syslog-welm.conf --path.settings=.
   I am running 5.6.0

### The stdout console text for the above syslog message:
{
       "eventid" => "172",
    "@timestamp" => 2017-09-24T18:54:17.993Z,
         "level" => "6",
           "log" => "Microsoft-Windows-ServerManager-MultiMachine/Operational",
           "pri" => "30",
      "@version" => "1",
          "host" => "10.30.30.180",
       "windata" => "Microsoft-Windows-ServerManager-MultiMachine\tS-1-5-21-3987990060-1383554934-422310714-1106\tN/A\tInformation\twin-poc-05t.soca.local\t10\tThe description for Event ID 172 from source Microsoft-Windows-ServerManager-MultiMachine cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.If the event originated on another computer, the display information had to be saved with the event.The following information was included with the event: {C97B1F12-4D97-49CE-840D-8EAB05502CCC}. FormatMessage failed with error 15100, The resource loader failed to find MUI file.\n",
        "source" => "win-poc-05t.soca.local",
       "message" => "<30>Sep 24 19:25:12 win-poc-05t.soca.local MSWinEventLog\t6\tMicrosoft-Windows-ServerManager-MultiMachine/Operational\t751465\tSun Sep 24 19:25:08 2017\t172\tMicrosoft-Windows-ServerManager-MultiMachine\tS-1-5-21-3987990060-1383554934-422310714-1106\tN/A\tInformation\twin-poc-05t.soca.local\t10\tThe description for Event ID 172 from source Microsoft-Windows-ServerManager-MultiMachine cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.If the event originated on another computer, the display information had to be saved with the event.The following information was included with the event: {C97B1F12-4D97-49CE-840D-8EAB05502CCC}. FormatMessage failed with error 15100, The resource loader failed to find MUI file.\n",
          "type" => "eventlog",
    "_timestamp" => "Sep 24 19:25:12"
}

### Kibana's field list for the target index after the syslog messages are received: 
(I have refreshed the Kibana Index Pattern)

@timestamp   	date					
@version  	string					
@version.keyword  	string					
_id  	string					
_index  	string					
_score  	number					
_source  	_source					
_type  	string					
host  	string					
host.keyword  	string					
message  	string					
message.keyword  	string					
tags  	string					
tags.keyword  	string					
type  	string					
type.keyword  


As you can see, the grok fields are not there. Yet the console output shows the fields, and no _tag _grokparsefailure.
I tried doing the add_field thing:
add_field => { "pri" => "%{pri}" }
Which seems a bit pointless - although this _does_ create a pri field, whose value is %{pri}, which is of course not correct.

Can anyone help out with the correct syntax/format/options/incantations to get elasticsearch to index the grok fields?

Many thanks,
Peter

[Midiman]

Hi,
I have solved this issue.

This might help other shaving similar issues.

The problem in my case was that one of my grok fields is: _timestamp, which just happens to be a metadata keyword in elastic, so it barfed every time it saw it, thus stopping that and all other fields from being indexed, but not always stopping the event from being stored (with the default fields that were already there).
So always make sure all your fields are unique and reserved.