Mutate WinlogBeat

[Nobby]

Hey guys,

I'm trying to use mutate to define the data type of various fields generated in some Windows logs. For example, Remote Desktop Gateway has number fields for user_data.BytesReceived, user_data.BytesTransfered and user_data.SessionDuration. I have the following config file for beats

input {
   beats {
     port => 5044
     ssl => true
    ssl_certificate => "*******"
    ssl_key => "*******"
  }
}
filter{
 mutate {                                                                                                              
  convert => [ "user_data.BytesReceived", "integer"] 
  convert => [ "user_data.BytesTransfered", "integer"] 
  convert => [ "user_data.SessionDuration", "integer"] 
 }
}

After restarting Logstash and pushing new data through, I am not getting Mapping conflicts from the change. When I created a similar filter for my NGINX logs, a new filter was created automatically and showed mapping conflicts in Kibana. In the case of NGINX, i was using Filebeat with explicit an document_type in the filter configuration for Logstash but prospectors don't seem to exist for Winlogbeat.

There are no errors in the Logstash logs. Is there anything I'm missing here?

[magnusbaeck (Magnus Bäck)]

You're using the wrong syntax for nested fields, see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

I'm not sure integer values stored in string fields creates mapping conflicts.

[Nobby]

Hey Magnus,

Thanks for the doco pointer. I change the filter to:

input {
   beats {
     port => 5044
     ssl => true
    ssl_certificate => "*******"
    ssl_key => "*******"
  }
}
filter{
 mutate {                                                                                                              
  convert => [ "[user_data][BytesReceived]", "integer"] 
  convert => [ "[user_data][BytesTransfered]", "integer"] 
  convert => [ "[user_data][SessionDuration]", "integer"] 
 }
}

And restarted Logstash. As you can see below, the types are still being detected as strings. A new index wasn't automatically created.

You get mapping conflicts when the index is originally of type string then you change to another type like date, integer, float etc which requires you to re-index old data against the new type.

[magnusbaeck (Magnus Bäck)]

You can't change the mapping of an existing field. You have to reindex.

That aside, have you verified by looking at the JSON document being stored in ES that the user_data subfields are integers?

You get mapping conflicts when the index is originally of type string then you change to another type like date, integer, float etc which requires you to re-index old data against the new type.

Numerical values can be stored in fields mapped as strings without creating mapping conflicts. The opposite isn't true.

Yes, if you want to change the mapping of a field you need to reindex but that's not a mapping conflict.

$ curl -XPOST localhost:9200/testindex/testtype/1\?pretty -d '{"stringfield": "hello"}'
{
  "_index" : "testindex",
  "_type" : "testtype",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "created" : true
}
$ curl -XPOST localhost:9200/testindex/testtype/2\?pretty -d '{"stringfield": 123}'   
{
  "_index" : "testindex",
  "_type" : "testtype",
  "_id" : "2",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "created" : true
}