Struggling to specify field types

Elastic Stack Kibana
#1

Hi everyone,

I've got a box running Kibana and Elasticsearch, with the information being handed across by fluentd rather than logstash. In hindsight I'd have picked Logstash, but the guide at the time covered fluentd. It's all working pretty well, and I've moved on to pushing Windows events through - which is where I'm struggling.

I'm using nxlog on the windows machines, and it's pushing the data directly to elasticsearch - unfortunately I cannot for love nor money get the date to be handled as a date, rather than a string. While beyond the scope of this forum (I assume), then nxlog output is:

URL    http://192.0.0.10:9200
ContentType application/json
Exec   set_http_request_path(strftime($EventTime, "/windowsevents-%Y.%m.%d/" + $SourceModuleName)); delete($EventReceivedTime); rename_field("timestamp","@timestamp"); to_json();

I create a new index with a pattern of windowsevents*, but the EventTime field is stated as a string, not a date - so I can't sort data by age. @timestamp clearly isn't getting populated, as the timestamp field isn#t being used. Has anyone used a similar setup? Is there a way to ask Elastic to treat a field as a date?

Apologies if I've missed out pertinent information - it's all a bit new to me.

#2

You may not have any luck here due to this being an NXLog issue, but I know there are a few users of it so hopefully someone can answer you :slight_smile:

#3

Hi there,

Thank you; I suspected that might be the case, but I wasn't sure if there was a means of specifying the data type within elastic/kibana. I wasn't having much luck on the nxlog side. Cheers!

#4

I am not familiar with fluentd, so do not know if it provides some type of default mapping like Logstash does or not. Check if you have any index templates defined that would apply to the index these logs go into. If this is the case you should be able to modify the mapping for the timestamp field so that Elasticsearch maps it as a date.

If you do not have a template defined and rely on dynamic mapping, you can create an index template with a mapping and specify the date formats Elasticsearch uses to identify date fields.

2 Likes
#5

Dear Christian,

You're a hero, thank you. Using the information you referenced, I displayed the templates in use by elasticsearch, and checked the JSON information in Kibana.

"nxlog":{
  "order":0,"template":"nxlog*","settings":{},"mappings":{
    "_default_":{
      "properties":{
        "EventTime":{"format":"YYYY-MM-dd HH:mm:ss","type":"date"}
      }
    }
  },
  "aliases":{}
  }
}

I was pushing the information in as "windowsevents", and using "windowsevents*" as an index template. Obviously I got a bit confused when I was following some other information, and went rogue with the names. I changed the output to read "nxlog*", and bingo! I get presented with "EventTime" as a Time-Field option.

Once again, I thank you!

#6