Logdate field with format YYYY-MM-dd HH:mm:ss,SSS converted to 2015-01-01T00:33:33.000Z

Elastic Stack Logstash
1

I am parsing a file with date format as YYYY-MM-dd HH:mm:ss.SSS but in Elastic, it is indexed as 2015-01-01T00:33:33.000Z with a T in between date and time.

date{
	match => ["Date", "yyyy-MM-dd HH:mm:ss" ]
	target => "Date"
	}

How to get the actual format of date, so that its format and type also remains DATE in ES ?

2

Hi @ushadatt,

This problem is likely related to Elasticsearch mapping.

Generally your "Date" field should be recognized automatically as a type:date, but only if the first value indexed was in a proper date format. You can check http://your-es-server:9200/_mapping?pretty , and search for the Date field there - it should show as something like this:

  "Date" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },

If not, you have a few options.

  1. If you think that the auto-recognition failed due to a bad initial value, then you can delete the index and begin again.
  2. Alternately, you could manually map the "Date" field to type:date by modifying Logstash's elasticsearch template. In Short, find the elasticsearch-template.json within your Logstash installation, copy the file somewhere convenient (e.g., /etc/logstash/), and modify it to suit your needs. Make sure you [specify the file's location in the elasticsearch output configuration]((https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template). See Index Templates for more detail.
3

Thanks @PhaedrusTheGreek

Actually my date mapping is correct in ES as I am getting the date field as type DATE.. But the format I am specifying in Logstash is changing in ES to something like this: 2015-01-01T00:33:33.000Z.
Couldn't figure out any solution for it :frowning:

4

@ushadatt, This is the due to the output of the date filter.

The solution is to not use the date filter at all. Simply index your date string as a regular field, and Elasticsearch can recognize it. I say 'can' because in your case, the date string will not parse because of the space between the date and time (by default). To work around that, you can either change your date format, or you can specify the mapping as a custom date format in Elasticsearch by updated the properties section of the logstash template like so:

"properties": {
        "Date": {
                "type": "date",
                "format": "yyyy-MM-dd HH:mm:ss"
        }
}

Here is the Elastic reference on the Date Format mapping:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html