Date Field in in Logstash

[Alok]

I have some date fields in CSV which I would like to have as Type Date in the Index.
When the index is created the type remains 'text' despite many ways of handling it.

The config file looks like as under :

input {
  file {
    path => "D:\test.csv"
    start_position => "beginning"
   sincedb_path => "/dev/null"
  }
}
filter {
  csv {
      separator => ","
     columns => 
    ["A","B","C","Date_field1","date_field2"] 
  }

     mutate{
        convert => {
        "B" => "integer"
          }
        }
  date {
match => [ "Date_field1", "dd-MM-yyyy HH:mm:ss.SSS" ]
target => "Date_field1"
}		
    }
output {
   elasticsearch {
     hosts => "localhost:9200"
     index => "test"
  }
stdout {codec=>rubydebug}

I am getting errors in logstash like "_dateparsefailure"

The index is created with Date_field1 being type text

Please help.

[yaauie (Ry Biesemeyer)]

First, what are some example date strings in your CSV? The _dateparsefailure tag indicates that none of the provided patterns matched, so it was unable to parse the string. The format specification for the match directive is described in detail in the Logstash Date Filter Plugin docs.

---

Second, you may want to configure your Elasticsearch output to manage your index templates for you using the template directive. Doing so will ensure that new indices handle fields in the way you expect (installing a template will not affect the mappings of existing indices).

When Elasticsearch receives a document for an index that has not yet been created, it looks through the registered templates for one or more whose pattern matches the index name. These templates define the mappings of fields to their types. When an Elasticsearch index receives a document that contains a field that is not yet defined on the index, it makes a "best guess" as to the type of the field. Subsequent documents will be coerced to that type.

What that means, is if the Date_field1 field has already been created as a text field in your index, no matter if you change it to a date in Logstash, when the document is inserted into Elasticsearch, since the index's existing type for the field is text, it will coerce the given value into text value.

[Alok]

Thanks. Will try via template way