Need help Mutating a Date column from CSV

(Celtics 2018) #1

Hi everyone,

I am quite new to using the Stack so I apologize for what may be a quite easy question, I am having trouble figuring this one out!

I am loading in a large CSV file into ES and I am mutating some of the fields to load in properly. I can get all of the data loaded in when I am only mutating for integer and float, but when I try to do DATE & GEO it throws errors. I understand that date is not a supported field type so I know something else needs to be done but I am unsure what. I am trying to mutate a DATE field where it looks like YYYY-MM-DD. Here is my config file:

input {
  file {
    path => "/Users/Celtics2018/documents/data/crime.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"

filter {
      separator => ","
      columns =>     ["ID","CaseNumber","Date","Block","IUCR",

    mutate {convert => ["ID", "integer"]}
    **mutate {convert => ["Date", "date"]}**
    mutate {convert => ["IUCR", "integer"]}
    mutate {convert => ["beat", "integer"]}
    mutate {convert => ["District", "integer"]}
    mutate {convert => ["Ward", "integer"]}
    mutate {convert => ["CommunityArea", "integer"]}
    mutate {convert => ["xCoordinate", "integer"]}
    mutate {convert => ["yCoordinate", "integer"]}
    mutate {convert => ["Year", "integer"]}


output {
  elasticsearch {
    hosts => "localhost"
    index => "crime"
    document_type => "LA"

Please let me know if I can provide any more information or if I am unclear. Also, I am trying to mutate Lat/Long fields to GEO, so any help on that would be great too :slight_smile:

Thank you!

(Christian Dahlqvist) #2

Parsing dates require the use of the date filter, and can not be done through mutate.

This blog post describes how geo_point fields need to be formatted and mapped.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.