Csv filter convert to date doesn't work

Elastic Stack Logstash
1

I have a column called Birthday that is formatted as dd/MM/yyyy (i.e., 3/8/1983) and would like Logstash to treat this as a date when passing to Elasticsearch so the mapping is auto-generated as a date.

This does not work - Elasticsearch is generating the mapping as text.

filter {
csv {
columns => [
"Age",
"Birthday",
"College",
"Name",
"Weight"
]
convert => {
"Age" => "integer"
"Birthday" => "date"
"Weight" => "integer"
}
}
}

Elasticsearch generates the mapping for this field like this:

"Birthday": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},

I also tried to use the mutate filter but that does not work either, it's still mapped as text.

What am I doing wrong?

2

What does an example document look like?

3

HI Magnus,

The actual setup can be found in my elasticsearch-injest-logstash Github project. I'm specifically loading the only CSV file found in the csv directory.

I simplified my question for this forum and reduced the number of actual columns so a sample row could look like this:

56,9/11/1960,Louisiana Tech,"Fowler, Bobby",230
30,9/30/1986,LSU,"Johnson, Quinn",255
4

You could use a date filter on that field, I can't comment on why it's not working though sorry.

5

I meant what a document going out of Logstash looked like but I was able to reconstruct it:

$ echo '56,9/11/1960,Louisiana Tech,"Fowler, Bobby",230' | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
{
       "message" => "56,9/11/1960,Louisiana Tech,\"Fowler, Bobby\",230",
      "@version" => "1",
    "@timestamp" => "2017-10-31T06:00:12.150Z",
          "host" => "lnxolofon",
           "Age" => 56,
      "Birthday" => "9/11/1960",
       "College" => "Louisiana Tech",
          "Name" => "Fowler, Bobby",
        "Weight" => 230
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

I'm not sure how the csv filter's date conversion is supposed to work but it doesn't appear to be doing anything for the date. As Mark says you can use a date filter to convert it to an ISO8601 timestamp that ES will recognize out of the box but you could also adjust the index mappings so that it treats mm/dd/yyyy strings as dates.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.