How to convert a list of string to a list of date in filter?

Elastic Stack Logstash
#1

The input had a field named valid_dates which is a string column from mysql(jdbc).It formatted like this

"2014-09-10 00:00:00,2014-08-21 17:02:02,2014-06-19 00:00:00,2014-08-20 14:29:09"

And the filter was

filter {
    mutate => {
        split => { "valid_dates" => "," }
    }
    ruby {
            code => "require 'date'
                    # I also tried this, failed too.
                    # event['valid_dates']=event['valid_dates'].split(",")
                    event['valid_dates']=event['valid_dates'].map {|dt| Datetime.strptime(dt,'%Y-%m-%d %H:%M:%S')}"
    }
}

The error was

Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}

BTW, I wanna know if logstash filter could support python or some other script languages.

#2

I believe the class is named DateTime, not Datetime.

BTW, I wanna know if logstash filter could support python or some other script languages.

I don't think that's very likely.

#3

Thank you @magnusbaeck. I've corrected the mistake. But it still failed.

Here are what I've tried
The pipeline was input(jdbc) - filter(mutate/ruby) - output(es)

  • Split the valid_dates string in filter and specify the mappings template in output es
filter {
    mutate {
        split => { "valid_dates" => "," }
    }
}
output {
    elasticsearch {
        index => "newmall"
        document_type => "shop"
        document_id => "%{docid}"
        template => "/home/es/logstash-2.2.2/bin/templates/newmall.json"
        hosts => "10.10.20.143:9201"
   }
}

And the template

{
  "template": "newmall",
  "mappings": {
    "shop": {
      "properties": {
        "valid_dates": {
          "type": "date"
        }
      }
    }
  }

But at last the type of valid_dates in es was string.

  • Use both mutate and ruby in filter
filter {
    mutate => {
        split => { "valid_dates" => "," }
    }
    ruby {
            code => "require 'date'
                    event['valid_dates']=event['valid_dates'].map {|dt| DateTime.strptime(dt,'%Y-%m-%d %H:%M:%S')}"
    }
}

The error was

Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: invalid date {:level=>:error}
Ruby exception occurred: invalid date {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}

It seemed that valid_dates didn't turn into array, and was recognized as null.

  • Use ruby only
ruby {
            code => "require 'date'
                    event['valid_dates']=event['valid_dates'].split(',')
                    event['valid_dates']=event['valid_dates'].map {|dt| DateTime.strptime(dt,'%Y-%m-%d %H:%M:%S')}"
    }

The error was

Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: invalid date {:level=>:error}
Ruby exception occurred: invalid date {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}

It seemed that ruby has recognized valid_dates as null. But in the first condition the mutate filter has split the field correctly.

#4

What does the event look like? Use a stdout { codec => rubydebug } output to find out.

#5