Morriaty
(Morry)
April 20, 2016, 9:37am
1
The input had a field named valid_dates
which is a string column from mysql(jdbc).It formatted like this
"2014-09-10 00:00:00,2014-08-21 17:02:02,2014-06-19 00:00:00,2014-08-20 14:29:09"
And the filter was
filter {
mutate => {
split => { "valid_dates" => "," }
}
ruby {
code => "require 'date'
# I also tried this, failed too.
# event['valid_dates']=event['valid_dates'].split(",")
event['valid_dates']=event['valid_dates'].map {|dt| Datetime.strptime(dt,'%Y-%m-%d %H:%M:%S')}"
}
}
The error was
Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::Datetime {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
BTW, I wanna know if logstash filter could support python or some other script languages.
I believe the class is named DateTime, not Datetime.
BTW, I wanna know if logstash filter could support python or some other script languages.
I don't think that's very likely.
Morriaty
(Morry)
April 20, 2016, 10:47am
3
Thank you @magnusbaeck . I've corrected the mistake. But it still failed.
Here are what I've tried
The pipeline was input(jdbc) - filter(mutate/ruby) - output(es)
Split the valid_dates
string in filter and specify the mappings template in output es
filter {
mutate {
split => { "valid_dates" => "," }
}
}
output {
elasticsearch {
index => "newmall"
document_type => "shop"
document_id => "%{docid}"
template => "/home/es/logstash-2.2.2/bin/templates/newmall.json"
hosts => "10.10.20.143:9201"
}
}
And the template
{
"template": "newmall",
"mappings": {
"shop": {
"properties": {
"valid_dates": {
"type": "date"
}
}
}
}
But at last the type of valid_dates
in es was string.
Use both mutate and ruby in filter
filter {
mutate => {
split => { "valid_dates" => "," }
}
ruby {
code => "require 'date'
event['valid_dates']=event['valid_dates'].map {|dt| DateTime.strptime(dt,'%Y-%m-%d %H:%M:%S')}"
}
}
The error was
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: invalid date {:level=>:error}
Ruby exception occurred: invalid date {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `map' for nil:NilClass {:level=>:error}
It seemed that valid_dates
didn't turn into array, and was recognized as null.
ruby {
code => "require 'date'
event['valid_dates']=event['valid_dates'].split(',')
event['valid_dates']=event['valid_dates'].map {|dt| DateTime.strptime(dt,'%Y-%m-%d %H:%M:%S')}"
}
The error was
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: invalid date {:level=>:error}
Ruby exception occurred: invalid date {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
It seemed that ruby has recognized valid_dates
as null. But in the first condition the mutate filter has split the field correctly.
What does the event look like? Use a stdout { codec => rubydebug }
output to find out.