Hi,
I try to use logstash.filters.aggregate plugin.
I get an aggregate exception.
Aggregate exception occurred
{:error=>#<NoMethodError: undefined method ` ' for nil:NilClass>, :code=>"\n\t\t\tmap['user']['scopes'] << {'id' => 'asd', 'name' => 'asd'}\n\t\t\tevent.cancel()\n\t\t", :map=>{}, :event_data=>{"@timestamp "=>2019-05-12T14:28:00.803Z, "user.scopes.id"=>nil, "user.scopes.name"=>"xxx", "@version "=>"1", "id"=>1000002, "type"=>"yyy", "user"=>{"tel2"=>nil, "address"=>nil, "relatedbapuser"=>nil, "university"=>"xxx", "titleId"=>xxx, "departmentId"=>xxx, "title"=>"xxx", "isdeleted"=>"false", "countryId"=>xxx, "relatedid"=>nil}}}
logstash.conf
aggregate {
task_id => '%{id}'
code => "
map['user']['scopes'] << {'id' => 'asd', 'name' => 'asd'}
event.cancel()
"
push_previous_map_as_event => true
timeout => 3
}
Do you have any suggestion about this exception?
Try adding
map['user'] ||= {}
map['user']['scopes'] ||= []
Thanks but not work
In the code block changes not effected to output. But remove fields in remove_field part works.
input {
jdbc {
jdbc_driver_library => "...\sqljdbc42.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "..."
jdbc_user => "..."
jdbc_password => "..."
jdbc_validate_connection => true
clean_run => true
schedule => "..."
type => "a_type"
statement => "..."
}
}
filter {
if [type] == "a_type"{
aggregate {
task_id => '%{user.id}'
code => "
map['user'] ||= {}
map['user']['id'] = event.get('user.id')
map['user']['scopes'] ||= []
map['user']['scopes'] << {'id' => event.get('user.scopes.id'), 'name' => event.get('user.scopes.name')}
"
remove_field => [ "user.id", "user.scopes.id", "user.scopes.name" ]
}
}
}
output {
if [type] == "a_type"{
elasticsearch {
hosts => ["localhost:9200"]
index => "xxx"
document_type => "yyy"
document_id => "%{id}"
}
}
}
In your sample data you do not have a [user.id] field, just [id]
Yes, you are right. I change some piece of config. But is it matter id or user.id field exist?
Some fields return from sql are below:
id, user.id, user.name, user.surname, user.email, user.scopes.id, user.scopes.name
You are right, that might prevent it working (if the task_id field does not exist) but it would not result in that error. I would expect that error if you commented out
map['user'] ||= {}
I suggest running "--log.level debug --config.debug --config.test_and_exit" and see if the configuration is the one you expect (that you do not, for example, have both my.conf and my.conf.bck getting picked up).
system
(system)
Closed
June 10, 2019, 4:17pm
7
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.