I am syncing data from sql to elastic and in filter in the code block I am appending to map eg features but what is happening is that when I am running the bulk sql query ,not all features are getting appended to each index. eg if index 1 has 10 features then 2 or 3 are getting ingested to elastic but when I am directly specifying the id in = in sql query then all 10 features are getting synced. But the bulk query if I am running separating then also it gives the correct count.
What could be the potential issue? Tried multple things but my map is not getting updated to full features for each hotelIds.
filter {
aggregate {
task_id => "%{HotelCode}"
code => "
map['HotelCode'] ||= event.get('HotelCode')
map['Features']||= []
map['Features'] << {
'FeatureId' => event.get('FeatureId'),
'FeatureIdName' => event.get('FeatureIdName'),
'Group' => event.get('GroupName'),
'Rank' => event.get('Rank')
}
event.cancel()
"
push_previous_map_as_event => true
timeout => 5
}
}