We are processing some MSSQL tables via JDBC input plugin. Some times, 3 rolls will be almost equal. Our goal is merge these lines together and transform the values that are different in an array. So, given this 3 events
Unfortunaly, aggregate plugin can't help you as it works today.
It needs a clearly identified "task end event" to push aggregate map into end task event.
In that case, we can clearly detect "task start event" : it is the first one with id=1.
To fill the need, I plan to add a new option in aggregate plugin :
push_previous_map_as_event => true
When this option is activated, each time a new task is detected (in your case, each time a new id is detected), it automatically push previous aggregate map as a new event in logstash.
Using that feature in your case, you can enrich aggregate map with each db line result, and when aggregate detects a new id, it push map with all aggregated data info as new event.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.