I don't think that the aggregate filter does what I want. I still want to keep all log lines, but with enriched log information from previous log lines. The following is an example output I wish to have:
conn_id=1234 accept connection from 192.168.0.1
conn_id=1234 authentication successful user1 [192.168.0.1]
conn_id=1234 search request xyz from user1 [192.168.0.1]
conn_id=1234 delete request xyz from user1 [192.168.0.1]
conn_id=1234 disconnect user1 [192.168.0.1]
Is there a way to do this with logstash? Also note that a connection might be open for a long time (several hours).
I could imagine extracting data into a key/value store and lookup this? Is something like this possible in logstash?
since you asked me this on IRC i figured I'll reply to this so it may help the next person:
The aggregate filter can do what you want. As long as your log lines are processed in order.
You can:
match your aggregation on the conn_id.
on each event, use the ruby code snippet to write whatever info you need into the map. For example, event 1 (accept connection) would write the IP address into the aggregation map.
On the next event (auth success) you already know that 2 has happened before (if that is not the case you need to consider that in your ruby logic). So you can simply write it back into the event as your code snippets are executed with the current event and the map mathed in context. So, essentially:
in 2.
map['ip'] = event['parsed_ip']
in 3.
event['ip] = map['parsed_ip']
Finally, if you know your last event, you can process it as end-event in the aggregation context. Write whatever you would like into your event. By matching the end-event the plugin will clean up the aggregation map so you don't waste memory.
Define some timeout value that is sensible as well in case you miss an end event.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.