Remembering Field Values


#1

Hi,

Does anyone have any solutions for how to store and re-use field values between events?

For example if we have a log file that contains something like:-

Counter 1
stuff....
stuff....
Counter 2
stuff...
stuff...
stuff...
Counter 3
stuff...
stuff...

and so on. I need to be able to add the counter number which is currently active to the fields I'm grokking out of the "stuff..." lines.

I believe that this is possible with the "aggregate" filter but I have never been able to get it to work, or seen an example of how it could be done. I know that there is a community plugin called memorize but I'd rather try to accomplish this with standard plugins, if possible.

Thanks!


(Fabien Baligand) #2

aggregate seems to be the right plugin for your need.
If you look at logstash "master" documentation, you should find 4 examples explaining how you can use aggregate plugin.
Have you seen it ?


#3

@fbaligand
Thanks, yes, I have seen it. But not been able to find a way to leverage those example to solve the -seemingly simple- problem outlined in my first post. Can you?


(Fabien Baligand) #4

Like that, your need is not precise enough so that I can help you.
Can you give some documents before aggregate plugin, and the expected result after aggregate plugin ?


(Fabien Baligand) #5

After reading one more time your need, I think this kind of configuration could do the job :

if [type] == "Counter" {
  aggregate {
    task_id => "%{host}%{path}"
    code => "map['counter_id'] = event.get('counter_id')"
  }
}

if [type] != "Counter" {
  aggregate {
    task_id => "%{host}%{path}"
    code => "event.set('counter_id', map['counter_id'])"
  }
}

#6

@fbaligand
Thank you very much for your interest. I tried to implement your code but no joy just yet.
Here is what I have; hopefully it will be obvious where I am going wrong?

The (test) log file I am reading looks like this:

cycle 1
name: jellybeans
some: chocolate
cycle 2
name: bubblegum
name: gumdrops
name: candycanes
cycle 3
name: sherbet
name: candyfloss
cycle 4
name: mintdrops
name: icecream

My conf file is this:

input {
	file {
		path => "/Users/me/Downloads/logstash-2.3.1/source.log"
		ignore_older => 0
	}
}

filter {
	
	if [message] =~ /.*cycle.*/ {
		
		grok {
			match => { "message" => ".*cycle\s(?<cycle>\d*)"}
		}
	}
	
	if [cycle] {
		aggregate {
			task_id => "%{host}%{path}"
			code => "map['cycle'] = event.get('cycle')"
		}
	} else {
		aggregate {
			task_id => "%{host}%{path}"
			code => "event.set('cycle', map['cycle'])"
		}
	}	
}

output {
	stdout {
		codec => rubydebug
	}
}

Which I think is pretty much representative of the code/behaviour you suggested. I am not differentiating messages by [type] however as all my events are coming from the same log file so I cannot see where or why I would use different types.. I was thinking that I could just trigger the aggregate events based on what the event contains (a cycle value or not).

The output I get is basically this:

Aggregate exception occurred. Error: undefined method `set' for #<LogStash::Event:0x6173639f> ; Code: event.set('cycle', map['cycle']) ; Map: {} ; EventData: {"message"=>"name: candyfloss", "@version"=>"1", "@timestamp"=>"2017-02-04T20:32:10.648Z", "path"=>"/Users/me/Downloads/logstash-2.3.1/source.log", "host"=>"MBP"} {:level=>:error}
Aggregate exception occurred. Error: undefined method `get' for #<LogStash::Event:0x13833cc8> ; Code: map['cycle'] = event.get('cycle') ; Map: {} ; EventData: {"message"=>"cycle 4", "@version"=>"1", "@timestamp"=>"2017-02-04T20:32:10.648Z", "path"=>"/Users/me/Downloads/logstash-2.3.1/source.log", "host"=>"MBP", "cycle"=>"4"} {:level=>:error}
Aggregate exception occurred. Error: undefined method `set' for #<LogStash::Event:0x35a0a565> ; Code: event.set('cycle', map['cycle']) ; Map: {} ; EventData: {"message"=>"name: mintdrops", "@version"=>"1", "@timestamp"=>"2017-02-04T20:32:10.649Z", "path"=>"/Users/me/Downloads/logstash-2.3.1/source.log", "host"=>"MBP"} {:level=>:error}
Aggregate exception occurred. Error: undefined method `set' for #<LogStash::Event:0x74778748> ; Code: event.set('cycle', map['cycle']) ; Map: {} ; EventData: {"message"=>"name: icecream", "@version"=>"1", "@timestamp"=>"2017-02-04T20:32:10.649Z", "path"=>"/Users/me/Downloads/logstash-2.3.1/source.log", "host"=>"MBP"} {:level=>:error}
{
       "message" => "cycle 1",
      "@version" => "1",
    "@timestamp" => "2017-02-04T20:32:10.645Z",
          "path" => "/Users/me/Downloads/logstash-2.3.1/source.log",
          "host" => "MBP",
         "cycle" => "1",
          "tags" => [
        [0] "_aggregateexception"
    ]
}
{
       "message" => "name: jellybeans",
      "@version" => "1",
    "@timestamp" => "2017-02-04T20:32:10.646Z",
          "path" => "/Users/me/Downloads/logstash-2.3.1/source.log",
          "host" => "MBP",
          "tags" => [
        [0] "_aggregateexception"
    ]
}
{
       "message" => "name: chocolate",
      "@version" => "1",
    "@timestamp" => "2017-02-04T20:32:10.647Z",
          "path" => "/Users/me/Downloads/logstash-2.3.1/source.log",
          "host" => "MBP",
          "tags" => [
        [0] "_aggregateexception"
    ]
}

As I am sure you are aware I am looking for the current value of cycle to be present in all events.

Really appreciate this!


(Fabien Baligand) #7

Ok, I saw in your error stack trace that you use logstash 2.3.1. The code that I provided requires logstash 2.4 or more.

With logstash 2.3.1, the right code are :
code => "map['cycle'] = event['cycle']"

And then :
code => "event['cycle'] = map['cycle']"


#8

Brilliant. Success!
Thank you very much, this is great news... I've needed this functionality for a long time.

Thanks again :grinning:


(Fabien Baligand) #9

Happy to see I solved your problem !


(Fabien Baligand) #10

If your problem is solved, could you mark my solution comment as "solution" to close the topic ?


#11

@fbaligand

Thats odd. I did exactly that about 17 seconds after testing your solution. The topic IS tagged as having a solution in my view of this thread:

Is there somewhere else | something else I should do?


(Fabien Baligand) #12

Sorry, I should buy some glasses :slight_smile:


#13

No problem at all. I'm still smiling at FINALLY having a solution to this problem.
THANKS AGAIN


(system) #14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.