How to Aggregate Log Content


(DilipKumarVarma Jetty) #1

Hi,

I have few log entries as below.

Thu Jul 09 2015 15:00:39 [serviceout][0x8060015e][crypto][info] mpgw(mpgwService): tid(6440055)[request][255.255.215.129] gtid(6440055): Data encryption succeeded
Thu Jul 09 2015 15:00:39 [serviceout][0x8060010b][crypto][info] mpgw(mpgwService): tid(6440055)[request][255.255.215.129] gtid(6440055): certificate validation succeeded
Thu Jul 09 2015 15:00:39 [serviceout][0x80e00073][latency][info] mpgw(mpgwService): tid(6440055)[255.255.215.129] gtid(6440055): Latency: 0 3 0 2 3 2 0 398 399 398 399 399 399 398 2 3 [https://HOSTNAME:PORT/URI]

How can i aggregate three entries using logstash Filter based on gtid and pick the URL from last entry.

what i am trying is as follows.

  • Do Grok Parse the message and create field that has gtid.
  • Do Grep to pick corresponding entries based on tid assign it to a field ---> looking suggestion to implement this.

Thanks,
Dilip


(Mark Walkom) #2

Maybe a multiline filter? But that might be a bit clunky :frowning:

I'll wait for Magnus to drop in and provide some advice :wink:


(DilipKumarVarma Jetty) #3

Hi,

Thanks for your suggestion.
I tried with Multi-line, but its not working as expected.

Is there any other way to achieve

Thanks,
Dilip


(Magnus Bäck) #4

Since the last line (with the URL) is basically a superset of the preceding lines, what kind of aggregation are you looking to do? That is, given the three-line snippet in your post, what is the expected outcome?


(DilipKumarVarma Jetty) #5

Hi Magnus,

My Implementation is as follows.

what i am trying is as follows.

Do Grok Parse the message and create field that has gtid.
Do Grep to pick corresponding entries based on gtid
Pick the URL from Latency record corresponding to gtid.
Create URL field for all three log entries.

so when the 3 log entries are shipped to Elastic Search, i should have URL filed for all the three log entries.

For Latency record it will not be an issue, but for other two log records i need to create URL field with the URL value from Latency record.

Thanks,
Dilip


(Magnus Bäck) #6

Oh, okay. There's no stock Logstash plugin that can help you here. I'd either write a custom plugin or have the third message sent to a broker and write a small service that consumes those messages and updates the remaining two entries in ES with the URL.


(DilipKumarVarma Jetty) #7

Plugin might be helpful in this case. Might be useful for others as well

Just out of curiosity can logstash-filter-aggregate suits for this scenario.

Thanks,
Dilip


(Magnus Bäck) #8

I don't think the aggregate filter helps here.


(system) #9