How to Aggregate Log Content

Elastic Stack Logstash
1

Hi,

I have few log entries as below.

Thu Jul 09 2015 15:00:39 [serviceout][0x8060015e][crypto][info] mpgw(mpgwService): tid(6440055)[request][255.255.215.129] gtid(6440055): Data encryption succeeded
Thu Jul 09 2015 15:00:39 [serviceout][0x8060010b][crypto][info] mpgw(mpgwService): tid(6440055)[request][255.255.215.129] gtid(6440055): certificate validation succeeded
Thu Jul 09 2015 15:00:39 [serviceout][0x80e00073][latency][info] mpgw(mpgwService): tid(6440055)[255.255.215.129] gtid(6440055): Latency: 0 3 0 2 3 2 0 398 399 398 399 399 399 398 2 3 [https://HOSTNAME:PORT/URI]

How can i aggregate three entries using logstash Filter based on gtid and pick the URL from last entry.

what i am trying is as follows.

  • Do Grok Parse the message and create field that has gtid.
  • Do Grep to pick corresponding entries based on tid assign it to a field ---> looking suggestion to implement this.

Thanks,
Dilip

2

Maybe a multiline filter? But that might be a bit clunky :frowning:

I'll wait for Magnus to drop in and provide some advice :wink:

3

Hi,

Thanks for your suggestion.
I tried with Multi-line, but its not working as expected.

Is there any other way to achieve

Thanks,
Dilip

4

Since the last line (with the URL) is basically a superset of the preceding lines, what kind of aggregation are you looking to do? That is, given the three-line snippet in your post, what is the expected outcome?

5

Hi Magnus,

My Implementation is as follows.

what i am trying is as follows.

Do Grok Parse the message and create field that has gtid.
Do Grep to pick corresponding entries based on gtid
Pick the URL from Latency record corresponding to gtid.
Create URL field for all three log entries.

so when the 3 log entries are shipped to Elastic Search, i should have URL filed for all the three log entries.

For Latency record it will not be an issue, but for other two log records i need to create URL field with the URL value from Latency record.

Thanks,
Dilip

6

Oh, okay. There's no stock Logstash plugin that can help you here. I'd either write a custom plugin or have the third message sent to a broker and write a small service that consumes those messages and updates the remaining two entries in ES with the URL.

7

Plugin might be helpful in this case. Might be useful for others as well

Just out of curiosity can logstash-filter-aggregate suits for this scenario.

Thanks,
Dilip

8

I don't think the aggregate filter helps here.