Visualizing aggregated logs

rsk0 · September 22, 2022, 5:36pm

Anyone aggregating duplicate log messages?

That is, are you counting up identical messages in your logging library or in Logstash, then emitting a single message saying

(original message) ...
This message was repeated 987 times.

When you try to visualize log rate in Kibana by sheer count of messages, your visualization is now incorrect.

I’ve been thinking about how to regain accuracy, insight into the actual message rate the source is generating. We should be able to see how much an application is actually trying to output.

So I thought it might be useful to add a field to indicate if a message is an aggregation.

Maybe something like “repeat_count”.

An ingest pipeline could assure it’s set, making it default to 1. Then rather than visualize by sheer count of messages you can visualize by sum of “repeat_count”.

Q1. Are there any existing solutions? Does ECS already have a field that I missed?
Q2. Do you agree about the problem?
Q3. Do you see this is a reasonable solution?

stephenb · September 22, 2022, 6:38pm

Hi @rsk0

You can set _doc_count Pretty much exactly for your use case...

Oddly, I was just playing with this the other day.... It works in Lens too etc for Record counts etc...

rsk0 · September 23, 2022, 1:18am

Brilliant! I love that this feature's in ES. Thanks for the pointer, Stephen.

Now I need a way to convey the _doc_count data from the upstream aggregators so that it turns into the _doc_count field once it gets into ES. I haven't seen a field in ECS for this, but let me know if you have.

If anyone has any ideas about how to represent this in ECS, I'd be glad to hear.

stephenb · September 23, 2022, 2:20am

I do not think there is an ECS Field that represents this.

You could add a tag : aggregation or something

And / or you could open a feature request

There looks like there may be a related issue open on this topic, perhaps your is slightly different

github.com/elastic/ecs

Add fields for counting repeated or related events

opened 03:11PM - 05 Nov 20 UTC

andrewthad

enhancement ready

**Summary** Add fields for counting repeated or related events. This is not a… concrete proposal. I'm just dumping information here in the hopes that over time, others may come up with other example, and a pattern may show itself. **Motivation**: In several firewalls, proxies, and load balancers that I've worked with (different vendors too), there is a notion of "how many times did event X happen?" Here are a few examples: * Palo Alto [traffic](https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/traffic-log-fields.html) and threat logs: "Repeat Count (repeatcnt) | Number of sessions with same Source IP, Destination IP, Application, and Subtype seen within 5 seconds." * A10 DDOS logs (CEF): Example of this is `CEF:0|A10| ... cnt=3327 src=192.0.2.33 dst=192.0.2.5 act=drop`. I believe that it this case, it's the number of ICMP packets from the same source to the same destination. * Fortinet: `countips` field means "Number of the IPS logs associated with the session", which is a little bit different. Maybe it should not use the same field as the others. It's also got `countweb` and `countapp` and several other count fields. So, these are not really repeat counts like they are for PA and A10. To my recollection, the notion of suppressing repeats and providing a counter of how many times the same thing happened shows up in log aggregation software like rsyslog (open source) and logrythm (paid). It's been a while since I've worked with either of those tools though, so I cannot provide an example, and I could be mistaken.

You could also create a runtime field...

Screen Shot 2022-09-22 at 7.31.30 PM

rsk0 · September 30, 2022, 12:09pm

Great info, thanks. I'll follow that GitHub issue. I commented to suggest event.count for a field name.

system · October 28, 2022, 12:09pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.