# Visualizing aggregated logs **URL:** \ **Category:** Logs\ **Tags:** ecs-elastic-common-schema\ **Created:** [September 22, 2022, 5:36pm UTC](https://discuss.elastic.co/t/visualizing-aggregated-logs/314966 "2022-09-22T17:36:02Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![rsk0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rsk0/32/124810_2.png) [@rsk0](https://discuss.elastic.co/u/rsk0)\ **Post date:** [September 22, 2022, 5:36pm UTC](https://discuss.elastic.co/t/visualizing-aggregated-logs/314966/1 "2022-09-22T17:36:03Z")
Anyone aggregating duplicate log messages? That is, are you counting up identical messages in your logging library or in Logstash, then emitting a single message saying > (original message) ... > This message was repeated 987 times. When you try to visualize log rate in Kibana by sheer count of messages, _your visualization is now incorrect_. I’ve been thinking about how to regain accuracy, insight into the actual message rate the source is generating. We should be able to see how much an application is actually trying to output. **So I thought it might be useful to add a field to indicate if a message is an aggregation.** Maybe something like “repeat\_count”. An ingest pipeline could assure it’s set, making it default to 1. Then rather than visualize by sheer count of messages you can visualize by sum of “repeat\_count”. Q1. Are there any existing solutions? Does ECS already have a field that I missed? Q2. Do you agree about the problem? Q3. Do you see this is a reasonable solution? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 22, 2022, 6:38pm UTC](https://discuss.elastic.co/t/visualizing-aggregated-logs/314966/2 "2022-09-22T18:38:31Z")
Hi @rsk0 You can set `_doc_count` 🙂 Pretty much exactly for your use case... > **[\_doc\_count field | Elasticsearch Guide \[8.4\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-doc-count-field.html)** Oddly, I was just playing with this the other day.... It works in Lens too etc for Record counts etc... ![Screen Shot 2022-09-19 at 12.37.30 PM](https://us1.discourse-cdn.com/elastic/original/3X/b/1/b1203974261c116b0a6a8af344eca6b2916cae2e.png) ---
**Author:** ![rsk0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rsk0/32/124810_2.png) [@rsk0](https://discuss.elastic.co/u/rsk0)\ **Post date:** [September 23, 2022, 1:18am UTC](https://discuss.elastic.co/t/visualizing-aggregated-logs/314966/3 "2022-09-23T01:18:54Z")
Brilliant! I love that this feature's in ES. Thanks for the pointer, Stephen. Now I need a way to convey the `_doc_count` data from the upstream aggregators so that it turns into the `_doc_count` field once it gets into ES. I haven't seen a field in ECS for this, but let me know if you have. If anyone has any ideas about how to represent this in ECS, I'd be glad to hear. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 23, 2022, 2:20am UTC](https://discuss.elastic.co/t/visualizing-aggregated-logs/314966/4 "2022-09-23T02:20:00Z")
I do not think there is an ECS Field that represents this. You could add a tag : aggregation or something And / or you could open a feature request 🙂 There looks like there may be a related issue open on this topic, perhaps your is slightly different > > > \*\*Summary\*\* > > Add fields for counting repeated or related events. This is not a… concrete proposal. I'm just dumping information here in the hopes that over time, others may come up with other example, and a pattern may show itself. > > \*\*Motivation\*\*: > > In several firewalls, proxies, and load balancers that I've worked with (different vendors too), there is a notion of "how many times did event X happen?" Here are a few examples: > > \* Palo Alto \[traffic\](https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/traffic-log-fields.html) and threat logs: "Repeat Count (repeatcnt) | Number of sessions with same Source IP, Destination IP, Application, and Subtype seen within 5 seconds." > \* A10 DDOS logs (CEF): Example of this is \`CEF:0|A10| ... cnt=3327 src=192.0.2.33 dst=192.0.2.5 act=drop\`. I believe that it this case, it's the number of ICMP packets from the same source to the same destination. > \* Fortinet: \`countips\` field means "Number of the IPS logs associated with the session", which is a little bit different. Maybe it should not use the same field as the others. It's also got \`countweb\` and \`countapp\` and several other count fields. So, these are not really repeat counts like they are for PA and A10. > > To my recollection, the notion of suppressing repeats and providing a counter of how many times the same thing happened shows up in log aggregation software like rsyslog (open source) and logrythm (paid). It's been a while since I've worked with either of those tools though, so I cannot provide an example, and I could be mistaken. You could also create a runtime field... ![Screen Shot 2022-09-22 at 7.31.30 PM](https://us1.discourse-cdn.com/elastic/original/3X/3/5/35c52d9fff7cae4304f170d75dd4972ba244dec4.png) ---
**Author:** ![rsk0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rsk0/32/124810_2.png) [@rsk0](https://discuss.elastic.co/u/rsk0)\ **Post date:** [September 30, 2022, 12:09pm UTC](https://discuss.elastic.co/t/visualizing-aggregated-logs/314966/5 "2022-09-30T12:09:00Z")
Great info, thanks. I'll follow that GitHub issue. I [commented](https://github.com/elastic/ecs/issues/1095#issuecomment-1263000547) to suggest `event.count` for a field name. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 28, 2022, 12:09pm UTC](https://discuss.elastic.co/t/visualizing-aggregated-logs/314966/6 "2022-10-28T12:09:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.