Hello, When I am using the aggregate filter in logstash (to get the total sales of each product, for example), is there a way to send the aggregated results to a different output and not line by line? Thank you!

Thank you!! We tried to implement the suggestion you gave. However, we have a large quantity of data and we want to read all the data and, in the end, show the aggregation results. We tried to maximize the timeout, but we never know when it will have the final results. Here it is the code, if you c…

If I am reading it correctly, that elasticsearch input re-reads the entire index once a minute. Is that right? The aggregate looks right, although a 4 second timeout is not very long. What problem are you having?

Yes, we notice that schedule does not make sense here, we deleted. The problem is that logstash takes some time to get the results from elasticsearch, and if that timeout defined is higher that the time that logstash takes to read elasticsearch, the logstash will stop before showing any aggregation…

[image] Francisca_Lima: if that timeout defined is higher that the time that logstash takes to read elasticsearch, the logstash will stop before showing any aggregation results lower, not higher. Why not use an 1800 second timeout?

But if logstash stops before that 1800 seconds, the results will not appear as desired because logstash execution stopped as soon as read everything from elasticsearch. Right?

Can you use push_previous_map_as_event ? That will cause logstash to flush the map when it exits.

We found the solution ordering the data from elasticsearch, because every time a new process came, the results were presented and the aggregations restarted. Thanks!

Is it possible to aggregate results (like process and total value) and then for each process do a lookup to elasticsearch and an aggregate to get as final result a list of: process, processID, total value? I tried to do another aggregation like this by processID, but in the final result it only app…

I cannot really answer that without understanding more about the data layout. Note that you can stash additional fields in the map, even if they are constant for a given task id, and they will get included in the aggregated event.

Logstash - aggregate results

Elastic Stack Logstash

Badger January 15, 2019, 6:30pm 2

Yes, you can drop the lines that are not aggregated and just keep the aggregations. See here for an example. If you need more details then you need to show us your input data and what your current aggregate filter configuration looks like.

Topic		Replies	Views
Aggregate filter Logstash	1	903	October 24, 2017
Aggregate filter timeouts and input completion Logstash	4	798	July 12, 2021
Aggregation EPS and Average Elapsed Time Logstash	9	704	March 19, 2019
Logstash Aggregate plugin not working Logstash	7	1288	October 27, 2020
Aggregate filter logstash not working when the input is elasticsearch Logstash	1	682	April 16, 2019

Logstash - aggregate results

Related topics