# Logsash pipeline terminating before timeout duration of aggregation filter plugin

**URL:** <https://discuss.elastic.co/t/logsash-pipeline-terminating-before-timeout-duration-of-aggregation-filter-plugin/268112>\
**Category:** Logstash\
**Created:** [March 23, 2021, 3:17pm UTC](https://discuss.elastic.co/t/logsash-pipeline-terminating-before-timeout-duration-of-aggregation-filter-plugin/268112 "2021-03-23T15:17:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![vigneshr35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vigneshr35/32/32636_2.png) [@vigneshr35](https://discuss.elastic.co/u/vigneshr35)\
**Post date:** [March 23, 2021, 3:17pm UTC](https://discuss.elastic.co/t/logsash-pipeline-terminating-before-timeout-duration-of-aggregation-filter-plugin/268112/1 "2021-03-23T15:17:46Z")

</div>

Hi All,

I am having a logstash pipeline that has an elasticsearch plugin for input and aggregation filter plugin. The aggregation filter plugin has a timeout duration of 15 seconds. Refer to the filter plugin configuration

```auto
aggregate {
		task_id => "%{[test.region.keyword]}"
		code => "
				map['total_count_per_region'] ||= 0; 
				map['total_count_per_region'] += event.get('Count');
				"
		push_map_as_event_on_timeout => true
		timeout_task_id_field => "test.region.keyword"
		timeout => 15 
		timeout_tags => ['_aggregatetimeout']
		timeout_code => "
						event.set('total_count_per_region', event.get('total_count_per_region'));
						"
	}

```

My pipeline configuration fetches documents from index as input and the filter plugin performs an aggregation using the aggregation plugin as shown above. This is going to generate aggregation result as an event after 15 seconds. But when I start my pipeline, it starts successfully without issues. However, before the aggregation events are generated(after 15 seconds) the pipeline is terminated and hence I don't get those events in my output.

```auto
 [2021-03-23T14:12:24,610][INFO][logstash.javapipeline][pTest] Starting pipeline {:pipeline_id=>"pTest", "pipeline.workers"=>3, "pipeline.batch.size"=>1500, "pipeline.batch.delay"=>600, "pipeline.max_inflight"=>4500, "pipeline.sources"=>["/etc/logstash/conf.d/test1.conf"], :thread=>"#<Thread:0x7eabfdb5 run>"}
 [2021-03-23T14:12:24,689][INFO][logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :pTest], :non_running_pipelines=>[]}
 [2021-03-23T14:12:28,601][INFO][logstash.javapipeline][pTest] Pipeline terminated {"pipeline.id"=>"pTest"}

```

Please help in how to resolve this

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 23, 2021, 4:52pm UTC](https://discuss.elastic.co/t/logsash-pipeline-terminating-before-timeout-duration-of-aggregation-filter-plugin/268112/2 "2021-03-23T16:52:33Z")

</div>

You are probably hitting [this](https://discuss.elastic.co/t/new-way-in-which-java-execution-breaks-the-aggregate-filter/239377).

---

<div class="post-metadata">

**Author:** ![vigneshr35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vigneshr35/32/32636_2.png) [@vigneshr35](https://discuss.elastic.co/u/vigneshr35)\
**Post date:** [March 29, 2021, 5:06am UTC](https://discuss.elastic.co/t/logsash-pipeline-terminating-before-timeout-duration-of-aggregation-filter-plugin/268112/3 "2021-03-29T05:06:23Z")

</div>

I figured out the reason for the issue I was facing. I was using the elasticsearch input plugin without a _schedule_ attribute and hence, by pipeline ran once when I started/restarted logstash and then terminated after the run. I had to schedule it to run once per day (which was my requirement as well) using the schedule attribute and that made sure that my pipeline was in running state.

This resolved the issue I have posted about. When the aggregation event was generated 15 seconds after my pipeline ran each day, my pipeline is still in running state and the events go through to the output successfully.

Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 26, 2021, 5:06am UTC](https://discuss.elastic.co/t/logsash-pipeline-terminating-before-timeout-duration-of-aggregation-filter-plugin/268112/4 "2021-04-26T05:06:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.