# Aggregate filter logstash not working when the input is elasticsearch

**URL:** https://discuss.elastic.co/t/aggregate-filter-logstash-not-working-when-the-input-is-elasticsearch/173016
**Category:** Logstash
**Created:** [March 19, 2019, 4:44pm UTC](https://discuss.elastic.co/t/aggregate-filter-logstash-not-working-when-the-input-is-elasticsearch/173016 "2019-03-19T16:44:58Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![imaad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imaad/32/48628_2.png) [@imaad](https://discuss.elastic.co/u/imaad)
#### Post date: [March 19, 2019, 4:44pm UTC](https://discuss.elastic.co/t/aggregate-filter-logstash-not-working-when-the-input-is-elasticsearch/173016/1 "2019-03-19T16:44:59Z")

</div>

Hello,

I am wondering why I don't have the same result when I use elasticsearch (instead of a file) as an input in my logstash configuration.

This my log file :

```
{"@timestamp":"2019-03-11T17:12:39.929+01:00","@version":"1","Correlation-ID":"correlation-5","http_type":"Request"}
{"@timestamp":"2019-03-11T17:12:40.588+01:00","@version":"1","Correlation-ID":"correlation-5","http_type":"Response"}
{"@timestamp":"2019-03-11T17:17:42.977+01:00","@version":"1","Correlation-ID":"correlation-6","http_type":"Request"}
{"@timestamp":"2019-03-11T17:17:45.045+01:00","@version":"1","Correlation-ID":"correlation-6","http_type":"Response"}

```

Conf Logstash :

```
input {
file {
    path => "/usr/share/logstash/file.txt"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}
filter{
	aggregate {
       task_id => "%{Correlation-ID}"
       code => "
	   map['LH-Correlation-ID'] = event.get('Correlation-ID')
		map['response_time'] ||= [] 
		map['response_time'] << {'eventtime' => event.get('@timestamp')}
		event.cancel()
	   "
	   	push_previous_map_as_event => true
     }	
}
output{
    stdout {codec => rubydebug }
} 

```

Rubydebug Result is fine :

```
{
             "@version" => "1",
           "@timestamp" => 2019-03-19T16:19:26.933Z,
        "response_time" => [
        [0] {
            "eventtime" => 2019-03-11T16:12:39.929Z
        },
        [1] {
            "eventtime" => 2019-03-11T16:12:40.588Z
        }
    ],
    "Correlation-ID" => "correlation-5"
}
{
         "@version" => "1",
       "@timestamp" => 2019-03-19T16:19:26.935Z,
    "response_time" => [
    [0] {
        "eventtime" => 2019-03-11T16:17:42.977Z
    },
    [1] {
        "eventtime" => 2019-03-11T16:17:45.045Z
    }
],
"Correlation-ID" => "correlation-6"
}

```

When I index this file into Elasticsearch and then query this data and do the same configuration except in the input section :

```
input {
elasticsearch {
			"index" => "business"
			"hosts" => ["http://localhost:9200"]
			 }
}

```

I get this result :

```
{
             "@version" => "1",
           "@timestamp" => 2019-03-19T16:17:22.849Z,
    "Correlation-ID" => "correlation-5",
        "response_time" => [
        [0] {
            "eventtime" => 2019-03-11T16:12:40.588Z
        }
    ]
}
{
             "@version" => "1",
           "@timestamp" => 2019-03-19T16:17:22.866Z,
    "Correlation-ID" => "correlation-6",
                   "tags" => [
            [0] "_aggregatefinalflush"
          ],
        "response_time" => [
        [0] {
            "eventtime" => 2019-03-11T16:17:42.977Z
        }
    ]
}

```

Did someone have an idea ??

Thank you

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 16, 2019, 4:45pm UTC](https://discuss.elastic.co/t/aggregate-filter-logstash-not-working-when-the-input-is-elasticsearch/173016/2 "2019-04-16T16:45:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.