Aggregate filter logstash not working when the input is elasticsearch

Elastic Stack Logstash
1

Hello,

I am wondering why I don't have the same result when I use elasticsearch (instead of a file) as an input in my logstash configuration.

This my log file :

{"@timestamp":"2019-03-11T17:12:39.929+01:00","@version":"1","Correlation-ID":"correlation-5","http_type":"Request"}
{"@timestamp":"2019-03-11T17:12:40.588+01:00","@version":"1","Correlation-ID":"correlation-5","http_type":"Response"}
{"@timestamp":"2019-03-11T17:17:42.977+01:00","@version":"1","Correlation-ID":"correlation-6","http_type":"Request"}
{"@timestamp":"2019-03-11T17:17:45.045+01:00","@version":"1","Correlation-ID":"correlation-6","http_type":"Response"}

Conf Logstash :

input {
file {
    path => "/usr/share/logstash/file.txt"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}
filter{
	aggregate {
       task_id => "%{Correlation-ID}"
       code => "
	   map['LH-Correlation-ID'] = event.get('Correlation-ID')
		map['response_time'] ||= [] 
		map['response_time'] << {'eventtime' => event.get('@timestamp')}
		event.cancel()
	   "
	   	push_previous_map_as_event => true
     }	
}
output{
    stdout {codec => rubydebug }
}

Rubydebug Result is fine :

{
             "@version" => "1",
           "@timestamp" => 2019-03-19T16:19:26.933Z,
        "response_time" => [
        [0] {
            "eventtime" => 2019-03-11T16:12:39.929Z
        },
        [1] {
            "eventtime" => 2019-03-11T16:12:40.588Z
        }
    ],
    "Correlation-ID" => "correlation-5"
}
{
         "@version" => "1",
       "@timestamp" => 2019-03-19T16:19:26.935Z,
    "response_time" => [
    [0] {
        "eventtime" => 2019-03-11T16:17:42.977Z
    },
    [1] {
        "eventtime" => 2019-03-11T16:17:45.045Z
    }
],
"Correlation-ID" => "correlation-6"
}

When I index this file into Elasticsearch and then query this data and do the same configuration except in the input section :

input {
elasticsearch {
			"index" => "business"
			"hosts" => ["http://localhost:9200"]
			 }
}

I get this result :

{
             "@version" => "1",
           "@timestamp" => 2019-03-19T16:17:22.849Z,
    "Correlation-ID" => "correlation-5",
        "response_time" => [
        [0] {
            "eventtime" => 2019-03-11T16:12:40.588Z
        }
    ]
}
{
             "@version" => "1",
           "@timestamp" => 2019-03-19T16:17:22.866Z,
    "Correlation-ID" => "correlation-6",
                   "tags" => [
            [0] "_aggregatefinalflush"
          ],
        "response_time" => [
        [0] {
            "eventtime" => 2019-03-11T16:17:42.977Z
        }
    ]
}

Did someone have an idea ??

Thank you

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.