New way in which java_execution breaks the aggregate filter

Elastic Stack Logstash
1

When java_execution is enabled, the periodic_flush does not occur, so you cannot use a timeout to trigger flushing an event from an aggregate filter. I do not think the final flush occurs either. This is in 7.8.0, and I think this was working until recently.

With this data

INFO - 12345 - Clicked One
INFO - 12345 - Clicked Two
INFO - 12345 - Clicked Three

and this configuration

input { stdin {} }
filter {
    grok { match => [ "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:user_id} - %{GREEDYDATA:msg_text}" ] }
    aggregate {
        task_id => "%{user_id}"
        code => "map['clicks'] ||= 0; map['clicks'] += 1;"
        push_map_as_event_on_timeout => true
        timeout_task_id_field => "user_id"
        timeout => 10
    }
}
output  { stdout { codec => rubydebug { metadata => false } } }

I get these messages

[2020-06-30T18:50:48,464][TRACE][logstash.filters.aggregate][main] Aggregate flush call with {:final=>false}
[2020-06-30T18:50:48,466][DEBUG][logstash.filters.aggregate][main] Aggregate remove_expired_maps call with '%{user_id}' pattern and 1 maps
[2020-06-30T18:50:48,487][DEBUG][logstash.filters.aggregate][main] Aggregate create_timeout_event call with task_id '12345'
[2020-06-30T18:50:48,513][DEBUG][logstash.filters.aggregate][main] Aggregate remove expired map with task_id=12345

and an event that looks like

{
   "user_id" => "12345",
    "clicks" => 3,
"@timestamp" => 2020-06-30T22:50:48.490Z,
  "@version" => "1"
}

If I enable java_execution then those messages disappear (the flush never occurs) and the map contents are never pushed.

2 Likes
2

This might explain why – after days of torturing my brain about finding the mistake in my configuration – disabling java execution helped to prevent my data from disappearing in nirvana. I was working on a huge pipeline and had not yet come around to creating a small test configuration to post the problem. So thanks a lot!

3

I have the same problem in 7.7.1 and was just about to post a question about it.
The problem appeared for me when I added "-w 1".
Adding "--java-execution=false" make it the counting example work again. However, my more complex setup with multiple aggregate blocks is still not working with --java-execution=false.

My environment is the following:

logstash 7.7.1
jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on 11.0.7+10-post-Ubuntu-2ubuntu218.04 +indy +jit [linux-x86_64]
java 11.0.7 (Ubuntu)
jvm OpenJDK 64-Bit Server VM / 11.0.7+10-post-Ubuntu-2ubuntu218.04

4

Just tried the test configuration with our server: Logstash 7.7.1. with logstash-filter-aggregate 2.9.1. The behavior is just like @Badger described.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

6

Thanks for highlighting this issue! A fix is in for v7.9.1, which should be coming out shortly.

1 Like