Logstash dash "-" (hyphen) problem with aggregation

Elastic Stack Logstash
1

Hello,
With my logstash configuration, I am reading JSON log, but in my JSON log I have fields like below;

"host-name":"comp-1.example.com",
"name-surname":"john-doe"

My configuration in logstash;

input {
    file {
        path => "/path/my_json_access.log"
        codec => json { charset => "UTF-8" }
    }
}
filter {
     mutate {
       add_field => { "computer_tag" => "something" }
}
aggregate {
    task_id => "%{host-name}"
    code => "map['host_count'] ||= 0; map['host_count'] += 1;
    map['hostname'] = event.get('host-name');
    map['name_surname'] = event.get('name-surname');
    timeout_task_id_field => "host-name"
    timeout => 100
    timeout_tags => ['_timeouttag1']
    timeout_code => "event.set('several_count', event.get('host_count') > 1)"
    push_previous_map_as_event => true
}}

output {
  if "_timeouttag1" in [tags] {
        rabbitmq {
            codec => "json"
            durable => true
            exchange => "myexchange"
            exchange_type => "fanout"
            host => "192.168.0.1"
            key => "logstash"
            password => "myp4ss"
            user => "test"
            workers => 1
    }}

At the end of the story, I cannot get any messages, or errors when I am starting logstash. I think there is something wrong with "dash", can you guys please give me advise or way for success.

My expected output is;
"computer_tag":"something",
"host_count":"12",
"hostname":"comp-1.example.com",
"name_surname":"john-doe",

2

So you only send events to rabbitmq if they have a _timeouttag1tag, but since you are using push_previous_map_as_event only one event (the last one) will timeout.

With that configuration I would expect nothing to happen for the first 100 seconds, then a single event written to rabbitmq.

3

I understand that, but whats the correct way to count host-name and publish when timeout ?
I would like to count "host-names" and publish to rabbit after 100 seconds.

4

Well the event that goes to rabbitmq will not have computer_tag since that mutate is only applied to the events that are discarded, not the event created by aggregate. Move the mutate after the aggregate.

Generally your aggregate looks good. I would add event.cancel to the code option of the aggregate and change push_previous_map_as_event to push_map_as_event_on_timeout

Which logstash version. I wonder if you are hitting this issue.

5

Thank you so much for your help Badger, I changed my config to this;

my logstash version 7.8.1

input {
    file {
        path => "/path/my_json_access.log"
        codec => json { charset => "UTF-8" }
    }
filter {
aggregate {
    task_id => "%{host-name}"
    code => "map['host_count'] ||= 0; map['host_count'] += 1;
    map['host-name'] = event.get('host-name');
    map['name-surname'] = event.get('name-surname');
    timeout_task_id_field => "host-name"
    timeout => 100
    timeout_tags => ['_timeouttag1']
    timeout_code => "event.set('several_count', event.get('host_count') > 1)"
    push_map_as_event_on_timeout => true
}
     mutate {
       add_field => { "computer_tag" => "something" }
}
}

output {
        rabbitmq {
            codec => "json"
            durable => true
            exchange => "myexchange"
            exchange_type => "fanout"
            host => "192.168.0.1"
            key => "logstash"
            password => "myp4ss"
            user => "test"
            workers => 1
    }}

Problems;

  • I am not getting any host_count field in my output.

  • I am getting output whatever json sends

  • computer_tag added additional to json input

  • I checked the logstash.yml and I have no #pipeline.java_execution line

Is there any problem with dash ( - ) characters in aggregation?

6

java_execution became the default in 7.0. The bug with flushers was introduced in 7.7 or 7.8 (I think) and fixed in 7.9.1. What version are you running? If it is an affected version try disabling java_execution.

I know of no problems with hyphens in field names.

7

7.8.1

8

That version definitely has the bug I linked to. Disable java_execution.

9

Badger thank you for your support, everything goes well now.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.