Aggregation error - nil can't be coerced into Fixnum

Elastic Stack Logstash
1

Hi,

I am trying to understand what is wrong with my aggregation:

My code:

 aggregate {
          task_id => "%{ResolutionDate}"
          code => "
                   map['Date'] = event.get('ResolutionDate')
                   map['TotalNonPrimeTimeUnavailabilityinSeconds'] ||= 0 
                   map['TotalNonPrimeTimeUnavailabilityinSeconds'] += event.get('NonPrimeTimeUnavailabilityinSeconds')
                   map['TotalBusinessTimeUnavailabilityinSeconds'] ||= 0 
                   map['TotalBusinessTimeUnavailabilityinSeconds'] += event.get('BusinessTimeUnavailabilityinSeconds')
                   if event.get('Classification').eql?('High')
                                map['CountClassificationHigh'] ||= 0; map['CountClassificationHigh'] += 1; 
                   elsif event.get('Classification').eql?('Low')
                                map['CountClassificationLow'] ||= 0; map['CountClassificationLow'] += 1;
                   end
                   #event.cancel()
          "
          push_previous_map_as_event => true
          map_action => "create_or_update"
          push_map_as_event_on_timeout => true
          timeout => 300
          inactivity_timeout => 30
 }

The error:

[2019-02-16T12:04:28,410][ERROR][logstash.filters.aggregate] Aggregate exception occurred {:error=>#<TypeError: nil can't be coerced into Fixnum>, :code=>"\n map['Date'] = event.get('ResolutionDate')\n map['TotalNonPrimeTimeUnavailabilityinSeconds'] ||= 0 \n map['TotalNonPrimeTimeUnavailabilityinSeconds'] += event.get('NonPrimeTimeUnavailabilityinSeconds')\n map['TotalBusinessTimeUnavailabilityinSeconds'] ||= 0 \n map['TotalBusinessTimeUnavailabilityinSeconds'] += event.get('BusinessTimeUnavailabilityinSeconds')\n if event.get('Classification').eql?('High')\n map['CountClassificationHigh'] ||= 0; map['CountClassificationHigh'] += 1; \n elsif event.get('Classification').eql?('Low')\n map['CountClassificationLow'] ||= 0; map['CountClassificationLow'] += 1;\n end\n #event.cancel()\n ", :map=>{"Date"=>2018-09-10T05:22:16.000Z, "TotalNonPrimeTimeUnavailabilityinSeconds"=>0}, :event_data=>{"Status"=>"Fechado", "Organization Support Group"=>"Sustenta\\xE7\\xE3o", "Create Date"=>"2018-09-09 16:20:24", "INCIDENTE"=>"INC000004054472", "Company"=>"AMDOCS do Brasil", "Resolution Date"=>"2018-09-10 08:22:16", "BusinessTimeContractCommitment"=>"99.5", "@metadata"=>{"host"=>"SHARONSA03", "path"=>"C:/Users/sharonsa/Work/GSS/VIVO/kpi/BaseIncidentes.csv"}, "type"=>"csv", "message"=>"INC000004054472,Fechado,AMDOCS do Brasil,Sustenta\\xE7\\xE3o,Suporte Funcional N2 - OMS - Ordens,2018-09-09 16:20:24,2018-09-10 08:22:16,,,\\r", "CreateDate"=>2018-09-09T13:20:24.000Z, "Designated Group"=>"Suporte Funcional N2 - OMS - Ordens", "path"=>"C:/Users/sharonsa/Work/GSS/VIVO/kpi/BaseIncidentes.csv", "@timestamp"=>2019-02-16T10:04:25.361Z, "NonPrimeTimeContractCommitment"=>"98", "Classification"=>"\\r", "@version"=>"1", "host"=>"SHARONSA03", "ResolutionDate"=>2018-09-10T05:22:16.000Z}}

Thanks
Sharon

2

There is no BusinessTimeUnavailabilityinSeconds field in the event_data shown, so event.get returns nil, but it needs a fixnum to add it to zero. It will not coerce that.

3

My aggregation code now looks like that:

     aggregate {
              task_id => "%{ResolutionDate}"
              code => "
                       map['Date'] = event.get('ResolutionDate')
                       if event.get('NonPrimeTimeUnavailabilityinSeconds').to_i > 0
                              map['TotalNonPrimeTimeUnavailabilityinSeconds'] ||= 0; map['TotalNonPrimeTimeUnavailabilityinSeconds'] += event.get('NonPrimeTimeUnavailabilityinSeconds').to_i;
                       end
                       if event.get('BusinessTimeUnavailabilityinSeconds').to_i > 0
                              map['TotalBusinessTimeUnavailabilityinSeconds'] ||= 0; map['TotalBusinessTimeUnavailabilityinSeconds'] += event.get('BusinessTimeUnavailabilityinSeconds').to_i;
                       end
                       if event.get('Classification').eql?('High')
                                    map['CountClassificationHigh'] ||= 0; map['CountClassificationHigh'] += 1; 
                       elsif event.get('Classification').eql?('Low')
                                    map['CountClassificationLow'] ||= 0; map['CountClassificationLow'] += 1;
                       end
                       #event.cancel()
              "
              push_previous_map_as_event => true
              map_action => "create_or_update"
              push_map_as_event_on_timeout => true
              timeout => 300
              inactivity_timeout => 30
     }

No more errors in the log, but I don't see the aggregated fields in the Kibana.

Should they appears on the event like all the other fields?

Thanks
Sharon.

4

Yes.

Note that in the event_data in your first post, you have "Classification"=>"\r", so that would not trigger either branch of the if/elsif.

5

Can't see any of the aggregation fields in the Kibana.

No errors / info/warnings in the log.

Any idea?

Thanks
Sharon.

6

Make one of the added fields unconditional

map['JustTesting'] ||= 0; map['JustTesting'] += 1;

If that shows up then the problem is that your events do not trigger any of the conditionals.

7

It doesn't show up.

8

When I said yes, I should have said no. Consider this file

{ "ResolutionDate": 3 }
{ "ResolutionDate": "foo" }

and this configuration

    json { source => "message" }
    aggregate {
        task_id => "%{ResolutionDate}"
        code => "map['JustTesting'] ||= 0; map['JustTesting'] += 1"
        timeout_task_id_field => "Date"
        push_previous_map_as_event => true
        map_action => "create_or_update"
        push_map_as_event_on_timeout => true
        timeout => 8
        inactivity_timeout => 4
    }

That will get you

{
           "message" => "{ \"ResolutionDate\": 3 }",
        "@timestamp" => 2019-02-16T20:39:49.724Z,
          "@version" => "1",
    "ResolutionDate" => 3,
              "path" => "/home/user/foo.json"
}
{
           "message" => "{ \"ResolutionDate\": \"foo\" }",
        "@timestamp" => 2019-02-16T20:39:49.726Z,
          "@version" => "1",
    "ResolutionDate" => "foo",
              "path" => "/home/user/foo.json"
}
{
    "JustTesting" => 1,
     "@timestamp" => 2019-02-16T20:39:49.837Z,
       "@version" => "1",
           "Date" => "3"
}
{
    "JustTesting" => 1,
     "@timestamp" => 2019-02-16T20:39:54.804Z,
       "@version" => "1",
           "Date" => "foo"
}

Note that you get the third event immediately after the second (that's being triggered by 'push_previous_map_as_event => true') and then the final event comes 5 seconds later (driven by 'push_map_as_event_on_timeout => true').

If the field ResolutionDate does not exist then nothing gets aggregated.

9

What ever I am trying to do, I am nor getting the aggregations.

I used in the task_id the field named: INCIDENTE, which for sure exist in any event, but still can't see the aggregated fields.

My aggregation looks like that:

 aggregate {
            task_id => "%{INCIDENTE}"
            code => "
                     if event.get('NonPrimeTimeUnavailabilityinSeconds').to_i > 0
                            map['TotalNonPrimeTimeUnavailabilityinSeconds'] ||= 0; map['TotalNonPrimeTimeUnavailabilityinSeconds'] += event.get('NonPrimeTimeUnavailabilityinSeconds').to_i;
                     end
                     if event.get('BusinessTimeUnavailabilityinSeconds').to_i > 0
                            map['TotalBusinessTimeUnavailabilityinSeconds'] ||= 0; map['TotalBusinessTimeUnavailabilityinSeconds'] += event.get('BusinessTimeUnavailabilityinSeconds').to_i;
                     end
                     if event.get('Classification').eql?('High')
                                  map['CountClassificationHigh'] ||= 0; map['CountClassificationHigh'] += 1; 
                     elsif event.get('Classification').eql?('Low')
                                  map['CountClassificationLow'] ||= 0; map['CountClassificationLow'] += 1;
                     end                           
            "
            timeout_task_id_field => "Incident"
            push_previous_map_as_event => true
            map_action => "create_or_update"
            push_map_as_event_on_timeout => true
            timeout => 200
            inactivity_timeout => 5
 }

I changed the log to debug mode but I can't see something wrong there.

10

Simple aggregation like that, still doesn't work for me:

         aggregate {
                    task_id => "%{INCIDENTE}"
                    code => "
                             map['JustTesting'] ||= 0; map['JustTesting'] += 1
                    "
         }

What do I need to continue and check to understand what it wrong?

Regards,
Sharon.

11

Nothing will trigger the aggregation being pushed as an event in that case. I think the minimum that will work is below. I created a file that contains

{ "INCIDENTE" : 0 }

Then I run with this configuration

input { file { path => "/home/user/foo.json" sincedb_path => "/dev/null" start_position => "beginning" } }
filter {
    json { source => "message" }
    aggregate {
                task_id => "%{INCIDENTE}"
                code => "
                     map['JustTesting'] ||= 0; map['JustTesting'] += 1
                "
                push_map_as_event_on_timeout => true
                inactivity_timeout => 2

     }
}
output { stdout { codec => rubydebug } }

Using a file input is important, because if you a generator the pipeline will stop executing, and there will be no thread running that can execute the timeout code. That config gets me

[2019-02-20T17:09:24,994][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
    "@timestamp" => 2019-02-20T22:09:24.992Z,
     "INCIDENTE" => 0,
          "path" => "/home/user/foo.json",
       "message" => "{ \"INCIDENTE\" : 0 }",
      "@version" => "1"
}
{
     "@timestamp" => 2019-02-20T22:09:30.075Z,
    "JustTesting" => 1,
       "@version" => "1"
}

If on the other hand I use a generator input

input { generator { count => 1 message => '{ "INCIDENTE" : 0 }' } }

Then with that same filter I get

[2019-02-20T17:14:15,587][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
    "@timestamp" => 2019-02-20T22:14:15.565Z,
      "sequence" => 0,
     "INCIDENTE" => 0,
       "message" => "{ \"INCIDENTE\" : 0 }",
      "@version" => "1"
}
[2019-02-20T17:14:15,734][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x3765a362 run>"}

with no aggregation.

12

I am getting as input a csv file.

The events looks like that:

INC000004074534,Fechado,Sonda IT,Gestão de Serviços de TI,Suporte Inicial N1,2018-09-14 20:07:39,2018-09-14 20:55:18,,,

the first field is the INCIDENTE.

So, I assume your aggregation example, should work for me too.

I will put it in the Logstash and test. I will keep update.

Thanks
Sharon.

13

Maybe the problem is that I don't have a source field ?

I can't see a source field in output in Kibana.

14

Can you show us that event in the JSON tab in Kibana?

15 
{
  "_index": "logstash2019.02.20incidents-kpi",
  "_type": "doc",
  "_id": "IC65DGkBptv9o3KvOjpa",
  "_version": 1,
  "_score": null,
  "_source": {
    "BusinessTimeContractCommitment": 25.5,
    "Company": "Sooo IT",
    "CreateDate": "2019-01-20T18:09:58.000Z",
    "Resolution Date": "2019-01-21 07:59:55",
    "Organization Support Group": "Gestão de Serviços de TI",
    "@version": "1",
    "Designated Group": "Suporte Inicial N1",
    "NonPrimeTimeSecondsPerWeek": 226800,
    "ResolutionDate": "2019-01-21T05:59:55.000Z",
    "host": "TTT03",
    "@timestamp": "2019-02-20T21:04:19.358Z",
    "BusinessTimeSecondsPerWeek": 378000,
    "message": "INC000004567194,Resolvido,Sonda IT,Gestão de Serviços de TI,Suporte Inicial N1,2019-01-20 20:09:58,2019-01-21 07:59:55,,,\r",
    "path": "C:/Users/sharonsa/Work/kpi/BaseIncidentes.csv",
    "type": "csv",
    "INCIDENTE": "INC000004567194",
    "Create Date": "2019-01-20 20:09:58",
    "Status": "Resolvido",
    "NonPrimeTimeContractCommitment": 95
  },
  "fields": {
    "CreateDate": [
      "2019-01-20T18:09:58.000Z"
    ],
    "ResolutionDate": [
      "2019-01-21T05:59:55.000Z"
    ],
    "@timestamp": [
      "2019-02-20T21:04:19.358Z"
    ]
  },
  "sort": [
    1548050395000
  ]
}
16

Maybe it is not the same one, but other with the same format exactly.

17

Do you have dynamic mapping disabled? Have you ever tried running logstash with this?

output { stdout { codec => rubydebug } }

I am wondering if logstash is adding the fields and elasticsearch is ignoring them. On your elasticsearch server try

curl -X GET "localhost:9200/logstash2019.02.20incidents-kpi/_mapping"

And look for "dynamic"

18

I can't find "dynamic" in it

19

OK, then I am out of ideas, at least for now.

20

Why can't I find the dynamic definition?

It need to be dynamic. I am not creating the mapping before loading my events.