Increment counter if document already exists

Elastic Stack Logstash
1

lets say i have an index iptables-%{+YYYY.MM}, which holds iptables kernel generated logs.

i would like to:

1- create the doc if the log line doesn't exists
or
2- increment a field called counter if it already exists

every doc would be saved for the first time with a counter, incrementing on every insert of the same key.

input {
    beats {
        add_field => { "counter" => 1 }
        port      => "5044"
    }
}

the document_id would be based on some fields from message;
this would allow me to always have only one of each message type.

filter {
    fingerprint {
        add_tag             => [ "alreadyseen" ]
        concatenate_sources => true
        key                 => "alreadyseen"
        source              => [ "SRC", "SPT", "DST", "DPT" ]
        target              => "[@metadata][fingerprint]"
    }
}

output {
    elasticsearch {
        action              => "update"
        doc_as_upsert       => true
        document_id         => "%{[@metadata][fingerprint]}"
        index               => "iptables-%{+YYYY.MM}"
        sniffing            => true
        template_overwrite  => true
    }
}

but i have no idea on how to increment my counter.
any help?

2

so, after reading https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docs-update.html#_scripted_updates i tried the following

elasticsearch {
    action              => "update"
    doc_as_upsert       => true
    document_id         => "%{[@metadata][fingerprint]}"
    manage_template     => false
    script              => "ctx._source.counter++"
}

but the counter field always concatenate the number 1 instead of adding

on 1st execution counter: 1
on 2nd execution counter: 11
on 3rd execution counter: 111

and so on

i tried all lines below

script => "ctx._source.counter++"
script => "ctx._source.counter += 1"
script => "ctx._source.counter = ctx._source.counter + 1"
script => "ctx._source.counter = ctx._source.counter++"
3 
add_field => { "counter" => 1 }

Unless you have an index template counter will be a string in elasticsearch. You could mutate it to be an integer (and create a new index, since the old index already has a mapping).

mutate { convert => { "counter" => "integer" } }

However, you will then hit another problem: a null pointer exception when you do the initial insert :slight_smile:

script              => "if (ctx._source.counter != null) {ctx._source.counter++}"
4

the strange thing is that i actually do have a index template mapping

"properties": {
  "@timestamp": {
    "type": "date"
  },
  "counter": {
    "type": "long"
  },

and kibana mapped it as number, so it seems right

anyway, got it to work with

script => "if (ctx._source['counter'] == null) { ctx._source['counter'] = 1 } else { ctx._source.counter++ }"

and removing add_field from the input

5

also,

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.