Problem updating counter on elastic document

Elastic Stack Logstash
1

Hi
I populate ELK database reading log files by logstash.
Each document is populated using many log rows, each one inserts specific fileds.

When logstash read a specific action, it update a boolean filed "action_done" to true and it works perfectly.

Now, I'd like to have another field that counts number of occurrences of action.
So I write a script in elasticsearch output in this way:

      if [action] == 'my action' {
        elasticsearch {
          hosts => [ "my_host" ]
          user => "logstash_user"
          password => "logstash_pwd"
          ssl => false
          manage_template => true
          template_overwrite => true
          template_name => "mytemplate"
          template => "my-template.json"
          id => "specific_output_id"
          index => "my_index"
          action => "update"
          doc_as_upsert => true
          document_id => "my_id"
          script_type => "inline"
          script_lang => ""
          script => "if (ctx._source.action_counter == null) { ctx._source.action_counter = 1 } else { ctx._source.action_counter++ }"
        }
      } else {
        elasticsearch {
          hosts => [ "my_host" ]
          user => "logstash_user"
          password => "logstash_pwd"
          ssl => false
          manage_template => true
          template_overwrite => true
          template_name => "mytemplate"
          template => "my-template.json"
          id => "my_output_id"
          index => "my_index"
          action => "update"
          doc_as_upsert => true
          document_id => "my_id"
        }
      }

With this new output configuration, action_counter is almost always not present in the documents and it never has the correct value.
Also action_done field is not always set to true.

Where is the error?
Thanks in advance

2

Specifically, in a certain time interval there should be 435 documents with action_done field set to true and action_counter field greater than 0.
I have 331 documents with action_counter field that has the right value of occurrences and action_done field is set to false.
I have 95 documents with action_done field set to true and action_counter field is not present.
I have only 9 documents where both action_done and action_counter are present and in these documents action_counter fileds are not correct.

Other fileds that I insert in documents when [action] == "my_action", are present only if action_done field is set to true (in 104 documents).

3

I set action_done to true in logstash filter in this way:

if [action] == 'my_action' {
  mutate {
    replace => {
      "[action_done]" => true
    }
    convert => {
       "[action_done]" => "boolean"
    }
  }
}

In my-template.json, I set the action_count field type in this way:

...
  "mappings": {
    "properties": {
      "action_counter": {
        "type": "integer"
      }
    }
  }
...
4

I changed the script syntax to:

script => "if (ctx._source['action_counter'] == null) { ctx._source['action_counter'] = 1 } else { ctx._source.action_counter++ }"

Same result...

5

Resolved with two changes:

  1. I delete the "else" from logstash output, so in case with [action] == 'my action'
    elatsicsearch-output is performed twice.

  2. In the my_action-elasticsearch-output, I added the parameter:

retry_on_conflict => 10
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.