[Logstash] elasticsearch filter error on mark logs

Elastic Stack Logstash
1

Hello,
I have two pipeline one main, listen filebeat and put data in elastic, and other with post process, the input is elastic, have one elastic filter (bellow), and put back data in elastic with updates.

The main goal of post process is know what is the most recent log, and for that I use filter lower than timestamp and process id.
My elastic filter in second pipeline is:

elasticsearch {
        add_tag => [ "obsoleteLogs" ]
        hosts => "elasticsearch:9200"
        index => ["filebeat-*"]
        query_template => "/markObsoleteLogs.json"
    }

And my file with query (markObsoleteLogs.json) is

{
  "size": 200,
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "must_not": [
              {
                "match_phrase": {
                  "tags.keyword": "obsoleteLogs"
                }
              }
            ],
            "should": [
              {
                "match_phrase": {
                  "@message.processInstId.keyword": "%{[processInstId]}"
                }
              },
              {
                "range": {
                  "timestamp": {
                    "lt": "%{[timestamp]}"
                  }
                }
              }
            ],
            "minimum_should_match": 2
          }
        }
      ]
    }
  }
}

But as result I don't have the most recent log as you can see, here:

{"timestamp": "2021-01-15T19:04:45.815Z",  "var_Qwerty": { "numero": "57354", "name:"abc"}, "processInstId": "512d1aa9568d11eb94640242ac120004", "tags": ["LogWithVars", "obsoleteLogs"], ...}

{"timestamp": "2021-01-15T19:04:45.140Z", "var_Qwerty": { "numero": "57354"}, "processInstId": "512d1aa9568d11eb94640242ac120004", "tags": ["LogWithVars"], ...}

The first is more recent but is marked with obsoleteLogs... You know what is the problem?

Thanks,
João

2

nobody? @Badger @Leslie_Hawthorn, can you help me? please

3

You are asking what query you should run in elasticsearch to find a particular document. That is an elasticsearch question, not a logstash question.

4

@Badger thanks for the answer but I tested the query in dev tools and works good!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.