Regexp is ignored in must_not clause

Elastic Stack Elasticsearch
1

I'm trying to exclude a log-entry from showing up in my hits, but it is necessary to use regular expression (expensive is allowed in configuration).

The log entry follows the pattern:

message:Aug 2 05:01:14 <hostname> <servicename>: (<item>@<IPv4>) [ERROR] <String with specific error message>

What is needed to exclude from my hits is the combination of "item" and "String with specific error message", as I need reaction if the error message is shown for any other values of "item>" than a specific value.
Furthermore "IPv4" is interchangeable

so, my JSON Request looks like this:

{
  "version": true,
  "size": 500,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "30s",
        "time_zone": "Europe/Copenhagen",
        "min_doc_count": 1
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "event.created",
      "format": "date_time"
    },
    {
      "field": "stamp",
      "format": "date_time"
    }
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": []
          }
        },
        {
          "bool": {
            "must": [
              {
                "match_phrase": {
                  "host.name": "<FQDN of host>"
                }
              },
              {
                "match_phrase": {
                  "log.file.path": "/path/to/logfile"
                }
              },
              {
                "match_phrase": {
                  "message": "error"
                }
              },
              {
                "match_phrase": {
                  "message": "<service>:"
                }
              }
            ]
          }
        },
        {
          "bool": {
            "must_not": [
              {
                "match_phrase": {
                  "message": "Can't open filename.xml: No such file or directory"
                }
              },
              {
                "match_phrase": {
                  "message": "Can't open other_filename.csv: No such file or directory"
                }
              },
              {
                "regexp": {
                  "message": {
                    "value": ".*<item>.*"
                  }
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-24H",
              "format": "strict_date_optional_time"
            }
          }
        }
      ]
    }
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}

(Sanitized for business purposes)

This gives me the same amount of hits that I expect by checking in Discover.

However, if I put ANYTHING behind "item" in my regexp, it seems to ignore it, ie:

{
   "regexp": {
      "message": {
         "value": ".*<item>.*<partial string>"
      }
   }
}

even though "item" and "partial string" co-exist in "message"

Where did I mess up?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.