Why my query does not work correctly after create new filed in logstash

[baber1223]

This is my sample data:

134.255.248.30 - - [20/Nov/2023:09:04:57 +0330] "GET /serve/finnotech/validateDest?key=3f94393b5eaab29a167e5edc8a99860cba121550053bd113d291d71f146a7fa0&parameters=%7B%22dest%22:%22IR520190000000208813572000%22%7D HTTP/1.1" 200 73 "-" "axios/0.19.2" 86

and this is my Grok Pattern :

%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

and du simulate it is showing follow structured data :

{
  "request": "/serve/finnotech/validateDest?key=3f94393b5eaab29a167e5edc8a99860cba121550053bd113d291d71f146a7fa0&parameters=%7B%22dest%22:%22IR520190000000208813572000%22%7D",
  "auth": "-",
  "ident": "-",
  "response": "200",
  "bytes": "73",
  "clientip": "134.255.248.30",
  "verb": "GET",
  "httpversion": "1.1",
  "timestamp": "20/Nov/2023:09:04:57 +0330"
}

So I have written follow pipeline

input{
   beats {

    port => 5071

}

}

filter{

  grok{
    match  =>  { "message" => '%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)' }

}

}




output{

  stdout{}
  elasticsearch {
   index => newo
   hosts => ["https://IP:9200"]
   cacert => '/etc/logstash/certs/http_ca.crt'
   user => "elastic"
   password => "password"

}

}

now it has created index with the name newo and after I created dataview it is showing all fields but when I filter bytes > 553

It is showing also less than 300

[baber1223]

Please see attached pic it is showing type of bytes field is text . Although it is defines number in the pattern

[sholzhauer]

Hi, logstash does not influence how the mappings are done on elasticsearch side. That is where your solution lies.

To make your field work correctly as a number you need to update the mapping on your indices and optionally update the dataview to recoginize the number as bytes.

Index mapping

You need to either update or create your index template and add a mapping field to it, based on your examples it should look like (addition):

{
  "properties": {
    "bytes": {
      "type": "float"
    }
}
}

Dataview

Once you have an index rollover (new index as mappings are only applied upon index creation) you can update the format in the dataview. When editing the dataview find your field and update the format to a byte:

Summary

Currently the field is behaving as a text which means the > and < operators will not function as expected. Updating the mapping and optionally the format will allow you to correctly use your field

[baber1223]

@sholzhauer

{
  "properties": {
    "bytes": {
      "type": "float"
    }
}
}

@sholzhauer

So thanks .

follow is part of my mapping for that index . My index name is neo

      },
      "bytes": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256

So Does it your mean have to update type of bytes manually ? How can do it ?

[sholzhauer]

You have to modify the "type": "text" into the float part

[baber1223]

@sholzhauer

Would you please say how can modify it ? I could not find any options in the mapping to do it

Does it correct ?

PUT neo/_mapping 
{
  "properties": {
    "bytes": {
      "type": "float"
    }
}
}

I got this error :

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "mapper [bytes] cannot be changed from type [text] to [float]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "mapper [bytes] cannot be changed from type [text] to [float]"
  },
  "status": 400
}